If methane falls after introducing low-carbon feed, it is tempting to attribute the change to the feed. But other conditions may have changed in the meantime. Chapter 11 of Technology and Solutions for Demonstrating Carbon Reduction looks at controlled trials for methane reduction and comparison design when verifying carbon-reduction effects. An important question is how weather and husbandry conditions over the same period were handled.
What the book suggests distinguishing first
Before looking at the results, be sure to define the primary question and primary metric. If conditions had to be changed along the way, it is helpful to record why and when, and how much of the results had been seen before the decision was made. Hiding a change or presenting it as if it had been planned from the outset makes review difficult. A demonstration with an inadequate control can still help us learn about field problems. However, stronger evidence is needed to make a strong causal claim from those data. I hope you will explain separately what you learned and what you established.
A farm demonstration involves more than installing equipment. Farmers have to keep up their routine husbandry while also making time for records and inspections. Installation photos and the number of participating animals show that activities took place, but emissions-reduction effects and business viability must be confirmed using separate evidence.
To understand a farm’s numbers, start by examining the space where those numbers are generated. Barn structure and airflow, where animals spend time, and routes for feeding and cleaning are the starting points for determining observation locations and maintenance plans. The design must also show how reliable data can be obtained continuously as part of daily farm operations.
In addition to the primary question, you can define secondary metrics in advance. Exploratory metrics can be used to identify new hypotheses, but do not present a post hoc discovery as if it were a result established in advance. When testing multiple metrics, address the effects of multiple comparisons and selective reporting.
Protocol-change discipline. Conditions sometimes need to change during a demonstration. Feed supplies may be interrupted, equipment may fail, or an unexpected situation may arise. The change itself is normal. The problem is when it is not recorded.
When conditions change, record the reason, timing, scope of results reviewed, and impact assessment. A plan may need to be revised because of a safety issue or data error. However, do not present an analysis decided on after looking at the results as if it had been specified in advance. Distinguish results under the original plan from those after the change, and verify newly generated hypotheses using separate data.
For a concurrent-control design, obtain random assignment and independent replication where possible. If the treatment is applied to an entire barn, the barn may be the experimental unit rather than the individual cow. A difference between one treatment barn and one control barn makes it difficult to separate the treatment effect from the difference between barns. Analysis should account for the hierarchy of farm, barn, animal, and date, as well as correlations from repeated measurements.
If a control is difficult, repeated baselines can help identify variability and time trends, but those data alone should not be treated as having eliminated all confounding. Even if a change is smaller than the natural variation in individual observations, the average effect can be estimated with enough independent replicates. Conversely, a large apparent difference may be due to season or growth. Present the effect estimate and confidence interval together with the limitations of causal interpretation.
What to look at in the field
Imagine a farm that changed its feeding method during a seasonal transition. It is difficult to separate the effects of the two changes using a single before-and-after comparison. You could consider a design with a control group or one that compares different conditions, but each has its assumptions and limitations. We also need to consider whether the farm can actually operate that way.
Animal welfare comes first. Animal-attached equipment, chamber confinement, and baited feeding devices each place some burden on animals. The knowledge gained must be valuable enough to justify that burden, and the design should first minimize it. If signs of distress appear, the study should be stopped even if that means losing data.
Continuity of operations comes first. If a demonstration delays feeding, limits ventilation, or changes shipping schedules, the farm incurs a loss. The demonstration design must fit within the farm’s existing workflow, and unavoidable burdens should be compensated.
Cross-checking feeding times with inventory and mixing records can help detect interruptions in feeding that purchase receipts alone would not reveal. However, such records cannot establish the actual intake of individual animals or the amount of emissions reduction. The value of the data cross-checking discussed in Chapter 9 is that it expands what can be observed and identifies where further confirmation is needed.
Questions to ask during investment review
When reviewing evidence of effects, ask about the comparison group, assignment method, and independent replication. A large number of readings across time from sensors does not mean there are many independent animals or farms. The volume of repeated measurements and the basis for generalizing results should be considered separately.
A project needs a budget and a reporting schedule. But a fixed announcement date does not by itself mean enough evidence has been collected. Check separately whether the observation period and data quality are suitable for the decision.
Registration need not be elaborate. Finalize a few dated documents and share them with relevant parties or have a third party retain them. These records help determine afterward whether changes were made.
In demonstrations across multiple farms, examine whether effects are maintained with new livestock types, husbandry conditions, seasons, and operating environments. Whether the staff and budget concentrated in a pilot can be maintained during general rollout is a separate question. Check the costs of training, maintenance, and data review required for scaling up. Areas with insufficient evidence should be documented as criteria for additional trials, remediation, changing the intended use, or stopping.
Questions to leave with before closing the book
Demonstrations do not always take place under ideal conditions. What matters is explaining the limits of the comparison and setting the scope of the claim accordingly. Write down side by side what changed and how far you can explain its cause.
About the author
Bongjun Lee graduated from the Graduate School of Management of Technology (MOT) at Sogang University. He spent seven years at Infitron Co., Ltd. and 20 years at Honeywell, working for a total of 27 years in gas measurement in service engineering, technical sales, and channel management. He is now the founder of AI Safety Korea Co., Ltd., where he is researching livestock methane monitoring and measurement, reporting, and verification (MRV) systems using sensors, AI, and data analytics.