A climate tech pilot is not complete when equipment is installed or farm participation is confirmed. A livestock methane pilot is a process for learning under what conditions a technology can be used and what needs to be improved. Chapter 11 of Technology and Solutions for Demonstrating Carbon Reduction emphasizes setting hypotheses and decision criteria in advance.
Define the concept and scope of application first
In a pilot contract, check operating and stopping conditions as well as the hypothesis. Responsibilities for safe installation and removal, continuity of farm operations, equipment failure, and missing data need to be defined. Agree on what participants will provide and what results they will receive. The final report should be more than a summary of a success story. It should record differences between the advance plan and actual implementation, when and why changes were made, and questions that remain unresolved. A pilot has value when the person making a scale-up decision can also understand its limitations.
A farm pilot is more than installing equipment. Farmers must continue their daily livestock work while making time for recordkeeping and checks. Installation photos and the number of participating animals show activities, but emissions-reduction effects and business viability require separate evidence.
When the trial questions and decision criteria are set in advance, participants can understand why records are being collected. Even when results differ from expectations, preserve the data and distinguish what to scale up from what to improve or stop.
In addition to the primary question, you can set secondary indicators in advance. Use exploratory indicators to develop new hypotheses, but do not present post hoc findings as outcomes that were established beforehand. When testing multiple indicators, address the effects of multiple comparisons and selective reporting.
Discipline in changing the protocol. Conditions may need to change during a pilot. Feed supplies may be interrupted, equipment may fail, or an unexpected situation may arise. A change itself is normal. The problem is when the change is not documented.
When making a change, document the reason, timing, scope covered by the results, and impact assessment. A plan may need to change because of a safety issue or data error. But do not present an analysis chosen after seeing results as if it had been specified in advance. Distinguish the results under the original plan from those after the change, and test any new hypothesis using separate data.
A crossover design is useful when the same animals can be compared under multiple conditions. Assign the treatment order, analyze period effects, and allow an interval that accounts for rumen adaptation and carryover effects. If effects persist or growth means animals do not return to their previous condition, a parallel design may be more appropriate. Choose the design based on the biological nature of the treatment, not convenience.
Animal welfare comes first. Animal-attached equipment, confinement in chambers, and feed-attraction devices all place some burden on animals. The knowledge gained must be valuable enough to justify that burden, and the design should first minimize it. If signs of distress appear, the trial must be stopped even if that means losing data.
Conditions and limits for field application
Imagine a project that installed sensors and feeding equipment but did not define its primary evaluation indicator. If the team chooses the item that looks best as the main outcome after seeing the results, interpretation may be biased. Distinguish the primary question, secondary indicators, and exploratory items, and record the reasons and timing of interim changes. Include the operational burden on the farm in the plan as well.
Continuity of operations comes first. If the pilot delays feeding, restricts ventilation, or changes delivery schedules, the farm incurs a loss. The trial design should fit within the farm’s existing workflow, and unavoidable burdens should be compensated.
Put stopping criteria and incident-reporting procedures in the contract. Verbal agreements are not enough. The document should specify who can call a stop and under what circumstances, how costs will be handled if the trial is stopped, and who must be notified of an incident and by when.
Record results that are lower than expected or show no clear effect as well. Including observations that differed from expectations and a review of possible causes in the report can help prevent the same problem from recurring in follow-up trials.
When a pilot ends, a decision has to be made. Yet this decision is often delayed. It is common to roll into a “second pilot” without reaching a conclusion. Costs keep accruing, while learning does not increase.
Cross-checking feeding times with inventory and mixing records can help reveal interruptions in feeding that purchase evidence alone would not show. However, these records alone cannot establish actual intake by each animal or the amount of emissions reduced. The value of the data cross-checks discussed in Chapter 9 is that they expand what can be observed and help identify where further confirmation is needed.
Evidence needed for investment review
Investors should check the pilot’s success conditions and its criteria for scaling up, improving, or stopping. They should separately ask whether the number of devices installed and the uptime rate are indicators that demonstrate technology effects. It is also necessary to assess whether the staff and costs used only during the pilot can be sustained in the full program.
A project needs a budget and reporting schedule. However, an announced reporting date alone does not mean that sufficient evidence has been collected. Separately check whether the observation period and data quality are fit for the decision.
In multi-farm pilots, assess whether effects hold across new livestock species, husbandry conditions, seasons, and operating circumstances. Whether the staff and budget concentrated on the pilot can be sustained in broader deployment is a separate question. Check the costs of training, maintenance, and data review needed for scale-up. Where evidence is insufficient, set criteria for further trials, improvements, a change in use, or stopping.
Pilot and full-program conditions differ. Interest, staff, and budget are concentrated in a pilot, but are spread more thinly in the full program. Across many fields, effects observed in pilots have not always been sustained in full-scale programs.
Criteria to keep in view when making a decision
A good pilot leaves evidence needed for the next decision. Results that fall short of expectations can also inform improvements to design and operations. When questions, records, and decision criteria are connected, a pilot can serve as evidence for a deployment plan.
About the author
Bongjun Lee graduated from the Graduate School of Management of Technology (MOT) at Sogang University. He spent seven years at Infitron Co., Ltd. and 20 years at Honeywell, working for a total of 27 years in gas measurement in service engineering, technical sales, and channel management. He is now the founder of AI Safety Korea Co., Ltd., where he is researching livestock methane monitoring and measurement, reporting, and verification (MRV) systems using sensors, AI, and data analytics.