AI model validation requires separating evaluation data according to the intended use. In methane data analysis, performance measured by mixing data from adjacent times at the same farm may differ from performance when applied at another farm. Chapter 8 of Technology and Solutions for Demonstrating Carbon Reduction emphasizes the boundary between training and evaluation and comparison with a simple baseline model.
Define the concept and scope of application first
Evaluation data should be separated into data used for model selection and data reserved for the final performance check. If data repeatedly used to compare several methods are presented as a final evaluation, actual performance in application may be overestimated. Define the evaluation procedure first to match the intended use. Data distributions and operating conditions may change in the field. Document criteria for detecting changes, as well as procedures for reevaluation and suspension of use. Alongside performance metrics, it is important to examine the actions and costs that could result from an incorrect prediction.
Reporting (R) preserves context and calculations. It documents the entire process by which raw data were processed under particular rules to reach a conclusion. This includes how missing data were filled, which criteria were used to exclude outliers, which model and version were used, and how uncertainty was estimated. A report is evaluated by reproducibility, not by how impressive its conclusion sounds.
The system boundary is the basis for defining which emission sources, gases, periods, and organizations are included in a calculation. If the boundary is unclear, different parties may expect different results for the same claimed reduction. Check the inclusion and exclusion rules and materiality criteria in the applicable methodology, and state at the beginning of the report whether the result covers the whole farm or a particular emission source.
An emission-factor-based baseline is also an option. IPCC Tier 1 uses default factors, while Tier 2 for enteric fermentation reflects feeding characteristics, gross energy intake, methane conversion rates, and other factors in greater detail. Average factors may not capture the effect of a technology at an individual farm, but using factors is not inherently unsuitable for compensation. The deciding questions are whether the factors and activity data are allowed under the applicable methodology, and whether its conditions of application, uncertainty treatment, and verification requirements have been met.
A practical path forward is to accumulate concentration observations together with reference emission-rate measurements, and link these to ventilation, weather, animal counts, and feed information. Prediction error should be assessed on farms and time periods not used to develop the model, and the scope of application should be stated. Even a large volume of simultaneous observations does not automatically increase model reliability if they lack representativeness or share a common bias.
Methane emissions associated with feeding and rumination vary even within a single day. The timing of peaks depends on feed, feeding practices, and animal behavior, so it cannot be fixed at intervals of a few dozen minutes. Timestamp errors between records and delays as gases move through tubing should be kept sufficiently small relative to the timescale of the change being analyzed.
If the time reference has not been confirmed, first estimate the size of the error and delay and assess their effects on the analysis. An error that is small for daily aggregation may be significant in analyzing events at the minute level. Data that cannot be recovered should be excluded from combined analysis at that timescale or the scope of interpretation should be limited, with the reason recorded.
Conditions and limitations of field application
Suppose sensor records from several days are randomly divided into training and evaluation sets. Good performance at fitting recurring daily patterns does not mean that the results explain conditions at a new farm. Evaluations for future operation and for expansion across farms require different data-splitting methods from the outset. Preprocessing and missing-data handling must also avoid using information from the evaluation data in advance.
First, separate the evaluation data. If readings from adjacent times at the same farm are randomly mixed, performance that has learned recurring patterns at that site may be mistaken for performance at a new farm. Use chronological separation when the goal is to predict future operation, and farm-level separation when the goal is deployment at other farms. Distinguish the validation data used for model selection from the data used for the final performance check, and run preprocessing and gap correction using rules established from the training data.
AI evaluation data should be separated according to the intended use. To predict future operations, consider chronological ordering; to deploy at other farms, consider separating data by farm. Also distinguish the data used to select the model from those reserved for the final performance check. Compare with a simple baseline model, and examine not only average error but also bias by farm and season and the conditions under which the model fails. This is necessary to explain its actual scope of use.
Specify what happens when the system fails. If calibration has expired, communications are missing, or the feed, season, or ventilation conditions have not been evaluated, limit or suspend the output and refer the case to a responsible person. Do not resume use immediately after retraining. Validate the changed model using independent evaluation data and record its deployment date and approver. A model’s role in supporting decisions and external verification of emissions reductions are separate functions.
Evidence needed for investment review
Investors should examine errors by farm and season and persistent bias in addition to average performance. They should also check what has improved compared with an existing simple prediction method. After retraining, it is necessary to check the changed performance using independent evaluation data and record the deployment date and approver.
Data lineage is the path showing which raw data and transformations produced a final figure. Record unit conversions, corrections, filtering, and aggregation rules along with the equipment and channel and the time of generation. Also retain who ran and approved each process and what was changed later, and why. The fact that raw data remain available and the fact that a qualified reviewer can recheck the processing are conditions that should be examined together.
If the questions and decision criteria for a trial are defined in advance, participants can understand what they are recording data for. Even when results differ from expectations, retain the data and judge separately what to scale up and what to improve or stop.
Criteria to keep in view when making a decision
Model validation does not end with presenting a high score. Confirm the scope of use using data suited to the deployment purpose and the cost of failure. For conditions outside the training scope, there must be a basis for reevaluation or restricting use.
About the author
Bongjun Lee graduated from the Graduate School of Management of Technology (MOT) at Sogang University. He spent seven years at Infitron Co., Ltd. and 20 years at Honeywell, working for a total of 27 years in gas measurement in service engineering, technical sales, and channel management. He is now the founder of AI Safety Korea Co., Ltd., where he is researching livestock methane monitoring and measurement, reporting, and verification (MRV) systems using sensors, AI, and data analytics.