A carbon data platform looks most reliable on a screen while it is connected to the internet. Sensor readings form a continuous curve, indicators for each farm are updated, and the report button works. On farms, however, connections fail because of gaps in mobile coverage, router faults, power quality, or equipment maintenance. A greater problem than a temporarily frozen screen is the loss of evidence needed to explain later what was actually measured.

Carbon Data differs in character from ordinary application logs. It may be used for before-and-after comparisons, external reviews, and supply-chain reporting, so not only the values but also the measurement time, device, calibration status, quality flags, and transformation history must remain available. If a disconnected interval is filled with a straight line for convenience, or the same data are counted twice after reconnection, the reduction estimate itself becomes unreliable. Edge–Cloud design is not merely a connectivity issue; it is an issue of evidence continuity.

Correcting a misconception: does cloud backup mean the data are safe?

Data already uploaded to the cloud can be protected, but data generated during a disconnection do not yet exist in the cloud. If the system relies only on temporary memory in the field device, those data may disappear after a reboot or when storage runs out. Conversely, leaving a file on a local disk is not sufficient. If the file is corrupted, its clock is wrong, its transmission status is unknown, or no one can determine who changed it, it is not a verifiable record.

It is also necessary to distinguish “the internet connection failed” from “the sensor stopped.” The sensor may have continued collecting normally while only the uplink was lost, or collection itself may have stopped because the gateway lost power. If both situations appear as the same blank interval, data users cannot determine the cause. Collection status, local storage status, and cloud receipt status must be observed separately.

Three times and one identifier are required

At least three kinds of time are useful for each field-data record. observed_at is when the sensor made the observation, received_at_edge is when the gateway received it, and ingested_at_cloud is when the cloud received it. Under normal connectivity they are nearly identical, but the differences grow during outages and retransmission. Preserving those differences makes it possible to distinguish measurement delay from transmission delay.

It is safer to exchange times in a format that includes an explicit UTC offset and convert them to local time only for display. RFC 3339, subsequently updated by RFC 9557, defines a consistent timestamp representation for internet protocols. During a period when a device clock is not synchronized, record clock_status, the estimated error, and the clock-correction event rather than discarding the values. Silently overwriting incorrect times can create a false relationship between feeding or ventilation events and changes in methane.

Each observation needs a globally collision-resistant event_id. When retransmitting an observation, retain the same ID instead of creating it as new data. The server must provide idempotency so that receiving an already processed ID again does not increase the reduction aggregate. The HTTP standard also explains that idempotent methods such as PUT are suitable for automatic retries because repeated identical requests have the same intended effect. Even when POST is used, a separate idempotency key and deduplication rules can be defined.

Field scenario: four errors after a 17-hour outage

Suppose a farm loses its internet connection from 3 p.m. until 8 a.m. the next day. The sensors and gateway continue operating and store readings locally at 10-second intervals. When the connection returns, the first risk is a transmission surge. Current real-time readings compete with 17 hours of backlogged data, so some messages may expire or arrive out of order.

The second risk is duplication. If the gateway sends data but the connection fails again before it receives a response, it cannot know whether the transmission succeeded. Resending the same batch after recovery is correct behavior, but if the server aggregates the same event twice, emissions are doubled. The third risk is a time error. If a device reboots during the outage and its clock resets, the order of observations may be reversed. The fourth is storage exhaustion. If capacity was not calculated in advance, the oldest raw data may be silently overwritten.

To prevent these four errors, the gateway must first commit data to a durable queue, assign event IDs and sequence numbers, and then transmit them. It should remove only data whose receipt and durable storage the cloud has confirmed, and do so according to the local retention policy. Current alerts and historical backlog should be sent at different priorities, while the server calculates missing ranges by farm, device, and sequence. Recovery should be declared complete not when the screen says “connected,” but only after reconciling the expected event count with the received, duplicate, and corrupted counts.

How to divide responsibilities between Edge and Cloud

The edge is the evidence repository closest to the measurement. It receives raw data, performs basic schema checks, associates device status, stores data locally with encryption, assigns sequence numbers, maintains the transmission queue, and provides essential local alerts. Even during an outage, operators must be able to view recent status and remaining storage. Local changes to important settings must also be recorded in an audit log.

The cloud is the layer that integrates multiple devices and time periods. It handles deduplication, long-term retention, version control, permission separation between farms, analytics, reports, and external APIs. It does not treat aggregates calculated at the edge as unquestionable truth, but links them to the raw data and calculation version. If a result recalculated in the cloud differs from the result generated in the field at the time, do not overwrite either value; record the difference and its reason.

The contract between the two layers must be specified in the message schema. Required fields include the identifiers for the farm, barn, device, and sensor; the event ID and sequence number; observation and receipt times; the measured value and unit; quality status; and the calibration and configuration versions. When the schema version changes, establish compatibility rules and a migration period so older devices are not suddenly rejected. Also decide whether unknown fields will be ignored and whether data missing a required field will be quarantined.

Design storage capacity and retention periods with numbers

“A sufficiently large disk” is not a requirement. Minimum capacity must be calculated as the number of sensors × sampling frequency × message size × maximum outage duration, plus index, log, and encryption overhead and a safety margin. Test whether the system can still withstand the maximum outage when more sensors are added or a high-resolution mode is enabled. Operators must be alerted when disk utilization crosses warning and critical thresholds.

Not all data need to be retained permanently, but the deletion order must follow evidentiary value. Raw data linked to safety incidents and carbon claims, calibration and configuration changes, and quality-decision logs have high priority. Debug logs and reproducible caches may have lower priority. Compression and aggregation are permissible, but the policy must state when originals are deleted, the reproducible transformation formula, and who is responsible for retention.

Backups must also be assessed by recoverability rather than by the mere existence of a copy. NIST SP 1339 emphasizes integrating OT backups with change management, creating and testing them regularly, and reviewing them in recovery exercises. It must be possible to restore gateway configurations, certificates, message schemas, models and rules, and device inventories; operational data and secret keys may require different backup methods. Recovery tests should be performed in an isolated environment that does not put the original system at risk.

Why reliability must not be separated from security

An attacker can make the system look disconnected without cutting the internet. They may delete the transmission queue, change the clock, replay old data, or manipulate the gateway configuration. Encrypting data in transit is therefore not enough. The system needs device-specific identities, certificate rotation, signed updates, protection for stored data, least privilege, and audit logs that are immutable or replicated externally.

For integrity checking, a hash of each message batch and a link to the previous batch can be retained. A hash, however, does not mean that the sensor reading accurately reflects reality. It detects changes after a record is created, but cannot correct a value that was wrong from the beginning because of manipulated settings. Calibration, physical seals, field inspections, and data-integrity controls must be operated together.

Implementation checklist

  • Has the maximum expected outage duration and its basis been defined for each farm?

  • Has local storage capacity been calculated from the number of sensors, interval, and message size?

  • Are observation time, edge receipt time, and cloud ingestion time distinguished?

  • Can duplicate, missing, and out-of-order events be detected using event IDs and per-device sequence numbers?

  • Does repeated retransmission leave aggregate results unchanged?

  • Are raw data retained until transmission completion is confirmed?

  • Do deletion priorities and field alerts operate when disk space is low?

  • Are clock synchronization failures and reboot intervals retained as quality states?

  • Have backup and recovery of configurations, certificates, schemas, and data been tested in practice?

  • Are configuration changes and manual actions during an outage also recorded in the audit log?

  • After recovery, are the expected count and the received, duplicate, and corrupted counts reconciled?

  • Does the display prevent cloud non-receipt from being mistaken for sensor non-measurement?

Conclusion: restore the evidence, not merely the connection

Internet outages are not exceptions; they are one of the normal operating conditions of a farm system. A sound Edge–Cloud architecture does not assume an uninterrupted network. It continues collecting during an outage, commits the data safely on site, resends them with the same identifiers, merges them in the server without duplication, and retains gaps and clock errors as quality states.

Recovery is not complete the moment the connection icon turns green. The sequence of events before and after the outage, data counts, device clocks, configuration versions, and integrity must be reconciled before the Carbon Data Evidence Chain is restored. Saying that carbon data remain “alive” does not mean merely that values are stored somewhere. It means that anyone reviewing them later can understand what was observed and how it was transmitted.

Sources