Planning Reliable OTA Updates for Embedded Linux Devices
A readiness checklist and failure-test matrix for connected products: plan trusted releases, recoverable installation and controlled fleet rollouts before devices leave the workshop.
Plan for the update that fails
A connected device needs to remain maintainable long after installation. An interrupted update at a remote site can turn a routine software release into a service visit, lost telemetry or an unavailable product. Over-the-air (OTA) updates therefore need a defined failure path as well as a successful download path.
For a technical founder, engineering lead or product owner, the useful planning question is: what will this device do if the next step fails? Work through the lifecycle below, use the matrix to specify evidence, and take the readiness checklist into your architecture review. AWS’s IoT Lens similarly treats recovery as part of update design, rather than an operational afterthought.
Define the complete update lifecycle
Start by assigning an owner and a durable device state to each transition. A completed transfer is not a completed update; the release must run acceptably and its outcome must reach the people operating the fleet.
- Release creation: build an identifiable artifact, record its source revision and dependencies, and define supported hardware and upgrade paths. Keep the released artifact immutable.
- Authorization: approve the release and select eligible devices. Separate permission to distribute a release from the device’s decision to accept it.
- Download: define retry limits, bandwidth budgets and behavior when connectivity disappears. Retain enough state to resume safely or restart deliberately.
- Verification: check trusted signatures, payload integrity, hardware compatibility and release policy before accepting the candidate for use.
- Installation: check resources, write the intended target and record progress. Specify how interruption at each write boundary is handled.
- Reboot and health confirmation: start the candidate, check essential product functions and confirm it only after those checks pass within the agreed window.
- Recovery and reporting: select an allowed recovery path on failure, retain diagnostic evidence and report both the attempted release and the version actually running.
An application update may replace a service or container while retaining the operating system. Plan its dependency contract, restart behavior and data compatibility. A whole-system update may replace the kernel and root filesystem together, so boot selection and recovery become central. State explicitly whether bootloader, modem or other peripheral firmware is included; each adds its own failure boundary. Neither approach removes the need to maintain the layers it leaves untouched.
Write down constraints before selecting tools
Create a device-family inventory before choosing an updater. A solution that fits a development board can be unsuitable for a production variant with less storage, a different boot chain or no accessible recovery connector.
- Hardware and boot: list board revisions, storage layouts, peripheral firmware and bootloader capabilities. Establish how the device identifies its own variant and which components can be updated independently.
- Storage and data: budget for installed images, download staging, temporary files, logs and migration backups. Identify credentials, calibration, configuration and application data that must survive updates.
- Power and availability: record likely power interruptions and permitted service downtime. Decide when installation and reboot are acceptable for the product’s actual workload.
- Connectivity: document intermittent links, metered connections, transfer limits and the longest expected offline interval. Agree on retry backoff and a way to refresh expired download authorization.
- Recovery and lifetime: identify who can physically reach the device, available rescue interfaces and how long security fixes, signing infrastructure and compatible upgrade paths must be maintained.
RAUC and SWUpdate are candidates to evaluate against these constraints. RAUC documents redundant-slot and recovery layouts; SWUpdate documents installation handlers and configurable hardware checks. Neither tool choice settles your partition map, health criteria or support process. Pin the versions used by the product and review their matching documentation during implementation.
For a closer look at which framework fits these constraints, our RAUC vs SWUpdate comparison covers A/B systems, signing, build-system integration and custom installation workflows.
Establish trust and compatibility
Transport Layer Security (TLS) protects a network connection. Artifact verification answers a separate question: was this release authorized by a signer the device trusts, and has the protected content changed? A checksum alone does not establish authenticity because an attacker who replaces the payload may also replace an untrusted checksum. SWUpdate’s signed-image design combines a signed description with hashes of its payloads.
Make signature enforcement part of the production configuration. Reject unknown signers and damaged payloads; test those rejection paths. Match the signed release metadata to an authoritative device identity and hardware revision, rather than trusting a filename or a cloud group alone. SWUpdate’s hardware compatibility check must be configured, which illustrates why a feature’s availability is different from its enforcement in your product.
Plan signing-key custody, release approval, rotation and compromise response. Keep private signing keys off devices and separate development trust from production trust. Define how devices receive replacement trust material before an old key is retired, including units that miss several releases. RAUC documents certificate-based bundle verification and separate development and release trust arrangements.
Device credentials used to authenticate to the cloud are a separate lifecycle. Assign an owner for provisioning, renewal and revocation. Include expired credentials and an incorrect device clock in integration planning: a device that cannot authenticate may be unable to retrieve the update intended to repair it.
Make installation and recovery a joint design
A/B partitioning keeps two system slots: run from one while preparing the other, then try the candidate on reboot. It can preserve a previous system for fallback, but costs storage and requires coordinated image, partition and bootloader integration. A dedicated recovery system is another possible layout, with different service-availability and rescue requirements.
Automatic fallback requires reliable active-slot detection, durable boot-selection state and bounded attempts to boot the candidate. A hang also needs a mechanism, such as a correctly integrated watchdog, to return control to the boot process. RAUC’s integration documentation covers bootloader coordination and marking a boot successful. Your product must decide when that success signal is justified.
Define health in terms of essential functions: can the service load its configuration, access required peripherals and process a representative operation? Distinguish a local software fault from an unavailable external network. A running process alone is weak evidence; requiring a permanently reachable cloud endpoint can also create needless fallback during a network outage.
Keep recovery compatible with downgrade policy
Recovery to an allowed known-good release and anti-rollback are different requirements. Anti-rollback prevents returning to a disallowed, potentially vulnerable version. An authentic signature does not make an old release acceptable, and secure boot alone does not establish a version-based minimum. U-Boot’s capsule-update documentation, for example, describes an explicit lowest-supported-version check.
Define where version policy is enforced and how its state is protected against reset or tampering. Before raising the minimum allowed version, ensure the recovery image remains permitted and usable. Review the ordering of confirmation, policy changes and recovery preparation so an interrupted transition does not remove the only viable fallback.
Software rollback does not undo a data migration
A previous root filesystem may still boot while being unable to read a shared database changed by the candidate. Choose backward-compatible schema changes, versioned data copies or a tested restore strategy. Define how to preserve records created after the update. RAUC discusses shared versus redundant data storage, but application-specific migration remains an integration responsibility.
Hypothetical industrial Linux gateway: a candidate boots and converts its local telemetry queue, then fails a sensor health check. Returning to the old slot only helps if the old application can still read the queue or recover an acceptable copy. This is an illustrative design scenario, not an Epic Microdev client project or completed hardware test.
Control the rollout and observe the outcome
Separate fleet orchestration from device-side installation. AWS IoT Jobs distributes remote operations to selected devices; its job configuration supports rollout rates, abort conditions and execution timeouts. A device agent still needs to interpret the job and coordinate verification, installation, health confirmation and recovery. Jobs alone is not a complete safe-update implementation.
As a planning approach, begin with laboratory devices, then a small representative group, then progressively wider cohorts. Include relevant hardware and connectivity variants in early groups. Set an observation window and measurable stop conditions before release: unexpected recovery, failed health confirmation or lost essential functionality should have an assigned decision-maker. Derive thresholds from your product’s risk and operating baseline.
Stopping new assignments does not necessarily stop a device already writing an image. Specify which local phases may be cancelled and which must finish or recover. Keep a version inventory that distinguishes desired, downloaded, installed and confirmed-running releases, together with hardware revision and last-seen time.
For actionable reports, retain the release identifier, update phase, error reason, retry count, active slot, health result and recovery outcome. Queue reports locally when offline. On reconnection, reconcile the current state with current rollout policy before retrying stale work. A device absent for months may need renewed credentials, intermediate upgrades or a supported migration path; silence is not evidence of success.
Failure-test matrix
Use this proposed matrix to agree acceptance criteria before implementation. The intended behaviors are design targets, not guarantees or tests performed by Epic Microdev. Adapt them to the boot chain, storage medium and updater configuration, then exercise interruption points repeatedly on representative hardware. A/B layouts alone do not prove power-loss safety or uninterrupted service.
On smaller screens, scroll horizontally to read all three columns.
| Failure scenario | Intended safe behavior | Evidence to collect |
|---|---|---|
| Interrupted download | Disconnect the network at several transfer points. Keep the current release usable; resume or restart within a bounded retry policy, then verify the complete candidate. | Transfer offsets, retry timing, bytes retransmitted, verification outcome and running version. |
| Power loss during installation | Cut power during writes and boot-state changes. For the selected layout, boot a confirmed slot or enter the defined recovery environment; do not select an incomplete candidate. | Power-cut timing, serial boot log, slot contents, persistent boot state and recovery outcome. |
| Corrupt or incorrectly signed image | Alter payload bytes and try an untrusted signer separately. Reject the candidate; preserve the working system and avoid booting unverified content. | Artifact identifier, verification error, signer identity and unchanged boot selection. |
| Wrong hardware variant | Offer a validly signed image for another board revision. Reject it before incompatible payloads are installed. | Reported board identity, signed compatibility metadata, rejection reason and storage state. |
| Failed boot or application health check | Force a boot hang and a separate application failure. Exhaust bounded attempts and recover to an allowed release without prematurely confirming success. | Attempt counters, watchdog resets, health-check results, selected slot and time to recovery. |
| Insufficient storage | Fill staging, data or log storage, and test an oversized image. Reject or defer when required space is unavailable; preserve the running system. | Free-space checks, required capacity, error code and temporary-file cleanup. |
| Incompatible persistent-data migration | Apply a schema change the previous app cannot read, then trigger fallback. Use the agreed compatibility or restore path; block activation if no acceptable path exists. | Schema versions, migration journal, restore result, data integrity and any lost or replayed records. |
| Device reconnecting after a long offline period | Reconnect with old software and expired download authorization. Reconcile release eligibility, renew access through the supported flow and follow a compatible upgrade path. | Last-seen and current versions, credential state, job reconciliation and migration sequence. |
| Attempted downgrade to a disallowed release | Offer an older correctly signed release through supported update and recovery paths. Reject it under the version policy while retaining an allowed recovery option. | Minimum-version state, policy decision, enforcement point and recovery boot result. |
Readiness checklist for your product
Answer each question with a design decision, an owner and a way to verify it. An unanswered item is an integration task to schedule before widening a rollout.
Discuss the architecture and integration requirements
Bring your board variants, bootloader and partition layout, current build process, connectivity limits and recovery requirements to an engineering discussion. The checklist and matrix can help turn those inputs into a scoped integration plan with explicit acceptance criteria.
Epic Microdev’s relevant service areas span device firmware, embedded Linux, connectivity and cloud integration. Discuss which parts your team already owns and where you need engineering support.
References
Official documentation consulted on 28 September 2026. These references explain individual mechanisms; validate the combined behavior against your product’s hardware and the software versions you ship.
- AWS IoT Lens: over-the-air update design
- AWS IoT Jobs: remote operations
- AWS IoT Jobs: rollout, abort and timeout configuration
- RAUC: bootloader and system integration
- RAUC: redundant systems and recovery scenarios
- RAUC: signing trust, persistent data and migration
- SWUpdate: installation and hardware compatibility
- SWUpdate: verifying signed update images
- U-Boot: explicit anti-rollback policy for capsule updates