Why you should be using portable zero-touch provisioning on the edge

Being connected is important, but for a dispersed, global, and mobile organization that works with increasingly sensitive data, sometimes it's better to stay disconnected for security, safety, operational, and other technical concerns. This type of deployment is often referred to as air-gapped. Common air-gapped environments include ships, vehicles, aircraft, remote industrial sites, nuclear facilities, and emergency field operations. These are locations where traditional network connectivity is unavailable or undesired, but where critical decision-making, predictive maintenance, resource estimation, safety monitoring, intrusion detection, and anomaly detection are essential.

Challenges of provisioning air-gapped edge environments

To address the challenges of operating in air-gapped environments (including lack of connectivity, limited hardware resources, manual update procedures, and the need for secure, autonomous systems) where agility in deployment and redeployment is critical, a portable edge zero touch provisioning (ZTP) solution can be adopted.

Across various industry use cases, several essential requirements consistently emerge, which can be integrated into the ZTP solution:

• Self-contained: No dependency on cloud API or external data sources
• Long-lived: Must run for weeks or months without updates
• Secure by default: Threat models assume local tampering or sabotage
• Hardware-aware: Limited compute, storage, power, and rugged devices
• Manually updatable: Updates over USB, SD cards, or other physical media
• Network-constrained: Low, intermittent, or no connectivity (GSM, satellite, or fully offline)

Portable edge ZTP architecture pattern

With common characteristics of air-gapped and resource-constrained environments identified, a portable edge ZTP architecture pattern can now be defined:

• Immutable infrastructure
  • Description: The system, including operating system and services, is deployed as read-only, signed images.
  • Benefits: Helps prevent drift and unauthorized changes, ensuring system integrity and security, which is crucial in an isolated environment.
• Hybrid workload:
  • Description: Combine OCI-compliant containers with traditional package-based or virtualized workloads using Image Builder to enable seamless coexistence, with locally-hosted registries or USB transfers for modular, isolated deployment
  • Benefits: Simplifies deployment, scalability, and management of applications, allowing for isolated updates without affecting other components.
• Offline, physical media, and network-isolated updates
  • Description: Updates are delivered by physical media (USB, SD card) or through local network-restricted methods (DHCP/HTTP boot) using formats such as OSTree or container images. These updates can be signed, encrypted, and verified to ensure security and integrity.
  • Benefits: Ensures reliable and efficient updates in isolated or network-restricted systems, ideal for environments with no or limited connectivity.
• Local control plane
  • Description: A lightweight orchestration layer that ensures autonomous operation by monitoring system health, automatically restarting failed processes, and reacting to system triggers (disk full, model failure).
  • Benefits: Simplifies redeployment, ensures system health, and enables autonomous operation without external connectivity.

How to design an air-gapped zero-touch provisioning solution

To address the requirements of air-gapped deployment, you can use a scalable solution based on this general architecture pattern. Key features of this solution include:

• Scalability: Deployable on both physical and virtualized x86_64 end devices, supporting a wide range of deployment environments
• Controlled versions: Includes the ability to check and verify the deployed version, ensuring transparency and traceability about what is currently running
• Isolated networks: Support for controlled, isolated networks using:
  • BIOS-based iPXE + TFTP/HTTP
  • EFI-based iPXE + TFTP/HTTPBOOT
  • ISO installation for devices without terminal access, relying on known MAC addresses
• Future-proof capabilities: Designed with evolution in mind, helping deliver easy maintenance, scalability, and future adaptability. Functional on Red Hat Enterprise Linux (RHEL) 8 and 9, with forward compatibility for RHEL 10.
• Management: Able to manage rollbacks to previous working images and OS versions
• Modular capabilities: Can install third-party drivers, such as hardware accelerators drivers

Embracing container-native OS with RHEL image mode

The architecture leverages container technologies to build and deploy RHEL-based systems.  Building on the foundation of OSTree,  Red Hat introduced RHEL image mode,  which enables the creation of bootable operating system images directly from a Containerfile. These images are stored and managed in a registry, emphasizing immutable system design, registry-backed distribution, fully transactional updates, and support for rollback mechanisms such as greenboot. The bootc command-line tool is the implementation that facilitates this image mode workflow.

Image builder tool creates images from a blueprint, which contains the image definitions.  After generation, the image can be downloaded in multiple formats. 

Image mode offers a simplified deployment experience over the isolated local network, while the anaconda-iso supports devices that lack access to it. An Anaconda-based installer image deploys a bootc container embedded within the image. When a device cannot connect directly to the laptop, the image can be written to a USB drive for a self-contained installation, providing a complete operating system without relying on any external services. 

The self-contained nature of this installation method is important not just for edge devices. It's vital for the portable laptop itself, making it possible to reinstall in isolated environments (like a ship far from port or a temporary disaster zone lacking network connectivity). To help ensure portability, a powerful laptop serves as the central hub for managing edge node deployment. The laptop runs RHEL, ensuring consistency across devices. All laptops run the same image, for homogeneity in services and configurations and device names. Key services in the laptop are:

• DHCP: Provides network boot services and assigns IP addresses during PXE boot. It also directs nodes to the correct boot file for image installation
• TFTP: Distributes system images to nodes, with TFTP handling smaller files. Use UEFI HTTP Boot when possible—it’s simpler and more secure
• HTTP: Distributes kickstart files, installation scripts, and larger image files over HTTP to support automated deployment
• Container registry: Stores the edge container images and application container images that will be deployed to the edge devices
• Podman: Manages container images and ensures consistency across deployments
• Red Hat Ansible Automation: Automates post-configuration tasks and device management

Laptop deployment

The laptop is deployed using an ISO image containing all necessary components. This ensures the laptop can be redeployed independently, even in isolated environments (ships, airplanes, temporary disaster zones, and so on). Red Hat Ansible Automation Platform (AAP) simplifies laptop post-installation, edge devices reinstallation, and edge device status checks.

Edge device image deployment

Similar to the laptop, the edge device images use RHEL, built as container-native images. These images include all necessary packages, such as hardware accelerator drivers, which can be installed from local software repositories. Once images are built, they can be placed into the ISO used for laptop deployment.

During laptop deployment using the ISO, the edge container images are uploaded to the laptop container registry. This registry then serves the OCI-compliant images to the final edge devices during its installation.

After the laptop is deployed, the edge devices images are started using Podman or a systemd service. The podman-autoupdate feature provides flexibility and agility, allowing seamless updates to the container image being served.

Upon starting the edge container image, its integrated web server hosts the OSTree commit as a repository. This enables remote device deployment over network provisioning using DHCP and either BIOS/TFTP or UEFI/HTTP boot. Kickstart configurations automate setup, including assigning fixed IP addresses and hostnames, or fetching application images.

Modern application distribution for edge

Application development typically occurs in an isolated, internet-controlled environment. This allows for downloading necessary dependencies, like libraries. After building, the application is usually packaged as an RPM and distributed within the same lab environment using a software repository as managed by Red Hat Satellite. 

Current deployment practices for edge solutions often involve installing an RPM or directly extracting application files onto standard, non-immutable RHEL systems lacking built-in rollback capabilities. Updates are applied incrementally, and rollbacks are manual and complex.

In the new model proposed, this process is evolving. Applications can still be packaged as an RPM, but instead of being deployed directly, they are now integrated through Image Builder (upstream OSBuilder). Image Builder creates a customized edge image, which can be delivered as a container image or an edge installer, depending on the deployment mode. These images are then further configured using systemd services for startup and lifecycle management.

This approach incorporates containerization, allowing users to package applications as OCI containers. These containers can be managed locally with Podman and systemd, or orchestrated with Red Hat’s build of MicroShift, a lightweight Kubernetes distribution derived from the edge capabilities of Red Hat OpenShift. To improve scalability and ease of management in edge deployments, this flexible approach enables various deployment models, from system services to workloads managed with Helm and Kustomize within MicroShift.

After the initial deployment, the focus shifts to ongoing system and application maintenance, known as Day 2 operations. These activities are important to keep the edge systems running smoothly and securely. In this setup, Day 2 tasks are managed using standalone Ansible playbooks, which are run manually from a laptop. These playbooks automate common tasks such as restarting application services, updating configuration files, and checking the health of the system. This approach helps ensure consistency and reduces the chance of human error during routine maintenance

The network-based provisioning process  largely follows the legacy approach, with adaptations to update OS version files (BOOTX64.EFI, initrd.img, vmlinuz, and so on). While most of the kickstart configurations can be reused, some adjustments are necessary. This approach introduces a new Kickstart bootc option, ostreecontainer, which streamlines the process of container-based OS provisioning, as explained in Master the art of bare metal deployments with image mode for RHEL.

Day 2 management evolution

How are Day 2 activities, such as updates to the operating system, applications, and configurations, to manage effectively in this kind of environment?

The plan is for the laptop to evolve into a MicroShift host running key services including Quay, Red Hat Edge Manager, AAP, and others. Quay operates as the containerized registry that serves the OSTree repository, which is used for both the initial deployment and subsequent upgrades of edge devices.

AAP will be leveraged to handle application management tasks such as restarting services, performing health checks, and enforcing configurations. While Edge Manager (currently in Technical Preview), is designed to provide a scalable, and secure way to manage both edge devices and their applications. 

Together, these components create a centralized, containerized management plane within the air-gapped environment.

Automated zero-touch deployment

Portable edge zero-touch provisioning architecture is a good way to get your edge devices up and running at scale, even if they're completely cut off from the internet. 

To be clear, this automates everything, so there's no more manual setup! By using immutable systems, container-native updates, and local orchestration, this architecture makes sure your edge devices are provisioned reliably, even with zero outside network access. This not only makes deployments smoother, it also boosts security and keeps things consistent across all your distributed edge setups. With this architecture, your edge devices will be humming along, no matter where they are. To learn more about edge computing, check out Hatville - the miniature city where edge computing comes to life.