A guide on how Uffizzi developed per-feature ephemeral test environments with simulated hardware for embedded systems to help them iterate faster, cut-down operational costs, and reduce the time to market.
During the last few Open Source Summit & Embedded Systems conferences, our team at Uffizzi met with several embedded system teams. We heard from them about the challenges they face in testing and verifying their designs in a way that is fast, saves costs, and allows fast releases/feedback.
We realized as the world becomes more and more dependent upon the creative applications of embedded systems, the need to be innovative, incorporate feedback, move fast, and do all that at a reduced total cost is essential. However, optimizing innovation with speed and cost widens the room for error. Very often, embedded systems require very high reliability, particularly for regulated industries where malfunctioning products can have serious consequences. To ship features with speed and minimal — ideally zero — errors requires feedback, integration, testing, and delivery to be continuous.
But, during our research, we learned that achieving this state of scalable and efficient continuous integration, testing, and delivery is challenging for embedded systems due to several reasons, most of which were persistent across projects. Listing some of the most notable ones:
These engineering challenges introduce the below operational challenges:
To alleviate the above-mentioned challenges faced by embedded systems, we ventured into designing a cheap and operative ephemeral environment solution for OpenTitan, an open-source, silicon, Root of Trust project, administered by the lowRISC foundation. The OpenTitan project develops both hardware and software/firmware, and so our approach to designing this solution was to create fully-configured, ephemeral environments, where each feature change — software or hardware change— would have its isolated environment with the SW built and HW simulators configured to simulate the OpenTitan device, and where that feature can be tested.
With these goals in mind, we built an ephemeral, simulated-hardware environment prototype solution for OpenTitan. We ran the same build and test mechanisms locally on our systems and compared the results with lifting the workload to an ephemeral test environment running on Cloud via Uffizzi. We are listing notable features of these ephemeral environments and the value they added to embedded systems development:
Summarizing these features, here is what our ephemeral environments built for embedded systems offer: integrated into GitOps practices, self-contained, cheap and fast, consistent, fully-configured, purpose-oriented, isolated, resource-optimized, scalable, ready-to-use, and shortens release cycles.
Let’s dive deeper into understanding how we built an ephemeral test environment for OpenTitan and how the environments work. Similar steps can be followed for configuring ephemeral test environments for other embedded system projects.
Step 1: Forking OpenTitan
We started with forking OpenTitan’s GitHub repo. You can find the fork here to follow along. As mentioned previously, ephemeral test environments for OpenTitan are added into OpenTitan’s CI/CD pipelines on GitHub so the process of spinning up a new environment for feature-testing and delivery is done continuously per PR.
Step 2: Containerizing OpenTitan
We containerized OpenTitan so it is self-contained and can easily be orchestrated. Since OpenTitan builds both SW and HW, several dependencies must be installed to compile the SW written in C and simulate the HW using Verilator, a SystemVerilog simulator. Containerizing the application ensures that all the needed dependencies are installed and that there are no outdated/conflicting dependencies. Building the SW without containerization would have introduced major bottlenecks because of the strict dependencies between tools, adding delays to the release lifecycle.
OpenTitan comes with an experimental Dockerfile, which had mostly all of the tools that were needed to run the OpenTitan ephemeral test environment with simulated HW. We based the Dockerfile used by Uffizzi (Dockerfile.uffizzi) on OpenTitan’s experimental Dockerfile and added the ttyd dependency to it - ttyd allows terminal sharing over the web. With this dependency added, developers will have a terminal interface to the OpenTitan environment, where they can build the SW and run on-host and on-device tests.
Step 3: Orchestrating the container
Created a docker-compose.uffizzi.yml file to orchestrate the OpenTitan container. This docker-compose file is where we add the Uffizzi integration, which will spin up the container using the defined image, and either deploy a new ephemeral environment on Uffizzi cloud or update an existing one. More on this in the next step.
This docker-compose defines an entrypoint-port into the terminal shared over the web through ttyd. We also used this compose to mount the OpenTitan project onto the container, the interface to the project is then provided by ttyd. Furthermore, this file also defines the resources (memory and CPU) needed by this container. Through trial and error, we found the optimal resource requirement for running OpenTitan ephemeral environment to be 4 vCPU and 16 GiBs.
Please note, this prototype requires more memory and CPU than the Uffizzi Platform defaults. Contact us if you’d like to request more resources for your project.
docker-compose.uffizzi.yml with entrypoint into OpenTitan
Step 4: Defining GitHub workflows to provision the ephemeral environments
The power of ephemeral environments lies within the realms of CI/CD. So Uffizzi leveraged GitHub actions to orchestrate building the ephemeral environments, collecting feedback, updating the environments as the feature gets iterated on, and finally deleting the environment if it is no longer required.
We built a 2-stage GitHub workflow, to extend GHA-triggered ephemeral environments support for open-source projects. The 2 stages of this workflow are:
Since embedded system projects heavily rely on HW simulators, especially early on in the development cycle, our focus with the OpenTitan ephemeral environment was to run the simulation effectively while optimizing resource consumption because running HW simulators can easily hike the resources consumption and cost. The uffizzi-preview.yml action helped achieve this by deleting the environment and freeing the resources as the PR is closed.
Step 5: Triggering environment creation
After adding the above 2 GitHub workflows in our fork of OpenTitan (if you’re following this example for your project, make sure the actions are added to the default branch of your repo, else the 2-stage workflow will not work as desired.) we opened a PR in this fork. This triggered the uffizzi-build.yml, which then triggered the uffizzi-preview.yml workflow and created a new environment, ready to run tests against, interfaced through a terminal shared over the web.
Considering we leverage Docker to build our image and also use docker-layers caching from GitHub actions, the first run of the uffizzi-build action with the uffizzi-preview action took around 10 minutes. However, subsequent updates to this feature branch were 4X times faster, and the two actions combined took around 3-4.5. We ran the actions a couple of times to ensure this behavior was consistent—check here. This is significant because reviewing features and receiving feedback in isolation, parallelly and continuously, as they are being developed, is scalable and allows faster iteration, hence faster releases.
Step 6: Building OpenTitan SW and running on-device and on-host tests on the environment
Once the environment is created successfully, a comment posted on the PR will give us a link to the ephemeral environment. Visiting the environment, we see a terminal with /home/dev/src as the entrypoint. This is where the OpenTitan application (both SW and HW) is mounted. Here are the components in this directory:
The code related to building and testing the SW and the HW are in the sw and the hw directories, respectively. There are three major parts to the OpenTitan software stack:
Building of the SW is handled by Bazel. The device software can be built for multiple OpenTitan devices and memories. Running tests on multiple OpenTitan devices (whether DV or Verilator simulation or an FPGA) which involves building several software images for multiple memories, building the necessary SW for the target device (FPGA/Verilator), etc was taken care of by Bazel.
Like many embedded systems, OpenTitan runs 2 categories of tests: on-host and on-device. On-host tests are compiled and run on the host machine, so these will be interacting directly with the container; these are hardware-independent tests and perform unit tests for device software. On-device tests are compiled and run on (simulated/emulated) OpenTitan hardware and so they’ll interact directly with the simulated OpenTitan hardware. The ephemeral test environment passed both these test suites successfully.
To run both the test-suites, we used Bazel, which built the OpenTitan SW for our target HW (Verilator, in this case). The on-device tests could take longer, so we utilized screen-sessions to let the tests run even when SSH disconnects.
The on-host test suite does not require HW simulation and so these are faster. The suite passed successfully on our ephemeral environment and took 1–2 seconds. The on-device tests, however, require HW simulation and take more time. According to the official OpenTitan docs, Bazel can take an hour to build the simulation. The ephemeral environment, with optimal resource planning, took between 35–40 minutes for the first build and subsequent tests took around 1–8 minutes depending upon the kind of test that was running. This was remarkably faster than the time it’d typically take to run on-device tests, as OpenTitan suggests. The ephemeral environment reduced that time by maximizing resource efficiency.
Below are the results of running on-device, sensor_ctrl_alert_test with optimized resources successfully on the ephemeral test environment. This test took 43 minutes in total, 35 minutes to build the HW, where the test would run and 8 minutes to build the sw this particular test was written for. This is at least 15 minutes faster than the time it’d take to run on-device tests the first time. Subsequent tests on this environment were faster.
As described by the OpenTitan official docs, the HW simulation takes an hour to build, add to that the time it’d take to build the sw. Our ephemeral environment solution significantly optimized the run time for on-device tests.
As can be seen, the power of containerized ephemeral environments came in very handy—we were able to set the optimal amount of resources needed, depending upon the type of test that we’d run in the environment, without running the risk of over or under-provisioning. The environments had dedicated resources, which prevented throttling.
Step 7: Running Comparative Analysis and Summarizing Performance
After comparative analysis, here are some important points we noted while running OpenTitan SW on simulated OpenTitan HW in Uffizzi ephemeral environments and running it locally:
We ran embedded systems without ephemeral environments to understand the impact and the value Uffizzi provisioned ephemeral test environments for embedded systems offer. As we saw, adding simulated ephemeral test environments into OpenTitan’s CI/CD practices not only reduced the pain encountered during embedded project development, right from planning to release and gathering feedback but also added capabilities to quicken turnaround times. Speed, resource and cost optimization, ease of maintenance and system updates, shorter development and release cycles, and continuous testing throughout development and release cycles are some of the most significant achieved benefits of developing embedded systems with ephemeral environments.
These environments will help reach the needed state of scalable and efficient continuous integration, testing, and delivery which has traditionally been hard for embedded technologies. The ephemeral test environment approach to developing, testing, and releasing will accelerate the time to market, keep customers happy by engaging with their feedback and reduce costs. If you, too, work with embedded systems and would like to reduce your time to market, adopt agile practices, cut down your operational costs, and improve product quality with early feedback, check Uffizzi out, and connect with us here.