February 3, 2023
min read

Best Practices for Your Ephemeral Environment

Learn how to use ephemeral environments to improve code delivery and efficiency with best practices and free open-source tools.

In software development, ephemeral environments improve code delivery and efficiency by removing the resource contention and maintenance imposed by persistent test environments.

Figure 1: Persistent vs. ephemeral environments

This article will describe the best practices for creating ephemeral environments and introduce a free, open-source project and a hosted service that can help you implement them faster and at a lower cost.

Summary of best practices for creating ephemeral environments

Best Practice Description
Containerize your code Containerization simplifies rapid code deployment by including the application runtime dependencies such as binaries, configuration files, and libraries.
Leverage Docker Compose or equivalent Compose defines a multi-container application in a YAML file and triggers its installation with a single command.
Manage secrets safely Store login credentials, tokens, and keys separately from your code repositories, and inject them during runtime using variable substitution.
Integrate ephemeral environments into CI/CD workflows Create ephemeral environments as part of standard workflows, such as pull requests.
Seed your application databases A SQL dump file in object storage (like AWS S3) can be loaded as a mounted volume during Docker Compose initialization, or your application can generate dummy data on startup.
Collaborate across teams using ephemeral environments Capture the value of ephemeral environments by having a collaboration plan among testing, DevOps, development, product management, and end-users.
Remove ephemeral environments once the feature branch is merged Ephemeral environments are intended to be temporary. Removing them is relatively simple once all of the previous measures described in this table have been implemented.
Use an EaaS service or open-source project to streamline the process Uffizzi offers a free, open-source solution and a hosted service designed to create and remove ephemeral environments.

Some historical context for ephemeral environments

Ephemeral environments are a familiar concept to modern users of static website hosting platforms such as Netlify and Vercel. Website designers use ephemeral environments to preview changes before deploying their code to the live website.

Creating previews for static code is relatively straightforward.

However, web-based applications and e-commerce sites are too complex to preview with similar technology. These applications rely on complex backend code, third-party APIs, microservices, and databases, which means that the ephemeral environment must include the full application stack, not only the frontend code.

Large software as a service (SaaS) vendors have been creating full-stack ephemeral environments for many years to test new features and bug fixes before merging the code branches. However, setting up and maintaining such platforms is expensive - most often requiring a dedicated team.

New free or open-source projects and hosted environment as a service (EaaS) platforms have emerged recently that bring the technology for creating ephemeral environments within the reach of everyday Dev teams of all sizes.

Explanations of ephemeral environment best practices

In the following sections, we’ll elaborate on each best practice summarized above. As part of the last best practice, we’ll introduce a free, open-source tool and a hosted service designed to create and manage ephemeral environments integrated into a software delivery pipeline.

Containerize your code

The average developer associates containers with microservices and the Kubernetes container orchestration platform. However, taking advantage of the basic benefit of a container doesn’t require refactoring an application into microservices or orchestrating containers.

When it comes to helping create ephemeral environments, containerization ensures that the application environment includes all the required binaries, scripts, libraries, configuration files, environment variables, and OS patches and can run on a standard container runtime.

Popular container runtime environments include containerd, cri–o, and Docker. The last of these provides simple steps for creating an application image and containerizing code. Using these instructions, you would create container images for the various software components of your application and place them in a container registry.

Store your code on a hosted, Git-based version control system

It may seem redundant to mention the need for checking your code into a Git repository since it seems so commonplace these days. Still, we include it as a best practice since Git repositories have gone beyond version control and offer continuous integration (CI) functionality that simplifies integration with third-party tools and collaboration across teams. Checking your application code into a service like GitHub is a great starting point for planning ephemeral environments.

For example, GitHub Actions can streamline the workflows for creating new ephemeral environments upon issuing new pull requests and deleting those environments upon merging the code branch.

Using a hosted service such as GitHub can also significantly shorten the time it takes to integrate with a third-party environment as a service provider if you choose to pursue that route in the future.

Leverage Docker Compose or equivalent

Docker Compose takes the logistical complexity out of running a multi-container application, which is key when spinning up and tearing down complex ephemeral environments.  Compose files are widely used and can be found in 800k+ Github repos.

With Compose, you define the application environment in a YAML configuration file and take advantage of commands to start and stop each service, view the status of the running services, and organize log files. The Compose getting started page is well organized and includes a simple example to familiarize newcomers with the technology.

Environments as a Service (EaaS)
Release Hub
Lightweight and Fast (All-Container Solution)
Easy Setup
(Based on Docker Compose)
Reusable Github Actions Workflow

See Comparison Table
Lightweight and Fast (All-Container Solution)
Easy Setup
(Based on Docker Compose)
Reusable Github Actions Workflow
Release Hub
See Comparison Table

Manage secrets safely

If you operate your application with a small team, managing secrets may not be a commonplace discussion topic. As you prepare to create your first ephemeral environment by integrating multiple tools and expanding the scope of users, it’s a good time to take inventory of your secrets.

Secrets include passwords, keys (used for encryption, SSH, and third-party API access), database credentials, tokens, and certificates such as the ones used with TLS and SSL.

Secrets can be stored in secret management services provided by CI/CD vendors, such as GitHub Actions, and injected as environment variables.

In the Docker Compose template example below, the Postgres credentials “POSTGRES_USER” and “POSTGRES_PASSWORD” are referenced using variable substitution through Docker Compose commands.

Figure 2: Docker Compose that includes Postgres secrets via variable substitution

Integrate ephemeral environments into CI/CD workflows

Ephemeral environments are most valuable when created automatically each time a developer submits a pull request on a new feature branch. That’s when the process is so efficient that it feels almost magical. Once created, the ephemeral environment will be accessible via an assigned URL that will remain active until the pull request is approved and merged with the main branch, at which time the environment automatically disappears.

If you are using GitHub Actions, CircleCI, or GitLab, you can create a final step in your continuous integration pipeline by triggering a workflow designed to deploy an ephemeral environment. Using variables in your Docker compose file allows building and deploying ephemeral environments automatically as part of your CI workflow. For example, GitHub Actions supports the concept of “outputs,” which you can use to pass tags associated with the appropriate container images and use them during the build process with Docker compose. This page of instructions from Uffizzi’s documentation explains the steps in detail.  

Seed your application databases

Full-stack ephemeral environments are complex because the applications are stateful and require server-side resources. The engineers engaged in functional application testing rely on these backend components to populate user interface elements, such as dashboards and reports, with real data.

Seeding the application database with data exported from a production environment, sometimes referred to as a SQL dump file, is a common technique to make ephemeral testing environments as realistic as possible for functional testing.  

Services like GitHub limit the size of the files stored in the code repositories but offer services for storing large files that are ideal for storing a SQL dump file. These files can also be stored in cloud storage services like AWS S3 and downloaded on database startup.

Database vendors like Postgres, MariaDB, and MySQL provide Docker container images, including functionality for populating a database upon startup. Essentially, the database vendors include scripts with the Docker container images to automatically execute *.sql, *sql.gz, and *.sh files stored in the /docker-entrypoint-initdb.d directory. As an example, you can see the Postgres scripts here.

See how Spotify packs more into every release with ephemeral environments
Read Case Study

Collaborate across teams using ephemeral environments

The best time to inform a developer that a new feature needs improvements or contains a bug is while the code is fresh on that developer’s mind, not days or weeks after the fact. This rapid feedback cycle is one of the key benefits provided by ephemeral environments.

The feedback loop should involve the developer, coding peer reviewer, product manager, QA engineer, and, if available, friendly end-users who can provide timely feedback.

Figure 3: The cross-departmental feedback loop enabled by ephemeral environments

Those implementing ephemeral environments must invest time in planning for the cross-departmental collaboration that will unleash their value. Some of the ingredients for successful collaboration across teams include the following:

  • Automatically inserting the ephemeral environment URL link (via API) in the feature or bug ticket related to the code branch for easy access
  • Creating a temporary chat room (e.g., a Slack channel) for each environment where collaborators can directly exchange messages
  • Enlisting the help of product managers to coordinate the feedback involving test engineers and friendly end-users
  • Taking advantage of the developer having the code top of mind to make small refinements before a major context switch

Keep in mind that the value of these environments is highly dependent on the quality of the data bootstrapped into them. It’s difficult for peers, QA, and users to comment on functionality if there’s no data, or only extremely simple data, in the environment.

Remove ephemeral environments once the feature branch is merged

What makes an environment ephemeral is that it goes away once the feature or bug-fix branch associated with it has been merged. This action should be automated as much as the creation of the environment to avoid wasting computing resources.

A continuous integration event triggers the workflow that creates the environment upon a developer submitting a pull request. A second event triggers a workflow that removes the environment upon merging the code branch.

Use an EaaS service or open-source project to streamline the process

Designing and implementing an ephemeral environment and integrating it with an existing continuous integration pipeline requires two or three engineers and takes a few months to complete. Traditionally, only larger engineering organizations can afford to make such an investment of resources.

Open-source projects like Uffizzi significantly reduce the necessary time and resources by providing the code, know-how, and blueprint for execution.

For those with even less time, Uffizzi also offers a hosted version to get teams started with the help of customer support. As the first step, you can try the live demo, which doesn’t even require filling out a form or creating a free account. It showcases the concepts covered in this article using a sample project hosted on GitHub to give you a taste of creating live ephemeral environments in only a few minutes.

Last thoughts on ephemeral environments

Ephemeral environments increase engineering efficiency by creating a temporary, isolated, full-stack test environment for every feature branch and shortening the time it takes to release new features and fix bugs. An engineering practice that was once only affordable for large engineering teams is now available to smaller teams with the help of open-source tooling and hosted services specializing in ephemeral environments.

Uffizzi logo
Environments as a Service
Learn More
preview icon
Empower your devs with an environment for every pull request
time icon
Avoid release delays that cost millions in missed revenue
Velocity icon
Improve development velocity at scale by up to 50%
Lifecycle icon
Plan capacity in real time with lifecycle management built in
Learn More