Internal developer platforms (IDPs) provide automated self-service solutions for developers to simplify and standardize software practices, infrastructure, environments, and operations. IDPs are configured by operations teams and utilized by developers and testers. Operations teams define resource provisioning criteria for specific environments or requests and establish standardized application configuration templates along with permission controls. This automation streamlines recurrent tasks like environment and resource deployments, enhancing maintenance efficiency through enforced standards or what some like to call “Access with Guardrails.”
The objective of an IDP is to improve the developer experience by reducing the cognitive load required to deploy an application. Developer teams, in turn, achieve autonomy by modifying configurations, deploying, provisioning environments, and executing rollbacks. IDPs can be either developed in-house or procured from external sources.
This article dives deep into the concepts behind internal developer platforms, showing how they are an extension of the DevOps evolution and what benefits IDPs bring to various engineering teams. We also review the core components of IDP and share examples of IDP tools and processes.
An internal developer platform (IDP) is a comprehensive, integrated set of tools, services, and infrastructure purpose-built to streamline and accelerate software development and operations within an organization. At its core, an IDP acts as a centralized hub that offers a wide range of resources and automation capabilities to support various stages of the software development lifecycle.
In the context of platform engineering, IDPs serve as a foundational layer for managing and provisioning resources, including infrastructure, containers, and services. Platform engineers leverage IDPs to design and maintain the underlying infrastructure that powers the development and deployment pipelines. This alignment with platform engineering ensures that development teams have access to a stable set of foundational resources for their projects, enabling faster development cycles and improved overall system resilience.
In summary, an IDP is a key enabler of modern software development and operations, streamlining workflows, providing resources, and fostering collaboration. Its close relationship with platform engineering ensures a robust system for fast, efficient, and scalable software delivery.
With the rise of DevOps, developers suddenly had to deal with complex cloud-native tools just to perform basic tasks like updating environment settings or troubleshooting database setups; this sometimes involved more than 10 different tools. Although DevOps aimed to give developers more control, it ended up increasing their cognitive workload levels, negatively impacting their experience and productivity. The idea that every developer was also a DevOps engineer is a concept that does not scale.
It has become evident that engineering organizations investing in internal developer platforms (IDPs) consistently outperformed the majority of organizations in terms of DORA metrics, including lead time, deployment frequency, change failure rate, and MTTR. This highlighted a successful progression in the DevOps transformation journey compared to those who encountered challenges along the way. The stories by Netflix and AirBnB are among the brightest examples of such transformations.
IDPs serve as a versatile toolset catering to the specific needs of platform engineers, operations teams, and DevOps practitioners. To illustrate how the workflow of infrastructure teams is being changed with IDP, we’ll review a use case here and then reference it further in the rest of the article.
Before the advent of IDPs, infrastructure teams constructed CI/CD pipelines using tools like Jenkins, GitLab CI, and Spinnaker and also configured monitoring and alerting systems. However, this approach often resulted in a lack of standardization and a proliferation of numerous static scripts throughout different teams. Additionally, infrastructure personnel had to allocate significant time to knowledge sharing, developer education, and documentation maintenance.
IDPs address these challenges by serving as a unified interface that enables developers to access and standardize IDP capabilities. One common implementation of such an interface is known as a software catalog. We’ll now examine one of the most widely used software catalog implementations: Spotify’s Backstage.
According to its official documentation, Backstage is an open platform for building developer portals. It is organized as a centralized software catalog combined with a set of plugins, which serve as extensible add-ons implementing certain features and integrations.
The software catalog is a core feature of the Backstage platform, which offers a centralized system to track access privileges and metadata for all the software components in the development ecosystem, such as services, websites, libraries, data pipelines.
This feature naturally enhances collaboration, acting sort of like a service discovery on steroids. The platform team can register all the capabilities it provides to development teams, and they can be discovered on their own.
By default, the software catalog has the ability to load skeletons of code, template in some variables, and then publish the template to some locations like GitHub or GitLab. The platform team can prepare typical software project templates, for example:
The template will contain a project skeleton and all the configurations needed for CI/CD, monitoring, and additional infrastructure dependencies. Instead of tailoring these kinds of projects for each team or each project, the platform team can provide a self-service type of product.
Examples provided by Backstage can be found here. We will also review a few examples in the following sections.
Gartner introduced the concept of a citizen developer in its analyst reports, a term that refers to an employee capable of creating application functionalities for personal or collective use via tools that aren’t explicitly restricted by IT or business units. It’s important to note that a citizen developer represents a persona, not a formal title or designated role.
While this concept may seem enticing, real-world challenges often arise. These include the absence of developer self-service, extended wait times for operations to perform routine tasks (e.g., provisioning databases or setting up environments), and the expectation that developers must comprehensively grasp complex toolchains to avoid reliance on operations.
These challenges impose additional cognitive load and divert developers from their coding tasks, sometimes leading to shadow operations. The result is a diminished overall developer experience, characterized by a lack of documentation, frequent context switching, and a limited understanding of underlying technologies and tools.
Consider a scenario based on the use case described in the previous section. Suppose that a developer needs to deploy a Spring Boot application with additional dependencies, such as MySQL, Redis for persistent data storage, and a Kafka cluster for messaging. In the pre-DevOps era, the developer would have to request this setup from the operations team, effectively facing a roadblock until the environment was provisioned.
With the emergence of DevOps, developers gained access to these tools to set up infrastructure using Terraform, install extra dependencies within Kubernetes using Helm, package everything into Docker images, and build Jenkins pipelines for testing and deployment. However, this approach introduces significant overhead on the development side, requiring the acquisition of new skills and often doubling the effort—what we refer to as increased cognitive load.
Enter the era of IDP, where developers are provided with an additional layer of abstractions. They can utilize a software catalog such as Backstage to select preconfigured templates like the Spring Boot Template and indicate their dependencies using YAML specifications. Furthermore, they can test this setup using environment as a service (EaaS) technology before deploying it to production.
Typically, IDPs consist of five core components, each serving a critical role in streamlining software development and operations. Let’s examine these components in detail.
Managing configuration files—often in scripts or using YAML—is complex, particularly for versioning and distinguishing environment-specific elements. Current setups often lack developer self-service, necessitating DevOps team involvement in environment setup (an area that is not its primary focus).
An IDP efficiently handles resources residing within your container orchestration platform (e.g., Kubernetes) and those external to it, such as databases and file storage, which are essential for modern applications. Moreover, a well-designed IDP should dynamically manage both internal and external resources, adapting them contextually, like configuring specific database types based on the environment’s requirements.
Let’s say you have a web application deployed on Kubernetes that requires a MySQL database (like from the previous example). With a well-designed IDP, when you deploy your app to a dev environment, it automatically provisions a lightweight database for testing purposes. When you move the same application to a production environment, the IDP recognizes the change in context and deploys a more robust and secure database solution to handle the increased traffic and data demands. This dynamic resource management ensures that the right resources are available for each environment, optimizing efficiency and scalability.
IDP is the sum of the tech and tools that an operations, DevOps, or platform engineering team glues together to build golden paths (GPs) for developers. A GP refers to an optimal and standardized route that developers can follow to deploy applications within an IDP. Evaluating GPs is often an early consideration when building an IDP or establishing a platform engineering team because it ensures that developers have a well-defined, efficient path for application deployment.
All the existing tools and infrastructure are part of the IDP; they integrate with it and are orchestrated by the IDP to enable continuous delivery or even continuous deployment (CD) processes.
An IDP typically establishes integration points with your current CI pipelines on one end and your underlying hardware infrastructure—like Kubernetes clusters, databases, and file storage—on the other. This stands in contrast to platform-as-a-service (PaaS) solutions, which often provide a complete, bundled package inclusive of infrastructure, frequently relying on proprietary technology stacks.
In order for these integrations to work seamlessly, the notion of a platform orchestrator was introduced. This is described in more detail in the discussion of IDP tooling below.
Difficulty in setting up new environments often necessitates requesting assistance from other teams, resulting in potential delays ranging from hours to days or even weeks, depending on team priorities. This challenge can lead to environments being left active when not in use, causing resources to idle and incur unnecessary costs. The inability to create environments promptly can significantly impact delivery speed, causing shared environments to be blocked for extended periods and hindering multiple teams from testing their services or features—a problem that occurs regularly.
IDPs empower developers to generate new, fully provisioned environments on demand, a concept known as environment as a service (EaaS), as mentioned above. Developers can spin up development, testing, or staging environments with ease, replicating production configurations and ensuring that testing environments closely mirror the production environment. This capability accelerates development cycles and ensures that applications behave predictably in different environments.
A good example of such an environment could be Virtual Kubernetes Clusters, where the declaration might look like as simple as this:
In an ideal setup, developers can fully focus on writing code and testing it in action. Everything else should be automated as much as possible, which an IDP can enable teams to set up. It provides a consolidated view of the most important debugging information (e.g., deployment logs and container logs) for any current or even past deployment. This centralized information is a great starting point for debugging a problem and can save a lot of time and avoid unnecessary headaches.
A good example of such an implementation is a Backstage Kubernetes plugin, a tool that’s designed for the needs of service owners, not cluster admins. Now developers can easily check the health of their services no matter how or where those services are deployed—whether on a local host for testing or in production on dozens of clusters around the world.
Various methodologies exist for handling access rights and permissions within software systems, but the predominant approach, particularly in enterprise contexts, is role-based access control (RBAC). Historically, RBAC configuration and permission management have often operated in isolation, occasionally overseen by security teams and, at times, lacking automation.
The objective of an IDP is to unify and streamline RBAC management, offering developers a self-sufficient mechanism to utilize RBAC while ensuring adherence to security policies. Looking again at the Backstage example, we can see that it has several plugins providing the integration with various cloud providers and Kubernetes RBAC and several options for incorporating them into software templates.
IDPs rely on a set of core DevOps tools to provide the functionality and automation necessary for efficient software development and operations. The detailed list of categories can be found here, but two essential groups in this tooling arsenal are service catalogs and platform orchestrators.
Service catalogs simplify resource access, while platform orchestrators automate infrastructure management. These components work together to provide a seamless and efficient environment for development and operations teams, ultimately enhancing the software development lifecycle.
Service catalogs within IDPs serve as central repositories of services, tools, and resources that development teams can leverage throughout the software development lifecycle. These catalogs streamline access to various services, reducing the time spent on resource discovery and provisioning.
Let’s take Spotify’s Backstage again, a popular service catalog tool. It offers a unified platform where teams can discover, provision, and manage a wide range of services, from databases to testing tools. By centralizing access to services, service catalogs simplify and accelerate the development process, ensuring that the right tools are readily available when needed.
Platform orchestrators are responsible for managing and orchestrating the underlying infrastructure and resources that support development and operations. They automate tasks related to resource provisioning, scaling, and lifecycle management.
These orchestrators are responsible for managing and coordinating the underlying infrastructure and resources that support development and operations. They automate tasks related to resource provisioning, scaling, and lifecycle management and enable IDPs to provide developers with consistent, scalable, and reliable environments for their applications. By automating infrastructure management, they reduce operational overhead and ensure that resources are used efficiently.
When building or evaluating an internal developer platform (IDP), several technical considerations are paramount to ensure that it aligns with your organization’s requirements. Here’s a concise look at these key considerations, broken down into several categories.
Developer experience (DevEx) is critical for developer satisfaction and productivity. An effective IDP should prioritize speed and ease of use, offering streamlined workflows and a user-friendly interface. When well designed, developers should be able to access and utilize platform resources swiftly, minimizing friction and cognitive load in the development process.
Many companies adopting IDPs and establishing platform engineering teams recognize the significance of having a dedicated product manager (or an engineer fulfilling that role). This individual plays a pivotal role in bridging the gap between the platform and developers, ensuring that the IDP aligns with developers’ needs and priorities.
Building a successful IDP necessitates a deep understanding of what its primary customers—developers—expect from the platform. This understanding is acquired through ongoing feedback mechanisms such as workshops, surveys, and direct interactions. More comprehensive information on product management within the IDP context can be found in dedicated blog posts and resources.
There are a couple of factors to take into account in this area:
Flexibility is an essential consideration when assessing an existing IDP or designing a new one. Take these aspects into account:
Cost is always a critical factor in any decision. Assess it while taking into consideration the following:
IDPs are catalysts for efficiency and innovation in modern software development. From streamlining operations to empowering application developers, IDPs are redefining the way teams collaborate and create. As organizations continue their digital transformation journeys, embracing IDPs becomes not just a choice but a necessity.
In this article, we unveiled the core components of IDPs, showcased their role in the evolution of DevOps, and highlighted how they benefit platform engineers, operations teams, and developers. We also emphasized the importance of thoughtful implementation and alignment with organizational goals. With IDPs at your disposal, you’re equipped to enhance the developer experience, reduce cognitive load, improve cost efficiency, and ensure faster delivery, higher quality, and a competitive edge.
Learn how platform engineering bridges the gap between complex cloud-native technologies and daily engineering workflows.
Learn how Internal Developer Platforms provide automated self-service solutions to simplify and standardize software practices, infrastructure, environments, and operations.
Learn how citizen developers differ from traditional developers and the important role they play in fostering tech agility in businesses.
Learn how Backstage, an open-source project developed by Spotify, can help unify an organization's software ecosystem into a single developer portal.