Infrastructure As Code Using Pulumi

Last Updated : 13 Feb, 2024

Infrastructure as code (IaC) is a method for automating the provisioning and administration of infrastructure. Infrastructure as code is fundamentally about applying software engineering principles, techniques, and tools to cloud infrastructure.

Prior to infrastructure as code, infrastructure was (and still is!) provisioned in a variety of ways, including pointing and clicking in a user interface (UI), running commands via a command-line interface (CLI), running batch scripts, and using configuration management tools that were not designed for cloud infrastructure. Each of these approaches has limitations; interactive methods including a UI or a CLI frequently cause issues with repeatability and consistency, whereas batch scripts or configuration management systems may be unable to manage infrastructure declaratively. Modern approaches leverage platforms like Pulumi to embrace and assist the entire software engineering lifecycle.

Why is infrastructure-as-code important?

1. Transition to the cloud

One trend, of course, is the ongoing migration to the cloud. More and more businesses are moving workloads from on-premises infrastructure to cloud environments.

It is worth noting the scope of this transformation. The term “cloud environments” is more broad than many people believe. Hyper-scaler clouds, such as Amazon Web Services (AWS) or Microsoft Azure, may come to mind first, but there is so much more to cloud architecture.

Specialized cloud providers
On-premises data centers run private cloud technologies like VMware vSphere.
Modern SaaS infrastructure firms such as Cloudflare, Snowflake, Confluent, Datadog, New Relic, and many others
Other cloud-based assets, such as Auth0, GitLab, and GitHub

All of these cloud environments can be provided or controlled via APIs, making them manageable with infrastructure as code tools.

2. Cloud modernization

The second trend is cloud modernization. At first look, this may appear to be redundant with the first trend, the ongoing transfer to the cloud. However, after enterprises have transitioned to the cloud, they tend to look for ways to optimize the value they derive from their cloud environment. This typically entails implementing technologies like serverless, containers, and Kubernetes. In certain circumstances, cloud modernization entails leveraging managed services to shift some of the heavy lifting to the cloud provider. In other circumstances, cloud modernization entails making more use of ephemeral, stateless workloads that persist for a limited time before being deactivated. When used appropriately, all of these approaches allow teams to provide value faster. These technologies and services necessitate a more detailed infrastructure management approach. IaC is well-suited to combining all of the primitives provided by the cloud provider into business-oriented solutions.

3. Frequent infrastructural modifications

Finally, the pace of change in a company’s infrastructure is increasing. Cloud adoption and modernization have contributed to an increase in the rate of change. However, there is a third reason: enterprises are discovering that leveraging the cloud’s intrinsic elasticity allows them to move faster.

For teams managing tens or hundreds of cloud services that change every few months, managing infrastructure through scripts or interactive means (such as a UI or a CLI) may still be feasible. More typically, teams are responsible for managing thousands or tens of thousands of resources that vary on a daily or even hourly basis. Embracing automation through infrastructure as code is the only way to manage that level of complexity.

What are the main components of infrastructure as code?

The key aspects of infrastructure as code are the same as those found in most software engineering environments. This includes:

1. Infrastructure as code mechanism

For all practical purposes, infrastructure as code requires a tool or engine to translate IaC instructions into something that cloud provider APIs understand and can use. Infrastructure as code tools may be offered and confined to a single cloud provider (for example, AWS Cloudformation), or they may support numerous cloud providers. Tools may only support YAML or JSON, require the usage of a specialized and proprietary domain-specific language (DSL), or support general-purpose programming languages like TypeScript/JavaScript, C#, Go, Python, and Java.

2. Version control

When infrastructure is specified as code, it may be checked into source control, versioned, and code-reviewed according to standard software engineering methods. Version control systems, such as GitHub, GitLab, allow you to see which changes were made, when they were made, and who made them.

3. Tests

As the complexity of a crucial system increases, people may become concerned about making changes. With infrastructure as code, teams can develop tests to guarantee that their infrastructure is correct. They can embed policies to ensure that all provisioned infrastructure and configurations meet compliance standards. Infrastructure components, once tested, can be reusable bits of code that encapsulate best practices and can be shared across multiple teams. No more reinventing the wheel.

4. CI/CD pipelines:

Assuming the infrastructure as code tool supports the functionality (which most do), changes to the code that defines the infrastructure can be deployed using existing CI/CD tools, much like CI/CD pipelines automatically build and deploy other types of software.

Advantages of Infrastructure as Code

Infrastructure as code reduces the complexity of cloud infrastructure by leveraging the same software engineering principles, techniques, and tools that have allowed previous software-based systems to scale up. Here are some of the advantages that infrastructure as code brings.

Repeatability and consistency: Infrastructure defined using IaC can be deployed in a very repeatable manner. Do you require a development environment that is a high-fidelity replication of the production environment? Or do you need to verify that the infrastructure is delivered consistently across several regions? This is readily performed with infrastructure as code.
Accountability: Changes to the infrastructure can be easily traced by using version control on code files.
Improved productivity: The majority of developers use an integrated development environment (IDE) on a regular basis. When infrastructure is code, you can use all of the tools that an IDE provides, including autocompletion and the ability to look up methods and their parameters.
Better alignment across teams: Infrastructure as code enables infrastructure and software development teams to embrace DevOps ideas and collaborate more closely. When infrastructure is coded and incorporated into your company’s software lifecycle, stakeholders are already familiar with a single language and set of norms. This same knowledge promotes cross-team communication, which is essential to DevOps.

Why Pulumi ?

Pulumi is a modern infrastructure as code (IaC) instrument that allows developers and infrastructure teams to characterize, convey, and oversee cloud foundation utilizing recognizable programming dialects like JavaScript, TypeScript, Python, Go, and .NET. There are a few advantages using of pulumi. They are :

Familiar Programming Languages: Pulumi permits you to compose framework code utilizing dialects you definitely know and love, like JavaScript, TypeScript, Python, and Go. This brings the obstruction down to passage for engineers who probably won’t be as OK with conventional IaC apparatuses that utilization space explicit dialects (DSLs) like YAML or JSON.
Modularity and abstraction: With Pulumi, you can characterize foundation utilizing reusable parts and reflections, making it more straightforward to oversee complex framework arrangements. This particular methodology helps in making viable and versatile foundation codebases.
Full Lifecycle Management: Infrastructure creation, modification, and deletion are all supported by Pulumi. This implies you can utilize Pulumi not exclusively to arrangement assets yet in addition to oversee them all through their lifecycle, making it more straightforward to deal with changes and updates over the long run.
Multi-Cloud and Hybrid Cloud Support: Pulumi upholds significant cloud suppliers like AWS, Sky blue, Google Cloud Stage (GCP), and Kubernetes, permitting you to oversee assets across numerous mists or half breed cloud conditions from a solitary codebase.
Immutable Infrastructure: Pulumi follows the standards of unchanging framework, where foundation changes are made by making new assets as opposed to adjusting existing ones. This aides in guaranteeing consistency and dependability across arrangements.
Integration with CI/Cd Pipelines: Pulumi can be coordinated into nonstop mix and consistent conveyance (CI/Disc) pipelines, empowering mechanized testing and sending of foundation changes close by application code.
Ecosystem and Community: Pulumi has a functioning local area and a developing biological system of modules and expansions, and that implies you can use local area contributed modules and offer your own foundation code with others.

Pulumi supports an variety of programming languages, They are:

TypeScript: TypeScript is a statically composed superset of JavaScript that orders down to plain JavaScript. It’s broadly utilized for web applications and gives highlights, such as, type checking and interfaces, which can further develop code quality and viability.
Python: Python is a high-level programming language with dynamic typing that is well-known for its readability and simplicity. It’s generally utilized in web improvement, data analysis and artificial intelligence.
Go: Go, otherwise called Golang, is a statically composed, incorporated programming language created by Google. It’s known for its straightforwardness, productivity, and simultaneousness support, making it ideal for building adaptable and performant applications.
.NET: C# and F# are two of the.NET ecosystem’s supported languages by Pulumi. C# is a flexible, statically composed language created by Microsoft, regularly utilized for building endeavor level applications on the .NET stage. F# is a useful first programming language also created by Microsoft, known for its brevity and expressive grammar.

How Pulumi Works ?

Pulumi orchestrates and manages infrastructure using a desired state (declarative) paradigm, allowing you to write infrastructure code in programming languages you already know, such as TypeScript, JavaScript, Python, Go, C#, and Java. This model includes the benefits of programming constructs such as loops, conditionals, and functions, as well as your IDE’s autocomplete, type checking, and documentation. When you write a Pulumi program, the final output will be the state you specify, regardless of the current condition of your infrastructure.

A language host executes a Pulumi program to compute the desired state of a stack’s infrastructure. The deployment engine compares the desired state with the stack’s existing state to determine which resources should be added, updated, or destroyed. To handle individual resources, the engine makes use of a number of resource providers (including AWS, Azure, Kubernetes, and over 150 more). As it runs, the engine keeps track of the status of your infrastructure, including all supplied resources and any pending operations.

The graphic below depicts the interplay between various system components:

1. Language hosts

The language host is responsible for running a Pulumi application and creating an environment in which resources can be registered with the deployment engine. The language host consists of two separate parts:

Pulumi’s language executor, pulumi-language, launches the runtime for the program’s language (e.g., Node or Python). This binary is distributed alongside the Pulumi CLI.

A language SDK is responsible for preparing your program for execution and monitoring its execution to detect resource registrations. When a resource is registered (with new Resource() in JavaScript or Resource in Python), the language SDK sends the registration request to the deployment engine. The language SDK is supplied as a standard package, just like any other code that may depend on your software. For example, the Node SDK is contained in the @pulumi/pulumi package available on npm, whereas the Python SDK is housed in the pulumi package available on PyPI.

2. The deployment engine

The deployment engine is in charge of computing the set of actions required to transform the current state of your infrastructure into the desired state specified by your program. When the engine receives a resource registration from the language host, it checks the existing state to see if the resource has been generated before. If it does not already exist, the engine creates it via a resource provider. If it already exists, the engine collaborates with the resource provider to establish what, if anything, has changed by comparing the resource’s former state to the new desired state as specified by the application. If there are changes, the engine decides whether it can update the resource in situ or if it has to replace it by creating a new version and destroying the old one. The selection is made based on the resource’s changing qualities as well as its type. When the language host informs the engine that the Pulumi program has been completed, the engine searches for any existing resources that have not received a new resource registration and schedules their deletion.

The deployment engine is built into the Pulumi CLI itself.

3. Resource providers

A resource provider is made up of two distinct pieces:

A resource plugin is the binary that allows the deployment engine to manage a resource. These plugins are saved in the plugin cache (found in ~/.pulumi/plugins) and can be managed with the pulumi plugin commands.
An SDK that has bindings for every sort of resource that the provider can manage.
The SDKs, like the language runtime, are accessible in standard package formats. For example, the @pulumi/aws package for Node is available on npm, while the pulumi aws package for Python is available on PyPI. When you add these packages to your project, they run pulumi plugin install in the background to download the resource plugin from Pulumi.com.

Putting everything together

Let us see through a basic example. Assume we have the following Pulumi application that creates two S3 buckets.

resources:
    webBucket:
        type: aws:s3:Bucket
    appBucket:
        type: aws:s3:Bucket

Now we’ll run pulumi stack init mystack. mystack is a new stack, therefore the “last deployed state” has no resources.

Next, we run pulumi up. Because this program is written in YAML, the Pulumi CLI invokes the YAML language host and demands that it run the application. When the first aws.s3.bucket object is created, the language host makes a resource registration request to the deployment engine and then continues to run the program. This is subtle but significant. When the call to new aws.s3.bucket returns, it does not imply that the S3 bucket has been formed in AWS; rather, it indicates that the language host has specified that this bucket is part of the desired state of your infrastructure. The language host continues to execute your application while the engine processes this request.

In this scenario, because the last deployed state lacks resources, the engine calculates that it must construct the web–bucket resource. It uses the AWS resource plugin to generate the resource, which in turn uses the AWS SDK to construct it. It’s worth noting that the engine does not communicate directly with AWS; instead, it requests that the AWS Resource Plugin build a Bucket. As new resource types become available, you can upgrade the version of a resource provider to obtain access to them without having to update the CLI. When the procedure to construct this bucket is completed, the engine saves information about the newly generated resource to its state file.

As the engine created the web-bucket bucket, the language host proceeded to run the Pulumi application. This resulted in another resource registration (for content-bucket). Because there is no connection between these two buckets, the engine can execute the request while also creating the web-bucket.

After both processes are performed, the language host departs, indicating that the program has finished execution. The engine and resource providers will thereafter shut down. The state of mystack now looks like this:

stack mystack
   - aws.s3.Bucket "web-bucket719a9"
   - aws.s3.Bucket "app-bucket394cf"

Take note of the additional prefixes added to these bucket names. The reason behind this is that Pulumi employs auto-naming by default, which prevents resource name collisions when you deploy multiple copies of your infrastructure. If preferred, this behavior can be turned off.

Let’s now modify one of the resources and launch Pulumi up once more. Because Pulumi is based on a desired state model, it will calculate the minimum set of modifications required to update your deployed infrastructure based on the most recent deployed state. Consider the scenario if we wished to enable public reading of the S3 web-bucket. We modify our algorithm to reflect this updated ideal state:

resources:
  webBucket:
    type: aws:s3:Bucket
    properties:
      acl: public-read # add acl
  appBucket:
    type: aws:s3:Bucket

Run pulumi preview or pulumi up to restart the full process. The call to aws.s3 and your program are launched by the language host. A fresh resource registration request is made to the engine by bucket. This time, however, the engine requests the resource provider to compare the current state from our last run of Pulumi up with the desired state specified by the program because our state already has a resource named web-bucket. The acl property’s default value of private has been changed to public-read, which the process detects. The engine verifies that it can alter this property without forming a new bucket by speaking with the resource provider once more. Based on this, the engine instructs the provider to change the acl property to public-read. The changed state is reflected in the current state after this operation is finished.

Additionally, a resource registration request for “app-bucket” is received by the engine. The resource does not need to be altered by the engine, though, as there are no changes between the desired and present states.

Let’s say that app-bucket is rename as data-bucket.

resources:
  mediaBucket:
    type: aws:s3:Bucket
    properties:
      acl: public-read # add acl
  dataBucket:
    type: aws:s3:Bucket

Because web-bucket’s intended state and actual state match this time, the engine won’t need to modify it. However, the engine discovers that there isn’t an existing resource with the name data-bucket in the current state when processing the resource request for data-bucket, therefore it needs to construct a new S3 bucket. The engine searches for any resources in the current state for which it did not find a resource registration after that procedure is finished and the language host has shut down. In this instance, the engine calls the resource provider to delete the app-bucket bucket that is currently in place because we deleted the registration of app-bucket from our program.

Creation And Deletion Order

Pulumi performs resource operations in parallel whenever possible, but recognizes that some resources may be dependent on other resources. If one resource’s output is used as an input for another, the engine stores the dependency between the two resources in the state and uses it when scheduling activities. This list can also be expanded using the dependsOn resource option.

If a resource has to be replaced, Pulumi will attempt to produce a new duplicate before deleting the old one. This is beneficial because it allows infrastructure updates to occur without interruption. The delete Before Replace option allows you to customize this behavior. If you deactivate auto-naming by giving a resource a specific name, it will be regarded as if it was designated as delete Before Replace (otherwise, the new versions’ create operation will fail since the name is already in use).

Declarative and Imperative Approach

With Pulumi, you may create your infrastructure in your choice programming language. When you run pulumi, the Pulumi CLI launches both the language host for your chosen programming language and the necessary providers (via the Pulumi engine). The Pulumi engine coordinates the suppliers, who are responsible for creating, modifying, or deleting your infrastructure. Pulumi’s strength stems from the separation of language support from the engine, which combines the best of both imperative and declarative techniques for your infrastructure as code.

Here’s a breakdown of each component of the Pulumi architecture:

Language hosts include imperative (JavaScript/TypeScript, Go, Python, C#/F#/.NET, Java) and declarative (YAML).
Pulumi engine: declarative.
Providers: imperative

Advantages of Choosing a Language Familiar To Developers

Reduce Learning Curve: Engineers are as of now acquainted with the linguistic structure, figures of speech, and best acts of the picked language, diminishing the time and exertion expected to become familiar with another dialect well defined for infrastructure as code.
Leveraging Existing Skills: Developers can use their existing skills, knowledge, and tooling ecosystem by using a familiar language, resulting in increased productivity and efficiency when writing and maintaining infrastructure code.
Further developed Code Quality: Experience with the language permits engineers to compose cleaner, more informal code, prompting further developed lucidness, practicality, and code quality generally.
Easier Collaboration: At the point when groups utilize a language they knew about, joint effort becomes simpler as understand, review, and add to the codebase all the more successfully.
Broader Adoption : By supporting famous programming dialects, Pulumi makes infrastructure as code open to a more extensive crowd of designers who might not have particular information in setup dialects or space explicit dialects.

Conclusion

In conclusion, Infrastructure as Code (IaC) is a worldview that empowers you to manage and arrangement Infrastructure resources utilizing code, bringing about automation, repeatability, and consistency in your deployments. Pulumi is a modern IaC device that allows you to Infrastructure your systems in well known programming languages like as Python, JavaScript, TypeScript, Go, and others. In this guide, we’ll go over the major standards and processes for implementing out Infrastructure as Code with Pulumi.