Creating SNS And SQS Using Terraform

Last Updated : 02 Apr, 2024

Terraform, is an open-source and specific infrastructure as a code (IaC) tool developed by HashiCorp. that helps the Terraform users to build, change, and version their infrastructure in the most secure and efficient way. using terraform, users can manage resources such as virtual machines, storage, and network through infrastructure provisioning and management.

terraform helps to implement infrastructure provisioning and management across different environments in a single process, regardless of whether these environments are on-premises or in the cloud.

Why To Use Terraform ?

Managing infrastructure was often done by manually and it was a process prone to mistakes. There would be no automation to set up infrastructure without IaC tools like Terraform, which meant manually doing everything leading to inconsistencies and slow deployments. However, Chef and Puppet among other configuration management tools offer some level of automation though they are more about server configurations than the provision of complete infrastructure components. Without comprehensive automation in place, scaling out infrastructure was a challenging task that involved repetition of manual activities and exposed organizations to human errors. Traditional infrastructure management lacked strong versioning tools which made it difficult to keep track of changes, work together well and roll back to previous configurations when necessary. Nevertheless, Terraform changed all this with its introduction transforming how we provision and administer infrastructures.

How Does Terraform Works?

The core components of Terraform are as follows

1. Terraform Core

2. Terraform Plugins

Terraform Core And Plugins Remote Plugin Calls

1. Terraform Core

It forms the backbone of Terraform itself. This piece is statically compiled binary written in Go programming language. The communication between terraform core and terraform plugins happens through RPCs (remote procedure calls). It discovers various plugins paths to include into its working-flow via RPC-based protocol regarding their remote registration and discovery procedures.

Terraform Command Line Interface (CLI), enable users to communicate through an interface that facilitates their infrastructure provisioning and management activities.

Responsibilities Of Terraform Core

The following are the responsibilities of Terraform Core:

Terraform reads and parses configuration files along with modules.

The state of infrastructure resources is managed by Terraform core.

Terraform Core keeps track of what’s already created and what needs to be changed.

for any changes it executes plan.

Terraform core communicates with plugins for integration via RPC.

2. Terraform Plugins

Terraform plugins are also written in Go programming language. And are executable binary files used by Terraform core with RPC.
Each plugin provides functionality for a particular service, like AWS, or run scripts or commands.
Every Provider and Provisioner used in Terraform functions as a plugin. They operate as independent process and communicate with the main Terraform program through an RPC.

Responsibilities Of Terraform Plugin

The following are the responsibilities of Terraform plugins:

set up libraries which allows you communicating with external services or making API calls.

Validating your credentials against the cloud provider.

Describe the resources and data sources which associates with specific services.

Running commands or scripts on the specified resource after it’s created, or when its deleted.

Terraform Workflow

The fundamental Terraform workflow consists of three primary steps:

1. Definition / Write

In the initial stage, users create Terraform configuration files in HashiCorp Configuration Language (HCL), a human-readable language.
For example, one may have a main.tf file which uses Terraform syntax to declare AWS EC2 instances, security groups and S3 buckets as resources.
Users then specify the resources they want to create with terraform, like virtual machines, networks or their respective configurations.

2. Terraform Plan

After writing the Terraform configuration files, you can execute the terraform plan command.

Terraform plan

The plan stage enables terraform to read the configuration files and compare what is desired state specified in your configuration file with the current state of your infrastructure.
When using the plan command it means that you are not affecting any changes on your infrastructure, rather it will show you how terraform intends to behave if you apply any changes bring in by your configuration.

3. Execution/ Apply

Running the terraform apply command allows user to apply the changes in their execution plan.

terraform apply

This implies that Terraform is going to execute all the planned actions and make every necessary change on infrastructure so as bring it to the desired state as described in configuration files.
Terraform will build, change or destroy infrastructure resources as required by the configuration file during this phase.
After a successful completion of applying process, infrastructure will be provisioned or updated based on the specification given in Terraform configuration files.

Core Components Of Terraform

The following are the core components of Terraform:

Terraform Core: Terraform core is responsible for parsing configurations, managing state files and executing plans that create, modify or delete resources within an infrastructure.
Configuration Language: In order to define infrastructure resources, their dependencies and configurations, Terraform uses a declarative language called HCL (HashiCorp Configuration Language). It is designed for simplicity and readability.
Providers: Specific infrastructure providers are interacted with by providers such as AWS, Azure, Google Cloud Platform and others. This allows Terraform to manage a set of resource offerings from each provider thus allowing cross-cloud provisioning and multi-provider support.
State Management: The file “terraform.tfstate” is where all the information about infrastructure is kept by Terraform. For example it records details like which resources have been created or how they are linked together so that Terraform can know what has already been deployed.
Modules: Modules are like building blocks in Terraform. They’re pieces of code that you can reuse to create your infrastructure more easily. without starting from scratch each time.
Commands and CLI: In managing and interacting with their infrastructures, Terraform provides a command line interface (CLI) for such operations. Using CLI users can initiate configurations, plan alterations, apply upgrades among others which is actually the main interface for carrying out Terraform operations.

Features Of Terraform

The following are the features of Terraform:

Declarative Configuration: Terraform configuration file Usually written in declarative syntax, a terraform configuration file is just a list of desired resources like, user simply describes their infrastructure needs in terms of what should be created instead of providing procedural instructions to Terraform on how each resource must be provisioned.
Modular Architecture: Terraform adopts a modular architecture, making it possible for users to arrange configurations in reusable modules. In turn, this prompts code reuse, streamlines the entire process of managing elaborate infrastructural frameworks and more importantly, promoting teamwork.
Manage Any Infrastructure: Additionally, Terraform is not just limited to cloud infrastructure management but extends even to on-premises infrastructure as well as hybrid cloud environments via provider plugins. Ultimately, this gives you the ability to control any kind of infrastructure with a single workflow.
Community Ecosystem: Moreover, the community ecosystem around Terraform is vibrant it has many providers, modules and integrations. So that terraform users can extend its capabilities and rapidly build their infrastructure using community-contributed resources.

Understanding Of Amazon SNS And SQS

Firstly, lets start with discuss about Amazon SNS,

Amazon SNS (Simple Notification Service)

This is a notification web service in Amazon called Amazon SNS (Simple Notification Service). It automates the process of sending notifications/messages.
it is Highly available, durable, secure and fully-managed publisher/subscriber (pub/sub) messaging service.
From one place to many places are messages sent through Amazon SNS so that user can send email by this way through HTTP/HTTPS endpoints or use them with amazon sqs or amazon lambda or even sms/text messages.
Amazon SNS consists of topics, which can be considered as different channels for different types of messages, and subscriptions, that are like the people or places that receive those messages.

Amazon SNS sends messages in two different ways. They are

1. Application To Application (A2A)

2. Application To Persion (A2P)

1. Application To Application (A2A)

Application-to-application (A2A) messaging in Amazon SNS enables a fast, push-based, many-to-many messaging way for different software programs to talk to each other. it sends message Asynchronously. which means sending messages between apps without needing to wait for a response.

Application-to-Application offers efficient messaging between distributed systems, microservices, and event-driven serverless applications. These applications include Amazon Simple Queue Service (SQS), Amazon Kinesis, Data Firehose, AWS Lambda, and other HTTPS endpoints.

SNS A2A approach

2. Application To Person (A2P)

A2P (Application-to-Person) functionality in Amazon SNS is like a tool that send messages to the customers in different ways. It can send messages as SMS texts, push notifications to their mobile devices, or even emails.

SNS A2P approach

Terminologies Of Amazon SNS

Topic: A communication channel within Amazon SNS that acts as a logical access point for publishing and subscribing to messages.
Publisher: Publishers in Amazon SNS is like sender that creates and sends a message to topic, and asynchronously communicates with the subscriber. a publisher sends messages only to a specific topic they have permission to publish.
Subscriber: subscriber refers to a recipient or endpoint that has opted in to receive notifications or messages related to a specific topic.

Applications of SQS

The following are some Applications of Amazon SQS

Ad-hoc Job Queueing: Sometimes, there are recurring tasks that need to be executed at specific intervals, like backing up data every night. With Amazon SQS which can be integrated with timer-based services like Amazon CloudWatch, you can set up a schedule to put these tasks in a queue and can pick up each task at the right time and get it done. It’s a handy way to manage recurring tasks.
Data processing: Amazon SQS can be used in data processing workloads. Imagine you have a group of producers or devices like mobile devices that are all producing messages or data, and that requires processing. Amazon SQS helps by organizing this data into a neat queue (like waiting in line). Then, a system can process each piece of data one by one, making sure nothing gets missed or lost. This is great for dealing with lots of data quickly and efficiently.
Decoupled Microservices: In a microservices setup, each service works independently, these small, independent pieces of software need to talk to each other without causing delays. SQS helps them do that by letting them send messages back and forth without having to know exactly who’s on the receiving end. This keeps things flexible and loosely connected.

Difference Between Amazon SNS and Amazon SQS

The following are the difference betwen Amazon SNS and Amazon SQS:

Amazon SNS	Amazon SQS
Amazon SNS is a publisher/subscriber messaging service.	Amazon SQS is a message queueing service.
it is designed for message distribution.	it is designed for message processing.
Amazon SNS enables publishers to send messages to multiple subscribers simultaneously.	Amazon SQS delivers messages to consumers by pulling them from the queues. Each message in the queue is typically processed by a single consumer.
in this service, Subscribers can be of different types, such as SQS queues, Lambda functions, or even email.	in this service Messages are stored in queues and can be processed by one or more consumers.
SNS offers real-time message distribution to subscribers opted in specific topics.	SQS provides asynchronous message processing with queues.
SNS allows for push-based message delivery.	SQS supports pull-based message retrieval.

Creating Amazon SNS And Amazon SQS Using Terraform: A Step-By-Step Guide

Step 1: Terraform Configuration file

Create a file with “.tf” extension to define SNS and SQS.
Define the AWS provider block .

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.31"
    }
  }
}

provider "aws" {
  region = "<your-aws-region>"
}

sns - sqs terraform file

Step 2: Initialise A terraform Working Directory

Run the following command to initialize a working directory.

Terraform init

It is one of the first commands we should run when starting to work with Terraform in a new directory.

It will Installs the necessary plugins (providers) for cloud interactions.

also Downloads referenced modules.

Initialized the backend for storing state files.

Validating configuration files for syntax and dependencies.

Terraform Init

Step 3: Defining AWS SNS And AWS SQS

resource "aws_sns_topic" "<your-desired-resource-name>" {
  name = "<sns-name>"
}

resource "aws_sqs_queue" "<your-desired-resource-name>" {
  name = "<sqs-name>"
}

Defining SNS And SQS

aws_sns_topic

aws_sns_topic is one of the Terraform resource offered by the AWS provider. It is used to create Amazon SNS topic, which is a communication channel used for sending messages and notifications to subscriber.
aws_sqs_queue is a terraform resource type. It is used to create an Amazon SQS queue, Messages are stored in queues and can be processed by one or more consumers.
The aws_sqs_queue resource block defines the configuration for creating an SQS queue within the AWS infrastructure. When Terraform executes, it interacts with the AWS API to provision the specified SQS queue with the defined attributes.

Step 4: Apply Terraform Configuration

execute following terraform command to see the execution plan and verify that Terraform will create the expected resources.

Terraform plan

terraform plan

If everything looks correct, apply the changes by running terraform apply. It will create our resources.

terraform apply

Applying Terraform Configurations

After the terraform apply the creation of resources, configuration of files are going to takes, you can see in the below screenshot.

Creating Resources

Step 5. Verify Created Resources In AWS

Verify resource existence in the AWS Management Console or through AWS CLI commands.
The below screenshot illustrates successful creation of Amazon SNS with name “gfg-sns”

Verifying Created SNS

The below screenshot illustrates the success creation of Amazon SQS with name “Amazon SQS”

Amazon SQS

Step 6: Clean up

We have successfully created Amazon SNS and SQS, now its time to clean up to prevent from unexpected and potentially large bills Executing following command we destroy created aws resources.

Terraform destroy

Cleaning Up the resources

Challenges Of Terraform

The following are the challenges of Terraform:

Manually Managed Infrastructure: Before Infrastructure as Code (IaC) tools like Terraform, people had to set up their infrastructure manually. This made it prone to errors and inconsistencies whereby things did not work similarly in different environments.
Insufficient Automation In Infrastructure Management: Chef and Puppet were some of the earliest tools that automated tasks but mostly focused on setting up and managing individual servers and not on entire infrastructures.
Managing Infrastructure Without Tracking Changes: there was no proper way to manage infrastructure changes. Thus this made collaboration hard and if by any chance the project turned out wrong, it could be difficult to fix.

How Terraform Deals With These Challenges

Terraform is key in enabling modern approaches like DevOps as well as cloud-native environments by enhancing flexibility, dependability along with speed in infrastructure management.

Automatic setup process: Terraform makes sure all infrastructure is taken care of on its own, automatically, also it enables deployments to be consistent and easier way of redoing the same tasks over and over again.
Track of Changes: By facilitating collaboration with others, any changes in your infrastructure can be easily tracked, even though if something goes wrong, you may go back to how things were before.
Terraform keeps things consistent: Terraform helps keeping things all at a time by writing down how the infrastructure should look using a HCL language. For this reason, if anything mistakenly shift the position or alter some attribute of an element across all files. It stops things from accidentally changing and makes sure everything looks the same everywhere.

Conclusion

It is beneficial for users to be able to construct a cost-effective scalable and reliable infrastructure by being able to control AWS resources such as SNS (Simple Notification Service) and SQS (Simple Queue Service) with Terraform. The infrastructures can be defined in a version controlled way so that developers can manage their infrastructure configurations through the use of Terraform’s declarative syntax and infrastructure as code principles, thereby ensuring reproducibility and reliability across environments.