Creating and delivering applications today requires packaging. Docker is one of the most popular container systems that allows developers to separate applications and their container dependencies. However, monitoring and logging events in the Docker container is important for the system’s security, troubleshooting, and overall health. This article will show how to configure Docker container events using ELK Stack.
Install Docker on your system
Before installing ELK Stack on our system, we have to install docker on our machine, you can install docker on any computer (RHEL, Ubuntu, Fedora, etc.), in my case I have Kali Linux. You can run the following command to install docker on your machine:
sudo apt updateAfter updating all these requirements on your local machine, You can execute the following command:
sudo apt install docker.ioAfter all these things, You need to start and enable Docker services on you local machine, so that we can install any containers on your Docker. To enable and start Docker on your local machine, execute the following command:
sudo systemctl start docker && sudo systemctl enable dockerNow to check whether the docker is running or not, we can execute the following command:
service docker statusSo as we have started our docker services on our local machine, Now we are ready to install any kind of container, image, service, etc.
Set Up Docker Containers
For demonstration purposes, create a prototype Docker container. Use whatever Docker image you like, such as the official Nginx image:
sudo docker run -d --name nginx-container -p 8080:80 nginxNow after this process, we need to install ELK Stack components so that we can access ELK on our docker container, To install Elastic search from ELK Stack on your machine, follow the command given below:
sudo apt install openjdk-11-jre-headless -y
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'
sudo apt update
sudo apt install elasticsearch -y
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
Copy and paste the above command on your terminal and you will have to enter your root password, After that the ELK Stack will be installed on your docker machine.
sudo apt install logstash -y
sudo systemctl start logstash
sudo systemctl enable logstash
And after that we need to install one more important component of ELK stack, that is Kibana. To install Kibana, Just copy and paste the command given below on your terminal.
sudo apt install kibana -y
sudo systemctl start kibana
sudo systemctl enable kibana
Configure Logstash for Docker
Create a Logstash configuration file for Docker by creating a file, e.g., docker-config.conf. In order to access Logstash Dashboard on your local machine/ Browser you need to do some configurations as given below:
sudo nano /etc/logstash/conf.d/docker-config.confAfter that add the following configuration script on the docker-config.conf
input {
file {
path => "/var/lib/docker/containers/*/*.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [path] =~ "json.log" {
json {
source => "message"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "docker_logs"
}
}
Save this script and name it as docker-config.conf
Start Logstash with the Docker Configuration
After all these steps now we are ready to use ELK Stack on our browser on our local machine, Just enter the command given below on your linux terminal and access the Logstash dashboard on your browser:
sudo systemctl restart logstashNow open any of browser that you are using on your local machine, and browse the URL: http://localhost:5601. In my case i am using firefox on Kali Linux. After opening the URL: http://localhost:5601 on your local browser the interface will be like as follow:
Click on add integrations and then you can browse or install any kind of integration with your Logstash dashboard.
Conclusion
By following the steps above, you have successfully configured ELK Stack to log events in Docker containers. This powerful combination improves your ability to monitor, analyze and troubleshoot your storage, providing insight into usage behavior and performance.
FAQ’s On ELK Stack to Log Docker Containerized Events
1. What is the ELK Stack, and how does it relate to Docker containerized events?
ELK Stack consists of Elasticsearch, Logstash and Kibana, providing a comprehensive solution for management and analytics. When integrated with Docker, it allows you to centralize and analyze logs generated by containerized applications.
2. How can I set up ELK Stack to monitor Docker container logs?
To configure ELK Stack for Docker, you need to configure Logstash to collect Docker container logs, send them to Elasticsearch for indexing, and visualize the data using Kibana. Detailed instructions can be found in the official documentation of each ELK.
3. What benefits does ELK Stack offer for analyzing Docker logs?
ELK Stack provides real-time auditing, powerful search capabilities, and customizable visualizations from Kibana. It helps identify problems, troubleshoot errors, and gain insight into the functionality and behavior of Docker-enabled applications.
4. Can ELK Stack handle logs from multiple Docker containers?
Yes, ELK Stack is made to manage logs from binary sources, including various Docker containers. Logstash can be configured to get logs from different containers, process them, and send the data to Elasticsearch for centralized storage and investigation.
5. Are there any best practices for optimizing ELK Stack performance with Docker logs?
Improving ELK Stack performance includes configuring Logstash to filter relevant data, properly managing Elasticsearch indexes, and allocating sufficient resources to each ELK. Also consider using tools like Filebeat to route logs to Logstash, thus reducing the load on the Docker log driver. Please refer to the ELK documentation for detailed optimization information.