Monitoring Docker Environment with Prometheus

Introduction:

Prometheus is a popular open-source monitoring and alerting toolkit used widely in the DevOps world. It allows users to collect and monitor metrics from various sources and provides powerful querying and alerting capabilities. In this article, we will walk you through the process of setting up Prometheus to monitor multiple containers and the host system using cAdvisor and Node Exporter. We’ll cover each step in detail, ensuring that you have a clear understanding of the process, and address any common errors that may arise.

Prerequisites:

Before we begin, make sure you have the following components installed and set up:

1. A Ubuntu or RHEL-based virtual machine (VM) or server where Prometheus will be installed.
2. Docker and Docker Compose installed on the VM to manage containers.
3. Basic knowledge of Linux command-line and package management.

Step 1: Installing Prometheus:

To begin, we’ll install Prometheus on our Ubuntu or RHEL-based VM. Prometheus has an official Docker image that simplifies the installation process. To install Prometheus, follow these steps:

1.1 Install Docker: If you don’t have Docker installed, follow the official Docker installation guide for Ubuntu or RHEL to install Docker on your system.

1.2 Set Up Docker Compose: Docker Compose helps manage multi-container applications. To install Docker Compose, use the following commands:

For Ubuntu:

sudo apt-get update
sudo apt-get install docker-compose

For RHEL:

sudo yum install epel-release
sudo yum install -y python-pip
sudo pip install docker-compose

Step 2: Setting Up Node Exporter for Host Metrics:

Node Exporter is used to collect host-level metrics and expose them to Prometheus. Here’s how to set up Node Exporter:

2.1 Download Node Exporter: Download the latest release of Node Exporter for your system from the official Node Exporter GitHub repository. Use the following commands to download and extract Node Exporter:

# Replace <VERSION> with the actual version number
wget https://github.com/prometheus/node_exporter/releases/download/v<VERSION>/node_exporter-<VERSION>.linux-amd64.tar.gz
tar xvfz node_exporter-<VERSION>.linux-amd64.tar.gz

2.2 Move Node Exporter Binary: Move the extracted Node Exporter binary to an appropriate location on your system, such as /usr/local/bin/.

sudo cp node_exporter-<VERSION>.linux-amd64/node_exporter /usr/local/bin/

2.3 Create Node Exporter User: Create a new system user named node_exporter to run Node Exporter as a non-privileged user.

sudo useradd -rs /bin/false node_exporter

2.4 Set Up Systemd Service: Create a systemd service file to manage the Node Exporter service. Create a new file named node_exporter.service in the /etc/systemd/system/ directory.

sudo nano /etc/systemd/system/node_exporter.service

Copy and paste the following content into the file (replace <VERSION> with the actual version number) and save it:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

2.5 Enable and Start Node Exporter Service: Enable the Node Exporter service and start it using systemd:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

2.6 Verify Node Exporter Service: Check the status of the Node Exporter service to ensure it is running without any errors:

sudo systemctl status node_exporter

Step 3: Setting Up cAdvisor for Container Metrics cAdvisor collects performance metrics from containers and exposes them to Prometheus. Follow these steps to set it up:

3.1 Create cAdvisor Docker Compose File: Create a new file named docker-compose.yml in a directory of your choice and paste the following content into it:

version: '3.7'
services:
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

3.2 Start cAdvisor Container: Run the following command in the directory containing the docker-compose.yml file to start the cAdvisor container:

docker-compose up -d

3.3 Verify cAdvisor Container: Check the status of the cAdvisor container to ensure it is running

docker ps

Step 4: Configuring Prometheus for Scraping Metrics:

Now that we have both Node Exporter for host metrics and cAdvisor for container metrics set up, we need to configure Prometheus to scrape these metrics. Follow these steps:

4.1 Create Prometheus Configuration File: Create a new file named prometheus.yml in a directory of your choice and paste the following content into it (Replace <your_host_ip> with your machine's IP address or domain name):

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['<your_host_ip>:9090']

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<your_host_ip>:9100']
  
  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['<your_host_ip>:8080']

Launching a sample container application

docker run -d --name web -p 8081:80 nginx

Now that the setup of the Docker container is complete, let’s go ahead and open the Prometheus UI by visiting and then running https://<PROMETHEUS_SERVER_EXTERNAL_IP>:9090 the following query by typing in the textbox
It should show something like the following

Prometheus — container_memory_usage_bytes

We can also view the time series of this metric by clicking on the Graph tab, but before we do so, let’s load our NGINX service by using the Apache Bench tool. Apache Bench is a load testing tool that will help us fire HTTP Requests to the NGINX endpoint using the command line. On your Docker server, run the following command to start a load test:

ab -n 100000 http://localhost:8081/

Output

It will hit the endpoint with 100,000 requests, and therefore it provides a fair amount of load to do a memory spike. (Note : It can give error if you do not have Bench tool installed so please google it and install Apache Bench Tool)
Now, if you open the Graph tab, you should see something like the following

To visualize node metrics, we can use the following PromQL statement to get the node_ cpu value of the Docker host(Prometheus have so many quires to monitor the docker environment)

Monitoring your Docker containers and the host system is essential for maintaining their performance and reliability. By following this complete guide, you have set up a robust monitoring solution using Prometheus, cAdvisor, and Node Exporter. Additionally, you have learned how to launch a sample containerized application and monitor its metrics using Prometheus. Embrace the power of monitoring, and your Dockerized applications will thrive with enhanced performance and stability.

Tired of Manual Setup for Prometheus? 🤯 Automate for Efficient Docker Monitoring! 📈 Discover essential metrics to monitor and fine-tune your Dockerized applications for optimal performance and reliability in my latest article

https://medium.com/@akdashish07/automating-prometheus-setup-for-efficient-docker-monitoring-f6f3a5cb6faa

Happy monitoring!!!!!!!!!