How to Install & Configure Prometheus monitoring tool on Ubuntu 20.04.

Prometheus is a free & open-source monitoring tool.It developed by SoundCloud.Prometheus is a metrics based event monitoring and alerting application.It is a web-interface & used for metrics monitoring, event monitoring & alert management.It supports/manage hundreds of services and microservices.

We can Add various type of exporter to the Prometheus system:-

  • blackbox_exporter
  • consul_exporter
  • graphite_exporter
  • haproxy_exporter
  • memcached_exporter
  • mysqld_exporter
  • node_exporter
  • statsd_exporter.

Node_exporter is used for generate metrics about resources like CPU, memory & disk usage.

Install Prometheus monitoring tool on ubuntu.

Update the system.

apt-get update

Create Users and Directory for Prometheus.

###Create Users

useradd --no-create-home --shell /bin/false prometheus  
&
useradd --no-create-home --shell /bin/false node_exporter


## Create Directory
mkdir /etc/prometheus
&
mkdir /var/lib/prometheus

Update the group and user ownership.

chown prometheus:prometheus /etc/prometheus
&
chown prometheus:prometheus /var/lib/prometheus

Download Prometheus monitoring tool 

cd /opt
&
wget https://github.com/prometheus/prometheus/releases/download/v2.26.0/
prometheus-2.26.0.linux-amd64.tar.gz

Here is the command output.

https://github.com/prometheus/prometheus/releases/download/v2.26.0/
prometheus-2.26.0.linux-amd64.tar.gz
Resolving github.com (github.com)... 140.82.112.3
Connecting to github.com (github.com)|140.82.112.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/6838921/4394b400-922b-

.....

Connecting to github-releases.githubusercontent.com (github-
HTTP request sent, awaiting response... 200 OK
Length: 65155818 (62M) [application/octet-stream]
Saving to: ‘prometheus-2.26.0.linux-amd64.tar.gz’

prometheus-2.26.0.linux-amd64.tar.gz   100%[==========>]  62.14M  20.8MB/s in 3.0s    

 ‘prometheus-2.26.0.linux-amd64.tar.gz’ saved [65155818/65155818]

Create a checksum of the Prometheus downloaded file.

sha256sum prometheus-2.26.0.linux-amd64.tar.gz

Here is the command output.

8dd6786c338dc62728e8891c13b62eda66c7f28a01398869f2b3712895b441b9  prometheus-2.26.0.linux-amd64.tar.gz

Extract the downloaded file.

tar -xvf prometheus-2.26.0.linux-amd64.tar.gz

Open prometheus extracted folder.

cd prometheus-2.26.0.linux-amd64

Copy all Prometheus files to /usr/local/bin directory.We have two libraries Prometheus and promtool in prometheus extracted folder.

cp /opt/prometheus-2.26.0.linux-amd64/prometheus /usr/local/bin/
&
cp /opt/prometheus-2.26.0.linux-amd64/promtool /usr/local/bin/

Provide Prometheus user ownership.

chown prometheus:prometheus /usr/local/bin/prometheus
&
chown prometheus:prometheus /usr/local/bin/promtool

Console and console_libraries directories to /etc/Prometheus/.

cp -r /opt/prometheus-2.26.0.linux-amd64/consoles /etc/prometheus
&
cp -r /opt/prometheus-2.26.0.linux-amd64/console_libraries /etc/prometheus
&
cp -r /opt/prometheus-2.26.0.linux-amd64/prometheus.yml /etc/prometheus

Provide the user and group ownership on the directories to Prometheus user.

chown -R prometheus:prometheus /etc/prometheus/consoles
&
chown -R prometheus:prometheus /etc/prometheus/console_libraries
&
chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

Check Prometheus Version.

prometheus --version

Here is the command output.

prometheus, version 2.26.0 (branch: HEAD, revision: 3cafc58827d1ebd1a6be40bab3d8d)
  build user:       root@a67cafebe6d0
  build date:       20210331-11:56:23
  go version:       go1.16.2
  platform:         linux/amd64

Check Promtool Version.

promtool --version

Here is the command output.

promtool, version 2.26.0 (branch: HEAD, revision: 3cafc58827d1ebd1a6be40bab3d8d)
  build user:       root@a67cafebe6d0
  build date:       20210331-11:56:23
  go version:       go1.16.2
  platform:         linux/amd64

Configure the Prometheus file.

vim /etc/prometheus/prometheus.yml
or 
cat /etc/prometheus/prometheus.yml

Here is the command output.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. 
Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds.The default is every 1minute
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 
'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
#The job name is added as a label`job=<job_name>`to timeseries scraped from this config
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090'] or targets: ['Server-ip:9090']

Start Prometheus Systemd file.

sudo -u prometheus /usr/local/bin/prometheus \
        --config.file /etc/prometheus/prometheus.yml \
        --storage.tsdb.path /var/lib/prometheus/ \
        --web.console.templates=/etc/prometheus/consoles \
        --web.console.libraries=/etc/prometheus/console_libraries

Here is the command output.

caller=main.go:380 msg="No time or size retention was set so using the 
default time retention" duration=15d
caller=main.go:418 msg="Starting Prometheus" version="(version=2.26.0, branch=HEAD)"
..
component=web msg="Start listening for connections"address=0.0.0.0:9090
...
caller=main.go:975 msg="Completed loading of configuration file" 
filename=/etc/prometheus/prometheus.yml totalDuration=6.371359ms 
remote_storage=2.263µs web_handler=829ns query_engine=1.357µs 
scrape=5.972424ms scrape_sd=30.904µs notify=16.945µs notify_sd=13.991µs rules=1.947µs
caller=main.go:767 msg="Server is ready to receive web requests."

....

Create a system service file.

vim /etc/systemd/system/prometheus.service

Add the following lines.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload,Start & Enable the prometheus service.

##Reload service
systemctl daemon-reload


##Start Service
systemctl start prometheus

##Enable Service
systemctl enable prometheus

Check prometheus service status.

systemctl status prometheus

Here is the command output.

● prometheus.service - Prometheus
   Loaded:loaded(/etc/systemd/system/prometheus.service;enabled;vendor preset:enabled)
     Active: active (running) ; 39min ago
   Main PID: 1886 (prometheus)
      Tasks: 6 (limit: 1160)
     Memory: 28.8M
     CGroup: /system.slice/prometheus.service
          └─1886 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml

Add port 9090 in ufw firewall.

ufw allow 9090/tcp

Access Prometheus web-interface.

http://server-IP:9090

Here is the output.

Fig 1

 

  • Click on Graph option.
  • Search process_cpu_seconds_total
  • Click on Execute.

Fig 2

 

Install Node Exporter on client server.

Download node exporter.

cd /tmp
&
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Extract the download file.

tar -xvf node_exporter-0.18.1.linux-amd64.tar.gz

Here is the command output.

node_exporter-0.18.1.linux-amd64/
node_exporter-0.18.1.linux-amd64/node_exporter
node_exporter-0.18.1.linux-amd64/NOTICE
node_exporter-0.18.1.linux-amd64/LICENSE

 Change the location of node export.

mv node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/

Create a node_exporter service file.

vim /etc/systemd/system/node_exporter.service

Add the following lines.

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload,Start & Enable the node exporter service.

##Reload service
systemctl daemon-reload

## start service
systemctl start node_exporter

## Enable service
systemctl enable node_exporter

Check the status of node exporter service.

systemctl status node_exporter

Here is the command output.

● node_exporter.service - Node Exporter
     Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: enabled)
     Active: active (running) ; 4s ago
   Main PID: 2381 (node_exporter)
      Tasks: 3 (limit: 1160)
     Memory: 1.5M
     CGroup: /system.slice/node_exporter.service
             └─2381 /usr/local/bin/node_exporter

Open port number 9100 on uew firewall.

ufw allow 9100/tcp

Show Metric list.

http://client-server-IP:9100/metrics

Here is the output.

#HELP go_gc_duration_seconds A summary of pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.2246e-05
go_gc_duration_seconds{quantile="0.25"} 2.7124e-05
go_gc_duration_seconds{quantile="0.5"} 3.1444e-05
go_gc_duration_seconds{quantile="0.75"} 4.6907e-05
go_gc_duration_seconds{quantile="1"} 7.6819e-05
go_gc_duration_seconds_sum 0.000232763
go_gc_duration_seconds_count 6
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 32
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.9579816e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.9179784e+07
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 96160

Configure the Target Server on Prometheus Server.

Open/Login to Prometheus server and Open the prometheus.yml file.

vim /etc/prometheus/prometheus.yml

Add the following lines.

- job_name: 'node_exporter_metrics'
  scrape_interval: 5s
  static_configs:
    - targets: ['client-server-ip:9100']

Restart the prometheus service.

systemctl restart prometheus

Open Prometheus Web interface

http://server-IP:9090

Check Targets.(Click on Status–>Targets. or http://server-ip:9090/targets)

 

Fig. 6

Query for node related metrics on Prometheus Server.

We have various option to find the node related metrics.

node_memory_MemFree_bytes
node_cpu_seconds_total
node_filesystem_avail_bytes
rate(node_cpu_seconds_total{mode="system"}[1m]) 
rate(node_network_receive_bytes_total[1m])

Search node_memory_MemFree_bytes & Click on Execute.

 

Fig. 7

 

 

Leave a Reply