RUN SAP BETTER

Prometheus | Telemetry

 

Prometheus is a free software application used for event monitoring and alerting. It records metrics in a time series database built using an HTTP pull model, with flexible queries and real-time alerting.

​​

Prometheus Components

​​

• Prometheus Server

  • Prometheus server is a first component of Prometheus architecture

  • Prometheus server is a core of Prometheus architecture which is divided into several parts like Storage, PromQL, HTTP server, etc

  • In Prometheus server data is scraped from the target nodes and then stored in the database

  • Storage

    • Storage in Prometheus server has a local on disk storage

    • Prometheus has many interfaces that allow integrating with remote storage systems

  • PromQL

    • Prometheus uses its own query language i.e. PromQL which is very powerful querying language

    • PromQL allows the user to select and aggregate the data

• Service Discovery

  • Next and very important component of Prometheus Server is the Service Discovery

  • With the help of Service discovery the services are identified which are need to scraped

  • To Pull metrics, identification of services and finding the targets are compulsory needed

  • Through Service discovery we monitor the entities and can also locate its targets

• Scrape Target

  • Once the services are identified and the targets are ready then we can pull metrics from it and can scrape the target

  • We can export the data of end point using node exporters

  • Once the metrics or other data is pulled, Prometheus stores it in a local storage

• Alert Manager

  • Alert Manager handles the alerts which may occurs during the session

  • Alert manager handles all the alerts which are sent by Prometheus server

  • Alert manager is one of the very useful component of Prometheus tool

  • If in case any big error or any issue occurs, alert manager manage those alerts and contact with human via E-mail, Text Messages, On-call, or any other chat application service

• User Interface

  • User interface is also a important component as it builds a bridge between the user and the system

  • In Prometheus, user interface are note that much user friendly and can be used till graph queries

  • For good exclusive dashboards Prometheus works together with Grafana (visualization tool)

  • Using Grafana over Prometheus to visualize properly we can use custom dashboards

  • Grafana dashboards displays via pie charts, line charts, tables, good data graphs of CPU usage, RAM utilization, network load, etc with indicators

  • Grafana supports and run with Prometheus by querying language i.e. PromQL

  • To fetch data from Prometheus and to display the results on Grafana dashboards PromQL is used

• What is Grafana ?

  • Grafana is a free and open source visualization tool mostly used with Prometheus to which monitor metrics

  • Grafana provides various dashboards, charts, graphs, alerts for the particular data source

  • Grafana allows us to query, visualize, explore metrics and set alerts for the data source which can be a system, server, nodes, cluster, etc

  • We can also create our own dynamic dashboard for visualization and monitoring

  • We can save the dashboard and can even share with our team members which is one of the main advantage of Grafana

• What is Node Exporter ?

  • Node exporter is one of the Prometheus exporters which is used to expose servers or system OS metrics

  • With the help of Node exporter we can expose various resources of the system like RAM, CPU utilization, Memory Utilization, disk space

  • Node exporter runs as a system service which gathers the metrics of your system and that gathered metrics is displayed with the help of Grafana visualization tool

​

Architecture

This diagram illustrates the architecture of Prometheus and some of its ecosystem components

​

Implementation Summary

​

• Documentation

  • Download Files | URL: https://prometheus.io/download/

  • Node Explorer Docs | URL: https://github.com/prometheus/node_exporter

  • Download Files Windows Node | URL: https://github.com/prometheus-community/windows_exporter/releases/

  • Windows Node Explorer Docs | URL: https://github.com/prometheus-community/windows_exporter

• Setting up Prometheus monitoring system and time series database

  • Installing Prometheus on Ubuntu

  • Organizing Prometheus Directories

  • Configuring Prometheus

• Setting up Node Exporter (Linux)

  • Node Exporter Monitor for Windows

  • Configuring Prometheus to Scrape Node Exporter

• Setting up Node Exporter (Windows)

  • Node Exporter Monitor for Windows

  • Configuring Prometheus to Scrape Node Exporter

​

Installing Prometheus on Ubuntu

​

• Open the terminal

• For non-root users, use the command "sudo" to perform administrative tasks​

• Update Repository | Command: apt update​​

• Download Prometheus | Command: wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz

  • ​Check the latest version before downloading

• Extract the file | Command: tar xvfz prometheus-*.tar.gz

• Navigate to the Prometheus Directory | Command: cd prometheus*/

​​

Organizing Prometheus Directories

​

• Create a System User for Prometheus

  • Command: groupadd --system prometheus

  • Command: useradd -s /sbin/nologin --system -g prometheus prometheus

• Create Directories for Prometheus

  • Command: mkdir /etc/prometheus

  • Command: mkdir /var/lib/prometheus

• Move the Binary Files

  • Command: mv prometheus /usr/local/bin

  • Command: mv promtool /usr/local/bin

• Move the Configuration Files

  • Command: mv consoles /etc/prometheus

  • Command: mv console_libraries /etc/prometheus

  • Command: mv prometheus.yml /etc/Prometheus

• Set Owner

  • Command: chown prometheus:prometheus /usr/local/bin/prometheus

  • Command: chown prometheus:prometheus /usr/local/bin/promtool

  • Command: chown prometheus:prometheus /etc/prometheus

  • Command: chown -R prometheus:prometheus /etc/prometheus/consoles

  • Command: chown -R prometheus:prometheus /etc/prometheus/console_libraries

  • Command: chown -R prometheus:prometheus /var/lib/prometheus

​

Configuring Prometheus

​

• Change Prometheus configuration file | Command: nano /etc/prometheus/prometheus.yml

  • Content: Check the content below

• Create Prometheus Systemd Service | Command: nano /etc/systemd/system/prometheus.service

  • Content: Check the content below​

• Reload Systemd | Command: systemctl daemon-reload

• Enable Prometheus | Command: systemctl enable prometheus

• Start Prometheus | Command: systemctl start prometheus

• (Optional) Check Prometheus Status | Command: systemctl status prometheus

​​

Tips | If you later make any changes to "prometheus.services", you will need to:

• Reload Systemd | Command: systemctl daemon-reload

• Restart Prometheus | Command: systemctl restart prometheus

​

Content of  "prometheus.yml"

​

# --- Starting on the next line ->

# Global Configuration
global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).

  external_labels:
    monitor: 'codelab-monitor'

 

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

 

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

 

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

 

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
# Override the global default and scrape targets from this job every 5 seconds.

# --- End ---

​

Content of  "prometheus.service"

​

# --- Starting on the next line ->

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

 

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

RestartSec=5s

 

[Install]
WantedBy=multi-user.target

# --- End ---

 

Node Exporter Monitor for Linux

 

• Open the terminal on the server you want to monitor

• For non-root users, use the command "sudo" to perform administrative tasks​

• Create a user named "nodeuser" | Command: useradd --no-create-home --shell /bin/false nodeuser

  • Options: "--no-create-home" and "--shell /bin/false" will ensure that no home directory is created for this user, and this user cannot login to the server

• Download Node Explorer | Command: wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

  • Check the latest version before downloading

• Extract the files | Command: tar xvfz node_exporter-*.*-amd64.tar.gz

• Go to directory | Command: cd node_exporter-*.*-amd64

• Move the directory | Command: mv node_exporter /usr/local/bin/

• ​Secure the directory | Command: chown -R nodeuser:nodeuser /usr/local/bin/node_exporter

• Manage the Node Exporter instance with Systemd | Command: nano /etc/systemd/system/node_exporter.service 

  • Content: Check the content below​

• Reload the systemd configuration | Command: systemctl daemon-reload

• Start the Node Exporter service | Command: systemctl start node_exporter

• Enable Node Exporter to start on boot | Command: systemctl enable node_exporter

• (Optional) Check Node Explorer | Command: systemctl status node_exporter

• (Optional) Check Metrics | Command: curl http://<node-ip-address>:9100/metrics

  • Or you can check using your browser | URL: http://<node-ip-address>:9100/metrics

​​

Content of  "node_exporter.service"

​

# --- Starting on the next line ->

[Unit]
Description=Node Exporter Service
After=network.target

 

[Service]
User=nodeuser
Group=nodeuser
Type=simple
ExecStart=/usr/local/bin/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

 

[Install]
WantedBy=multi-user.target

# --- End ---

​

Configuring Prometheus to Scrape Node Exporter

​

• Open the terminal on the Prometheus

• For non-root users, use the command "sudo" to perform administrative tasks​

• Config Node on Prometheus | Command: nano /etc/prometheus/prometheus.yml

  • Add the lines in the "scrape_configs" section, after the "prometheus" job_name config: Check below

• Restart Prometheus | Command: systemctl restart prometheus

• Check the Telemetry em Prometheus | URL: http://<prometheus-ip-address>:9090

  • Go to menu: Status -> Targets

​

Content of  "prometheus.yml"

​

# --- Starting on the next line ->

  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets: ['<node-ip-address>:9100']

# --- End ---

​​

Tips: If you need add a second linux node, you just need to install the node_exporter and add one more "target" line on "prometheus.yml" file

​

Node Exporter Monitor for Windows

​​

• Check the last version of node explorer for windows

• Download the ".msi" file

• Install it on windows machine

​

Configuring Prometheus to Scrape Node Exporter

​

• Open the terminal on the Prometheus

• For non-root users, use the command "sudo" to perform administrative tasks​

• Config Node on Prometheus | Command: nano /etc/prometheus/prometheus.yml

  • Add the lines in the "scrape_configs" section, after the "prometheus" job_name config: Check below

• Restart Prometheus | Command: systemctl restart prometheus

• Check the Telemetry em Prometheus | URL: http://<prometheus-ip-address>:9090

  • Go to menu: Status -> Targets

​

Content of  "prometheus.yml"

​

# --- Starting on the next line ->

  - job_name: 'windows'
    scrape_interval: 5s
    static_configs:
      - targets: ['<node-ip-address>:9182']

# --- End ---