Creating a monitoring dashboard using Prometheus and Grafana

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

Grafana is a graphical tool that connects to Prometheus and helps build a visualization dashboard.

In this blog

1. We shall spinup Prometheus as a docker container on a Ubuntu based server in Azure cloud.
2. Install Node exporter , a tool that scrapes metrics from the host linux machine
3. Spin Up Grafana as another docker container on the same Ubuntu server
4. Create a datasource from Grafana to Prometheus
5. Build a dashboard to view the Node exporter metrics

• Spin Up an Ubuntu server in Azure cloud
• copy the following Prometheus configuration file : prometheus.yml to /home/ubuntu/config directory

scrape_configs:
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['xx.yy.ww.zz:9090']

  - job_name: 'node'
    static_configs:
    - targets: ['xx.yy.ww.zz:9100']

where xx.yy.ww.zz is the public ip address of the Azure server.

The job name ‘node’ tells the Prometheus instance to scrape from the Node Exporter via xx.yy.ww.zz:9100

• download , install and startup node exporter

sudo wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz

tar zxvf node_exporter-1.0.1.linux-amd64.tar.gz

cd node_exporter-1.0.1.linux-amd64

./node_exporter

• Access node exporter at http://xx.yy.ww.zz:9100/metrics

• Fireup Prometheus docker container

sudo apt-get install docker.io

sudo docker run --name prometheus -d -p 0.0.0.0:9090:9090 --volume="$PWD/config":/etc/config prom/prometheus --config.file=/etc/config/prometheus.yml

The ip address is entered 0.0.0.0 above in order to facilitate remote access of the server. Its ironic that all content on the web that provides information on setting up Prometheus or for that matter any other tool, is setup to run on localhost or 127.0.0.1. Who would ever run a server on localhost ?

• Fireup Grafana docker container

sudo docker run -d --name=grafana -p 0.0.0.0:3000:3000 grafana/grafana

• Ensure that ports 9090,3000 and 9100 are opened up for inbound traffic in Azure and also in Ubuntu using

sudo ufw allow <port>

• Open up a new browser window and navigate to http://xx.yy.ww.zz:9090 to view the Prometheus console.

• Incase of any issues stop and restart the docker containers using the commands

docker container stop 65dccb7ea5eb970da62f639457bbc28962ac90f7bc84f2fcb10b1fa3a9691aa1

docker container rm 65dccb7ea5eb970da62f639457bbc28962ac90f7bc84f2fcb10b1fa3a9691aa1

• Create a new datasource for Prometheus in Grafana.

• Open Grafana at http://xx.yy.ww.zz:3000 in browser and create a new dashboard.

• Enter node exporter metrics id in the field and click load.
• View the real time metrics generated.