Prometheus and Grafana Tutorial: Set Up Monitoring and Dashboards

Overview

Prometheus collects metrics by scraping HTTP endpoints, and Grafana turns that data into dashboards and alerts. Together they form the most common open-source monitoring stack. This tutorial walks through installing both, instrumenting an application, and building your first dashboard.

How Prometheus Works

The model is pull-based: Prometheus reaches out to targets on a schedule, rather than the application pushing data. This makes target health easy to reason about.

Step 1: Run Prometheus Locally

Create a directory and a configuration file prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']

Start Prometheus with Docker:

docker run -d --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Open http://localhost:9090 to reach the Prometheus UI. The built-in query console lets you run PromQL expressions directly.

Step 2: Add a Node Exporter

The node exporter exposes CPU, memory, disk, and network metrics from a Linux host.

docker run -d --name node-exporter \
  -p 9100:9100 \
  --pid host \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /:/rootfs:ro \
  prom/node-exporter

Add it to the Prometheus config (already shown above), reload Prometheus, then query:

node_cpu_seconds_total
node_memory_MemAvailable_bytes
node_filesystem_avail_bytes

PromQL Essentials

Example: CPU usage per second, averaged across cores.

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Step 3: Instrument an Application

For a Python Flask app, use the official client library:

pip install prometheus-client flask

from flask import Flask
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time

app = Flask(__name__)

REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency')

@app.before_request
def start_timer():
    request.start_time = time.time()

@app.after_request
def record_metrics(response):
    REQUEST_COUNT.labels(method=request.method, endpoint=request.path).inc()
    REQUEST_LATENCY.observe(time.time() - request.start_time)
    return response

@app.route('/metrics')
def metrics():
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}

@app.route('/')
def index():
    return "Hello"

Add the app to the Prometheus config as a new scrape job, pointing at app:5000/metrics.

Step 4: Install Grafana

docker run -d --name grafana \
  -p 3000:3000 \
  grafana/grafana

Log in at http://localhost:3000 with default credentials admin/admin. Change them immediately.

Step 5: Connect Grafana to Prometheus

1. Go to Connections → Data sources → Add data source.
2. Select Prometheus.
3. Set the URL to http://prometheus:9090 (or http://host.docker.internal:9090 if running separately).
4. Click Save & test.

Building Your First Dashboard

Save the dashboard with a meaningful name and export the JSON if you want to version it in Git.

Alerting Rules

Create alerts.yml:

groups:
  - name: availability
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"

Reference it in prometheus.yml under rule_files, then reload Prometheus. Alerts route through Alertmanager.

Best Practices

• Keep label cardinality low. Do not use user IDs or request IDs as labels.
• Use rate() on counters and raw values on gauges.
• Record expensive queries as recording rules instead of re-running them in every dashboard.
• Set retention and remote-write to long-term storage like Thanos or Mimir for multi-year data.
• Version dashboards as JSON in Git; do not rely on manual edits to production Grafana.