Monitoring & Observability:
An Introduction to Prometheus & Grafana

Last updated: April 16, 2025

Table of Contents

1. Why Monitoring & Observability?

In today's complex world of distributed systems, microservices, and cloud infrastructure, understanding the health and performance of your applications is crucial. This is where monitoring and observability come in.

1.1 Monitoring vs. Observability

While often used interchangeably, they represent different approaches:

  • Monitoring: Focuses on tracking the health of systems using predefined metrics and thresholds. It answers known questions like "Is the CPU usage high?" or "Are we running out of disk space?". It typically involves collecting metrics, visualizing them on dashboards, and setting up alerts.
  • Observability: A broader concept referring to the ability to understand the internal state of a system by examining its external outputs – often described by the "three pillars": Metrics (numeric measurements over time), Logs (timestamped records of events), and Traces (tracking requests as they flow through distributed systems). Observability helps answer unknown questions and debug novel issues.

Prometheus and Grafana are foundational tools often used together, primarily focusing on the metrics pillar of observability, which forms the basis of traditional monitoring.

1.2 Importance in Modern Systems

Effective monitoring and observability are essential for:

  • Detecting and diagnosing problems quickly.
  • Understanding system performance and bottlenecks.
  • Ensuring reliability and availability (SRE practices).
  • Making informed decisions about scaling and capacity planning.
  • Validating the impact of changes and deployments.

2. Introducing Prometheus

2.1 What is Prometheus?

Prometheus is a powerful, open-source systems monitoring and alerting toolkit originally built at SoundCloud and now a graduated project of the Cloud Native Computing Foundation (CNCF). It focuses on collecting and storing time-series data (metrics with timestamps) and provides a flexible query language (PromQL) and built-in alerting capabilities.

2.2 Core Architecture

The main components include:

  • Prometheus Server: The core component that scrapes (fetches) metrics from configured targets, stores the data in its time-series database (TSDB), and evaluates alerting rules.
  • Exporters: These are agents deployed alongside the applications or systems you want to monitor. They expose metrics in a format Prometheus understands, typically over an HTTP endpoint (e.g., /metrics). Common examples include node_exporter (for OS/hardware metrics) and exporters for databases, message queues, etc.
  • Client Libraries: For instrumenting your own application code to expose custom metrics directly.
  • Pushgateway: An intermediary gateway for short-lived jobs (like batch scripts) that can't be scraped directly, allowing them to push their metrics to it, which Prometheus then scrapes.
  • Alertmanager: Handles alerts fired by the Prometheus server. It deduplicates, groups, and routes alerts to various receivers like email, Slack, PagerDuty, etc.

2.3 Data Model

Prometheus stores data as time series. Each time series is uniquely identified by:

  • A metric name (e.g., http_requests_total, cpu_usage_percent).
  • A set of key-value pairs called labels (e.g., {instance="server1", method="POST", path="/api/users"}).

This multi-dimensional data model based on metric names and labels is highly flexible for querying and aggregation.

2.4 Pull vs. Push

Prometheus primarily uses a pull model. The Prometheus server periodically sends HTTP requests to scrape metrics from configured targets (exporters). This simplifies configuration (targets don't need to know where Prometheus is) and makes it easy to determine target health (if a scrape fails, the target is likely down). The Pushgateway provides an exception for scenarios where pulling is not feasible.

3. Querying with PromQL

3.1 Introduction to PromQL

Prometheus includes a powerful functional query language called PromQL (Prometheus Query Language). You use PromQL to select, aggregate, and transform time-series data stored in Prometheus. It's used for building dashboards (in Grafana) and defining alerting rules.

3.2 Basic Query Examples

  • Select all time series with the metric name http_requests_total: 
    http_requests_total
  • Select http_requests_total only for the job named api-server and method GET: 
    http_requests_total{job="api-server", method="GET"}
  • Select metrics within a specific time range (e.g., last 5 minutes): 
    http_requests_total{job="api-server"}[5m]
    This returns a range vector.

3.3 Aggregation Operators

PromQL provides functions and operators for aggregation and calculations:

  • Calculate the sum of all HTTP requests across all instances: 
    sum(http_requests_total)
  • Calculate the average CPU usage percent across instances with the label env="production": 
    avg(cpu_usage_percent{env="production"})
  • Calculate the per-second rate of HTTP requests over the last 5 minutes: 
    rate(http_requests_total[5m])

4. Introducing Grafana

4.1 What is Grafana?

Grafana is the leading open-source platform for data visualization, monitoring, and analysis. While Prometheus excels at collecting and storing metrics, Grafana provides the user interface for exploring, visualizing, and understanding that data through beautiful and interactive dashboards.

4.2 Connecting Prometheus as a Data Source

Grafana supports a wide variety of data sources, including Prometheus, Loki (for logs), Elasticsearch, InfluxDB, SQL databases, and more. Adding Prometheus as a data source is straightforward: you just need to provide the URL of your Prometheus server (e.g., http://prometheus-server:9090).

4.3 Building Dashboards

Grafana dashboards are composed of panels, each displaying data from one or more queries.

  • Panels: Choose from various visualization types like time series graphs, gauges, bar charts, tables, heatmaps, single stats, etc.
  • Queries: Within each panel, you write queries (using PromQL when connected to Prometheus) to fetch the data you want to visualize.
  • Variables: Create dynamic dashboards using template variables (e.g., select different servers, environments, or services from dropdowns).
  • Alerting: Grafana also has its own alerting engine that can trigger alerts based on query results from various data sources.

5. Prometheus & Grafana Working Together

5.1 The Typical Workflow

The combination forms a robust monitoring stack:

  1. Collect: Exporters run on your targets, exposing metrics.
  2. Store: Prometheus scrapes these exporters at regular intervals and stores the metrics in its TSDB.
  3. Query: Users or Grafana use PromQL to query the Prometheus server for specific metrics.
  4. Visualize: Grafana fetches the query results and displays them in informative dashboards.
  5. Alert: Prometheus evaluates predefined alerting rules based on PromQL expressions. If an alert fires, it sends it to Alertmanager, which then routes it to the appropriate notification channels (configured separately).

5.2 Example Scenario (Monitoring a Web Server)

  1. Install node_exporter on the web server host.
  2. Configure Prometheus to scrape the node_exporter endpoint (e.g., http://web-server-ip:9100/metrics).
  3. Add Prometheus as a data source in Grafana.
  4. Create a Grafana dashboard with panels querying Prometheus for metrics like:
    • CPU Usage: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)
    • Memory Usage: node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
    • Disk I/O: rate(node_disk_read_bytes_total[1m]), rate(node_disk_written_bytes_total[1m])
    • Network Traffic: rate(node_network_receive_bytes_total[1m]), rate(node_network_transmit_bytes_total[1m])
  5. (Optional) Set up alerting rules in Prometheus (e.g., alert if CPU usage is over 80% for 5 minutes) and configure Alertmanager to send notifications.

6. Getting Started & Next Steps

6.1 Installation Overview

Prometheus and Grafana can be installed in various ways:

  • Docker/Docker Compose: A popular and easy way to run them locally or in production. Official Docker images are available.
  • Native Binaries: Download pre-compiled binaries for your operating system.
  • Package Managers: Available in many Linux distribution repositories (e.g., via apt or yum).
  • Kubernetes: Often deployed using Helm charts or Operators within Kubernetes clusters.

Configuration for Prometheus is typically done via a YAML file (prometheus.yml) where you define scrape targets and alerting rules.

# Example prometheus.yml snippet
global:
  scrape_interval: 15s # Default scrape interval

scrape_configs:
  - job_name: 'prometheus' # Job monitoring Prometheus itself
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter' # Job for monitoring servers
    static_configs:
      - targets: ['server1.example.com:9100', 'server2.example.com:9100']

6.2 Finding Exporters

A key part of using Prometheus is finding or building exporters for the systems you need to monitor. Many official and community-maintained exporters exist for common databases, message queues, hardware, APIs, and more. Check the official Prometheus documentation and GitHub for available options. If an exporter doesn't exist, you can often use client libraries (Go, Java, Python, Ruby, etc.) to instrument your application code directly.

6.3 Exploring Alertmanager

While Prometheus *defines* and *fires* alerts, Alertmanager is responsible for processing and routing them effectively. It's a separate component that needs configuration (alertmanager.yml) to define routing rules and receiver integrations (email, Slack, PagerDuty, etc.).

7. Conclusion

Prometheus and Grafana form a powerful, industry-standard combination for metrics-based monitoring and observability. Prometheus provides efficient time-series data collection, storage, and a flexible query language (PromQL), while Grafana offers best-in-class visualization and dashboarding capabilities. Together, they enable developers and SREs to gain deep insights into system health and performance, facilitating faster troubleshooting and more reliable operations. Starting with this duo provides a solid foundation for building comprehensive monitoring solutions.

8. Additional Resources

Related Articles

External Resources