A Beginner’s Guide to Prometheus: Recording Rules, Alerts, and Blackbox Exporter

Prometheus is a powerful monitoring and alerting toolkit that has gained significant popularity in the DevOps community for its simplicity and reliability. In this guide, we’ll dive into some practical examples of three essential components of Prometheus: Recording Rules, Alerts, and Blackbox Exporter. Let’s get started!

Part 1: Understanding Recording Rules

Recording rules in Prometheus allow you to pre-compute frequently needed or computationally expensive expressions and save them as new time series data. This can improve query performance and simplify complex queries.

Step 1: Define a Recording Rule

Let’s say we want to calculate the average response time of our web server. We can define a recording rule for this:

groups:
- name: example
  rules:
  - record: job:http_avg_response_time_seconds
    expr: avg(http_request_duration_seconds)

Step 2: Reload Prometheus Configuration

After defining the recording rule, reload the Prometheus configuration to apply the changes.

Step 3: Query the Recorded Metric

Now you can query the newly recorded metric http_avg_response_time_seconds in Prometheus to get the average response time.

Part 2: Configuring Alerts

Alerting in Prometheus allows you to define conditions based on your metrics and send alerts when these conditions are met. Let’s set up a simple alert for high CPU usage.

Step 1: Define an Alert Rule

groups:
- name: example
  rules:
  - alert: HighCPUUsage
    expr: node_cpu_usage > 90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High CPU usage on instance {{ $labels.instance }}"
      description: "CPU usage is {{ $value }}% on instance {{ $labels.instance }} for more than 5 minutes."

Step 2: Reload Prometheus Configuration

Reload the Prometheus configuration to apply the new alert rule.

Step 3: Test the Alert

Simulate high CPU usage on one of your instances and monitor if the alert triggers in Prometheus.

Part 3: Exploring Blackbox Exporter

Blackbox Exporter allows you to monitor external services and endpoints from Prometheus. It’s useful for probing endpoints over HTTP, HTTPS, TCP, DNS, etc.

Step 1: Configure Blackbox Exporter

First, configure Blackbox Exporter to monitor the endpoints you’re interested in. Here’s a simple configuration for monitoring a web server:

modules:
  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      preferred_ip_protocol: "ip4"
      valid_status_codes: []  # Defaults to 2xx
      fail_if_ssl: false
      fail_if_not_ssl: false

Step 2: Reload Blackbox Exporter Configuration

Reload the Blackbox Exporter configuration to apply the changes.

Step 3: Monitor Endpoints in Prometheus

Query the metrics exposed by Blackbox Exporter in Prometheus to monitor the health and availability of your external services and endpoints.

Conclusion

In this guide, we’ve covered the basics of Recording Rules, Alerts, and Blackbox Exporter in Prometheus with practical examples. These are foundational components that can help you effectively monitor your systems and applications. Experiment with these examples and explore further to unlock the full potential of Prometheus in your monitoring setup!