Prometheus Blackbox Exporter with kube-prometheus-stack

Andrii Abramov

Published in

DevOps.dev

2 min readAug 26, 2021

In this story, I will describe how to set up monitoring of external services using Prometheus Blackbox Exporter.

The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP.

Prerequisites:

Existing k8s cluster
helm installed (if not, see Installing Helm)

Install Grafana / Prometheus

I suggest using kube-prometheus-stack as it is the simplest way to get Prometheus and Grafana running with awesome preinstalled dashboards for k8s monitoring:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts$ helm repo update# You can choose custom release name instead of 'prom'
$ helm install prom prometheus-community/kube-prometheus-stack

Install Blackbox Exporter

Again, let’s use the existing Helm chart for Blackbox Exporter:

# If you haven't already added this repository yet
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update# You can choose custom release name instead of 'blackbox-exporter'
# Note: blackbox-exporter DNS depends on release name
$ helm install blackbox-exporter prometheus-community/prometheus-blackbox-exporter

Tell Prometheus to scrape probes from Exporter

Let’s say we would like to monitor the most important resources for each developer:

In order to start probing, you need to update values.yaml or provide them using your own values file for kube-prometheus-stack:

additionalScrapeConfigs:
  - job_name: blackbox
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      # Add URLs as target parameter
      - targets:
        - https://example.com
        - https://www.google.com
        - https://stackoverflow.com
        - https://scala-lang.org
        - https://helm.sh
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      # Important!     
      target_label: target
      # Ensure blackbox-exporter is reachable from Prometheus
    - target_label: __address__ 
      replacement: prometheus-blackbox-exporter.monitoring:9115

After applying this update, Prometheus will start scraping metrics from blackbox-exporter which will “forward” this probe to the target.

Visualizing metrics in Grafana

I am using an existing dashboard that provides an overview for each target endpoint: https://grafana.com/grafana/dashboards/14928 (spoiler, I am the author).

How to import public dashboard:

Open Grafana
On the left, click “+” -> “Import”
Paste 14928 as ID
Select Prometheus instance to use (if there is no Prometheus Datasource added, create a new one)

And voila! You have everything you need to know about your targets.

Conclusion

In this post, we have set up monitoring of external services using http_2xx module from Prometheus Blackbox Exporter. Further, you can add custom thresholds and alerts. But Blackbox Exporter is designed not only for external services, you can easily probe your internal local services, e.g. in k8s cluster.