Create a Homelab Uptime Dashboard with Prometheus, Blackbox Exporter, and Grafana
Build a small service-health Lab where Prometheus sends HTTP probes through Blackbox Exporter, Grafana visualizes probe_success and latency, and a disposable web service provides...
Expected Outcome
A working endpoint-health dashboard that proves a service is reachable from the monitoring stack, records probe latency, and visibly transitions from healthy to failed and back during a controlled outage test.
Assumptions
A Linux host with Docker Engine and Docker Compose v2.
At least 2 GB of free RAM for Prometheus, Grafana, Blackbox Exporter, and the disposable test service.
A browser that can reach Grafana on the Lab host.
This Lab measures endpoint availability; host CPU, memory, disk, and hardware metrics are a separate Node Exporter use case.
Bill of Materials
Prometheus container image at a reviewed tag.
Prometheus Blackbox Exporter container image at a reviewed tag.
Grafana container image at a reviewed tag.
A small Nginx container used only as the disposable HTTP probe target.
Build Steps
- Verify Docker Compose v2 and create the Lab directory
Use the current Compose plugin rather than the retired standalone docker-compose executable.
Changes system state: review before running
docker --version docker compose version mkdir -p ~/service-health-lab/{prometheus,blackbox} && cd ~/service-health-lab - Define the Blackbox HTTP probe module
Blackbox Exporter is a multi-target exporter: Prometheus asks it to probe another endpoint and receives metrics about that probe. Define a simple HTTP module that expects an HTTP 2xx response.
Read-only command: verify target and scope
cat > blackbox/blackbox.yml <<'EOF' modules: http_2xx: prober: http timeout: 5s http: preferred_ip_protocol: ip4 EOF - Configure Prometheus to probe the disposable service through Blackbox Exporter
The relabeling pattern passes the original service URL to Blackbox Exporter's /probe endpoint and then directs the scrape to the exporter container. Prometheus should not scrape localhost:9100 and call that a remote service-health check; inside a container, localhost is the Prometheus container itself.
Read-only command: verify target and scope
cat > prometheus/prometheus.yml <<'EOF' global: scrape_interval: 15s scrape_configs: - job_name: blackbox-http metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - http://test-service/ relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115 EOF - Define the monitoring stack
Replace each <reviewed-tag> placeholder with a current tag you have deliberately selected before deployment. The disposable Nginx target gives the Lab a safe service that can be stopped without affecting real infrastructure.
Read-only command: verify target and scope
cat > compose.yaml <<'EOF' services: test-service: image: nginx:<reviewed-tag> restart: unless-stopped blackbox-exporter: image: quay.io/prometheus/blackbox-exporter:<reviewed-tag> restart: unless-stopped command: - --config.file=/etc/blackbox_exporter/config.yml volumes: - ./blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro prometheus: image: prom/prometheus:<reviewed-tag> restart: unless-stopped command: - --config.file=/etc/prometheus/prometheus.yml volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - prometheus-data:/prometheus ports: - "127.0.0.1:9090:9090" depends_on: - blackbox-exporter - test-service grafana: image: grafana/grafana:<reviewed-tag> restart: unless-stopped volumes: - grafana-data:/var/lib/grafana ports: - "3000:3000" depends_on: - prometheus volumes: prometheus-data: grafana-data: EOF docker compose config --quiet - Start the stack and validate the raw probe path
Bring up all four services. Query Blackbox Exporter directly first so probe configuration problems can be separated from Prometheus scrape problems.
Changes system state: review before running
docker compose up -d docker compose ps docker compose exec -T prometheus wget -qO- 'http://blackbox-exporter:9115/probe?target=http://test-service/&module=http_2xx' | grep -E '^probe_(success|duration_seconds)'
- Validate Prometheus target health
Query Prometheus for the Blackbox metric instead of assuming a configured scrape target is working.
Read-only command: verify target and scope
curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=probe_success'
- Build the Grafana service-health dashboard
Open Grafana at http://<lab-host>:3000, add Prometheus at http://prometheus:9090, and create a stat panel using probe_success plus a time-series panel using probe_duration_seconds. A value of 1 means the probe succeeded; 0 means the probe failed.
- Run a controlled outage and recovery test
Stop only the disposable Nginx target. Wait for at least one Prometheus scrape interval and confirm the metric falls to zero. Start the target again and verify the metric returns to one. This proves the dashboard reacts to a real endpoint-state transition rather than simply displaying static container status.
Read-only command: verify target and scope
docker compose stop test-service
Manual or UI step
sleep 20
Read-only command: verify target and scope
curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=probe_success' docker compose start test-service
Manual or UI step
sleep 20
Read-only command: verify target and scope
curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=probe_success'
- Keep host telemetry separate from service uptime
If you later need host CPU, memory, filesystem, or kernel metrics, add Node Exporter as a separate job. Prometheus documents Node Exporter as a host hardware/OS exporter; when containerized for host monitoring it requires host namespace/path handling. Do not treat Node Exporter as a substitute for HTTP/TCP endpoint probes.
Validation
The Compose file parses successfully with Docker Compose v2 and no standalone `docker-compose` executable is required.
A direct Blackbox probe of `http://test-service/` returns `probe_success 1` while the service is running.
Prometheus stores `probe_success` for the test-service URL and Grafana reads those metrics from Prometheus.
Stopping the disposable service causes `probe_success` to become 0 and Grafana shows the failed state.
Restarting the service causes `probe_success` to return to 1.
The operator can explain why endpoint probes and Node Exporter host metrics answer different operational questions.
Troubleshooting
If the direct Blackbox probe fails, inspect exporter logs and confirm `test-service` resolves on the Compose network before changing Prometheus.
Read-only command: verify target and scope
docker compose logs --tail=120 blackbox-exporter docker compose exec -T prometheus getent hosts test-service
If the direct probe succeeds but Prometheus has no metric, inspect the target and relabel configuration rather than changing the exporter module.
Read-only command: verify target and scope
curl -fsS http://127.0.0.1:9090/api/v1/targets
If Grafana shows no data, query `probe_success` directly in Prometheus first. Fix the data source or PromQL only after Prometheus contains the expected series.
Cleanup or Rollback
Always restart the disposable test service if you want the Lab to remain in a healthy state.
Stop the Lab with `docker compose down`; keep named volumes if you want to preserve Grafana dashboards and Prometheus history.
only when you have explicitly decided to destroy the Lab's Prometheus/Grafana data.
Changes system state: review before running
docker compose down -v
Next Improvements
Add a TCP or DNS Blackbox module to compare different service-health checks.
Replace the disposable endpoint with one non-critical real service only after the probe and alert boundaries are understood.
Add Node Exporter separately if you need host-resource telemetry in addition to endpoint availability.