Supercharge your blackbox_exporter modules

2019-01-07 2 min read Informatique

I needed to measure DNS resolution performance of two domains from two different locations using three DNS resolvers at each location.

The example configuration demonstrates how to use Prometheus relabel configs to pass multiple targets to one blackbox_exporter. Two web search results (referenced below) hinted on using regexes to pass more than one parameter for each target. I did a little cleanup of the regex pattern and used the @ separator instead of :. The resulting configuration is easy to follow and can be adapted for other blackbox use-cases.

Each DNS query must be configured as separate modules in blackbox.yml. The blackbox.yml content is copied on both blackboxes.

modules:
  dns_udp_host1_cname:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip4
      query_name: "alias.example.com"
      query_type: "CNAME"
      valid_rcodes:
      - NOERROR
      validate_answer_rrs:
        fail_if_not_matches_regexp:
        - "alias.example.com.\t.*\tIN\tCNAME\thost.example.com."
  dns_udp_host1_a:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip4
      query_name: "host.example.com"
      query_type: "A"
      valid_rcodes:
      - NOERROR
      validate_answer_rrs:
        fail_if_not_matches_regexp:
        - "host.example.com.\t.*\tIN\tA\t.*"

This is what the scrape_config in prometheus.yml looks like:

- job_name: dns_probes
  scrape_interval: 60s
  metrics_path: /probe
  static_configs:
    - targets:
      - blackbox1:9115@dns_udp_host1_cname@192.168.13.1:53
      - blackbox1:9115@dns_udp_host1_cname@ns1.example.com:53
      - blackbox1:9115@dns_udp_host1_cname@ns2.example.com:53
      - blackbox1:9115@dns_udp_host1_a@192.168.13.1:53
      - blackbox1:9115@dns_udp_host1_a@ns1.example.com:53
      - blackbox1:9115@dns_udp_host1_a@ns2.example.com:53
      - blackbox2:9115@dns_udp_host1_cname@8.8.8.8:53
      - blackbox2:9115@dns_udp_host1_cname@ns1.example.com:53
      - blackbox2:9115@dns_udp_host1_cname@ns2.example.com:53
      - blackbox2:9115@dns_udp_host1_a@8.8.8.8:53
      - blackbox2:9115@dns_udp_host1_a@ns1.example.com:53
      - blackbox2:9115@dns_udp_host1_a@ns2.example.com:53
  relabel_configs:
    - source_labels: [__address__]
      regex: '(.*)@.*@.*'
      replacement: $1
      target_label: 'instance'      # instance label for Prometheus datapoints
    - source_labels: [__address__]
      regex: '.*@(.*)@.*'
      replacement: $1
      target_label: __param_module  # module parameter to blackbox exporter
    - source_labels: [__address__]
      regex: '.*@(.*)@.*'
      replacement: $1
      target_label: module          # module label for Prometheus datapoints
    - source_labels: [__address__]
      regex: '.*@.*@(.*)'
      replacement: $1
      target_label: __param_target  # target parameter to blackbox exporter
    - source_labels: [__address__]
      regex: '.*@.*@(.*)'
      replacement: $1
      target_label: resolver        # resolver label for Prometheus datapoints
    - source_labels: [__address__]
      regex: '(.*)@.*@.*'
      replacement: $1
      target_label: __address__  # The blackbox exporter's real hostname:port.

This pattern of @ separators, regexes and replacement string is easy to read and expand to pass more labels or parameters.

To debug your DNS queries and validation, you can curl the following URL on your blackbox_exporter:

http://blackbox1:9115/probe?module=dns_udp_host1_a&target=ns1.example.com&debug=true

References

Upstream documentation:

Ideas from community discussion:

sysadmin monitoring prometheus
Alexandre de Verteuil
Alexandre de Verteuil
Senior Solutions Architect

I teach people how to see the matrix metrics.
Monkeys and sunsets make me happy.

Related