Debugging process_exporter group naming configuration

Introduction

This article provides debugging methods to print out detailed process information collected by process_exporter to help with troubleshooting and developing process group naming configuration.

Documentation: https://github.com/ncabatoff/process-exporter#configuration-and-group-naming

Using ps

You can use this ps command to print relevant process details.

ps -eo pid,comm=Name,exe=EXE,args=Comm

It lists all processes with their PID, command name (comm), executable path (exe) and command line (cmdline) that process_exporter configs can use for all the currently running processes.

Using process_exporter

Follow this example to print out detailed process information exactly as collected by process_exporter.

Write this content in a file named process_exporter.yaml:

process_names:
    # This process group name template concatenates several process properties,
    # separated by a tab character.
  - name: "{{.PID}}\t{{.Comm}}\t{{.ExeBase}}\t{{.ExeFull}}\t{{.Matches.cmdline}}"
    cmdline:
      - '(?P<cmdline>.+)'

Then, start process_exporter like as follows. I installed process_exporter from the apt repository, so the binary is called prometheus-process-exporter. You can also download and run the process-exporter standalone binary from GitHub.
https://github.com/ncabatoff/process-exporter/releases

prometheus-process-exporter \
  -web.listen-address :9257 \
  -config.path process_exporter.yaml

The following shell pipeline scrapes the process_exporter metrics and formats the groupname label value for each process.

The columns are coming from the group name template in process_exporter.yaml:
PID, Comm, ExeBase, ExeFull, cmdline.

curl -s http://localhost:9257/metrics \
| grep "^namedprocess_namegroup_num_procs" \
| sed -E 's/.*groupname="(.*)"}.*/\\1/' \
| sort -n \
| less

Understanding the process_exporter source code

process_exporter uses the prometheus/procfs library to get process information from /proc.
https://github.com/ncabatoff/process-exporter/blob/master/proc/read.go#LL10C3-L10C31
https://github.com/prometheus/procfs

Specifically, it uses the Proc object class in the proc.go module.
https://github.com/prometheus/procfs/blob/master/proc.go

From there, you can check exactly how process_exporter collects process information from the /proc filesystem.