Debugging process_exporter group naming configuration

Introduction

This article provides debugging methods to print out detailed process information collected by process_exporter to help with troubleshooting and developing process group naming configuration.

Documentation: https://github.com/ncabatoff/process-exporter#configuration-and-group-naming

Using ps

You can use this ps command to print relevant process details.

ps -eo pid,comm=Name,exe=EXE,args=Comm

It lists all processes with their PID, command name (comm), executable path (exe) and command line (cmdline) that process_exporter configs can use for all the currently running processes.

Using process_exporter

Follow this example to print out detailed process information exactly as collected by process_exporter.

Write this content in a file named process_exporter.yaml:

process_names:
    # This process group name template concatenates several process properties,
    # separated by a tab character.
  - name: "{{.PID}}\t{{.Comm}}\t{{.ExeBase}}\t{{.ExeFull}}\t{{.Matches.cmdline}}"
    cmdline:
      - '(?P<cmdline>.+)'

Then, start process_exporter like as follows. I installed process_exporter from the apt repository, so the binary is called prometheus-process-exporter. You can also download and run the process-exporter standalone binary from GitHub.
https://github.com/ncabatoff/process-exporter/releases

prometheus-process-exporter \
  -web.listen-address :9257 \
  -config.path process_exporter.yaml

The following shell pipeline scrapes the process_exporter metrics and formats the groupname label value for each process.

The columns are coming from the group name template in process_exporter.yaml:
PID, Comm, ExeBase, ExeFull, cmdline.

curl -s http://localhost:9257/metrics \
| grep "^namedprocess_namegroup_num_procs" \
| sed -E 's/.*groupname="(.*)"}.*/\\1/' \
| sort -n \
| less

Using pstree to analyze subprocess relationships

Process exporter has a -children parameter.

-children (default:true) makes it so that any process that otherwise isn’t part of its own group becomes part of the first group found (if any) when walking the process tree upwards. In other words, resource usage of subprocesses is added to their parent’s usage unless the subprocess identifies as a different group name.

See the documentation here:
https://github.com/ncabatoff/process-exporter?tab=readme-ov-file#running

The pstree command can help you analyze subprocesses that would be included in their parents’ group. The following pstree command will list processes with their names and PIDs in a visual tree structure:

pstree -p -T

• -p: show PIDs.
• -T: Hide threads and only show processes.

Understanding the process_exporter source code

process_exporter uses the prometheus/procfs library to get process information from /proc.
https://github.com/ncabatoff/process-exporter/blob/master/proc/read.go#LL10C3-L10C31
https://github.com/prometheus/procfs

Specifically, it uses the Proc object class in the proc.go module.
https://github.com/prometheus/procfs/blob/master/proc.go

From there, you can check exactly how process_exporter collects process information from the /proc filesystem.