Track eth-docker version in metrics#2662
Conversation
|
Interesting, and. When running multiple Eth Docker, which is advanced, the intended way to get metrics is to also lean into an advanced setup. Something like central-proxy-docker for scraping (if they migrated to alloy already), and a central visibility stack (Grafana, [Prometheus|VictoriaMetrics|Mimir|Thanos], Tempo, Loki) to remote write it all to. Having several Grafana stacks running is decidedly sub-optimal. I'll need to think on whether I want to support that - it wouldn't hurt as long as that doesn't introduce issues for people who don't, and, it's not "the right way" to do visibility.
That'd be confusing when not pinned. Why not grab the version from README.md? That way the interim |
What I did
Scrape eth-docker version so it can be tracked along with EL and CL versions
Motivation
When managing a significant number of nodes it can be easy to lose track of what needs to be updated. When creating Grafana dashboards it would be nice to have access to the eth-docker version to spot any instances that are behind others. Releases can require migration steps, and being able to see all your eth-docker versions on a dashboard is useful information.
Changes
For the scraping:
ethdnow writes the checked-out tag (viagit describe --tags --abbrev=0, so it still resolves to the latest release tag even when trackingmainrather than a pinned release) and commit as aneth_docker_version_info{tag="...",commit="..."}gauge, through node-exporter's existing-but-unused textfile collector.NODE_EXPORTER_COLLECTOR_MOUNT_PATH's default now points at a real, eth-docker-managed directory instead of an inert placeholder, so this works with no.envchanges required.After implementing scraping, testing with multiple eth-docker instances on one host surfaced an existing issue: Alloy's
discovery.dockerreads the host Docker socket, so it discovers every container on the host, not just its own project's. Most foreign containers are still unreachable (different Docker network) so this was harmless - except node-exporter, which runs withnetwork_mode: hostand ametrics.host=host.docker.internallabel specifically so Alloy can reach it cross-network. That same override makes it reachable from every instance's Alloy, so each instance's Prometheus ends up storing all instances' node-exporter metrics, correctly labeled but mixed together - which caused two differenteth_docker_version_infoseries.To fix this:
ethdnow pinsCOMPOSE_PROJECT_NAMEin.envto whatever Compose already resolves it to (read back fromdocker compose config, never computed independently), andconfig.alloyscopes both metrics and logs discovery to containers in this instance's own project - falling back to today's unscoped behavior if that resolution ever fails, rather than discovering nothing. This fixes cross-instance bleed for every metric and log, not just the new one.