Skip to main content
  1. Posts/

Prometheus Swarm Discovery

·246 words·2 mins· loading · loading ·
DevOps Docker Golang Grafana Prometheus Docker-Swarm

What is it?
#

When you are using Docker Swarm as your container orchestration you often need to develop some tools to work with it by it’s way. And one of the problems is how to monitor each container running inside a cluster. First thing, that I’ve tried was a custom Lua-script running on the Nginx/OpenResty balancer to expand whole internal cluster information including containers list, their IP addresses and special endpoints to connect to them. But then I’ve found a simpler and more elegant solution.

The problem
#

Despite this tool is named as “proof of concept”, it is very well-written application and can be used even for production purposes. But I’ve faced one major issue with it: memory leaked too quickly. You can see how it was on the main article screenshot above. It was able to eat more then ~4 GB or RAM per day.

Memory usage before
Prometheus Swarm Discovery container memory usage before

Solution
#

You can find the related PR here and it seems I’ve fixed all memory leaks in the code, you can see it on the image below. If you want to use the updated version, feel free to clone my repository or use an automatically build image on Quay.io - quay.io/soar/prometheus-swarm-discovery.

The patched version is available here:

Memory usage after
Memory usage after

@soar
Author
@soar
Senior SRE/DevOps engineer

Related

Memory usage as art
·5 words·1 min· loading · loading
Sysadmin Docker
Better nsq_tail
·191 words·1 min· loading · loading
Development DevOps Golang