Oct. 31, 2019, 1:48 p.m.
Posted by soar

Prometheus Swarm Discovery

What is it?

When you are using Docker Swarm as your container orchestration you often need to develop some tools to work with it by it's way. And one of the problems is how to monitor each container running inside a cluster. First thing, that I've tried was a custom Lua-script running on the Nginx/OpenResty balancer to expand whole internal cluster information including containers list, their IP addresses and special endpoints to connect to them. But then I've found a simpler and more elegant solution.

The problem

Despite this tool is named as "proof of concept", it is very well-written application and can be used even for production purposes. But I've faced one major issue with it: memory leaked too quickly. You can see how it was on the main article screenshot above. It was able to eat more then ~4 GB or RAM per day.

Prometheus Swarm Discovery container memory usage


You can find the related PR here and it seems I've fixed all memory leaks in the code, you can see it on the image below. If you want to use the updated version, feel free to clone my repository or use an automatically build image on Quay.io - quay.io/soar/prometheus-swarm-discovery.

Now it's much better