Autoscaling & Performance tracking

[mattdonnelly10]

Is there a cloud autoscaling configuration for Fishnet yet, such as K8s HPA or KEDA? Also, is there an official way to collect container logs to track how many games/positions you've analyzed across containers? When run on the desktop, I get the stats in a json file, but with containers there is just the stdout logs.

[niklasf]

Hi. There's currently no off-the-shelf autoscaling. I've seen two successful ways to do it:

• Run one sentinel instance. Auto-scale based on its CPU usage.
• Auto-scale based on the queue status: https://github.com/lichess-org/fishnet/blob/master/doc/protocol.md#status

The stats file is off by default in containers, but can be enabled with the STATS_FILE environment variable (see docker-entrypoint.sh). We could also consider exposing stats in other/additional ways, if you have suggestions.

[mattdonnelly10]

Thanks, enabling the stats file was a good suggestion, but I decided to just use Promtail, Loki, and Grafana to track all the stats. For the auto-scaling, I used the http-json Prometheus exporter to scrape the queue metrics and then a Kubernetes HPA to auto-scale containers as the queue changes.

Link here in case anyone finds this issue looking for something similar:
https://github.com/mattdonnelly10/fishwatch