Repository navigation
configs.monitoring: OOMKill ignores data gaps - #112
Merged
Merged
Conversation
When VictoriaMetrics has no sample shortly before the increase() window (the monitoring server was down, or the series is new), MetricsQL's increase() treats the first sample in the window as an increase from 0 and returns the whole counter. After styx lost power for ~9 minutes, OOMKill fired for coaxmetal with all 5 OOM kills since its boot 49 days ago, although the counter never moved. Only alert when the series has a sample in the minute before the window (two minutes for the 60s smartctl scrape). Then increase() has a real previous value, and still handles counter resets on a monitored host's reboot. increase_pure() does not help: it assumes counters start at 0 and so counts the whole value as well. NvmeMediaErrors uses increase() on a counter the same way and would fire for any drive with past media errors after a gap, so it gets the same guard. MemoryPressure's rate() is not affected: rate() does not count the first sample in the window as an increase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After styx was without power, its VictoriaMetrics had a gap and increase() returned the whole OOM counter of coaxmetal (5), firing OOMKill. Guard with present_over_time before the window; same for NvmeMediaErrors. Tested with vmalert-tool unittest and live against styx's data.