Alert Policies define alert conditions across your system.
For example, you can have a separate policies for Production and Dev instances.
Violations are created when a threshold is increased.
There are 2 types of violations:
- Warning Violations
- Critical Violations
Warning Violations do not generate any notifications. They mostly serve as an indicator that you might need to look into something.
Incidents group alert events across your systems. They also help control the amount of notifications you get.
Incidents are started when atleast one Critical Violation occurs. All Critical Violations opened from then are then placed into that same incident.
Once all the Critical Violations close, that incident is closed too.
Looking inside an Incident, you can see all the violations across the various applications that caused it.
You are only notified when an Incident begins and when it stops, not for every single violation inside the incident. This prevents you from getting bombarded with notifications.
You can configure various integrations to be notified on the channel you wish, eg Slack, PagerDuty, Webhook, etc.