Monitors
A monitor is a proactive mechanism for tracking changes in your data and alerting you when certain conditions are met.
Monitors in Streamdal are set up based on rules that define the conditions under which an alert should be triggered. Different types of monitors allow you to track various aspects of your data, including schema changes, data flow rates, semantic data conditions, and more.
Types of Monitors
Streamdal provides a variety of monitors to cater to your specific data monitoring needs:
-
Schema Change Monitors: These monitors detect changes to the schema of collections from an inbound event source. Schema changes can happen when a third-party is used as an event source or an internal change affects how events are produced. Schema Evolution and Schema Conflict are the two types of schema changes that can be monitored.
-
Data Flow Monitors: These monitors focus on the rate of data flow in terms of the size of events (bytes), number of events, and errors. They are used to ensure data is flowing at expected rates and to detect any anomalies or issues.
-
Semantic Data Monitors: These monitors validate the semantics of data by checking the consistency and validity of specific fields in the data. The ‘Field’ rule lets you check a field value for conditions such as string contains, is empty, equal to, boolean true, and PII fields values such as credit card and social security numbers.
-
Dead Letter Monitors: These monitors track events that end up in the Dead Letter due to bad schemas or incorrect encodings. Alerts are triggered when an event with a bad schema or encoding is detected.
Configuring Monitors
Configuring a monitor in Streamdal involves selecting a type of monitor, defining the rule or rules for when an alert should be triggered, and attaching an alert to the monitor. Depending on the type of monitor, additional configuration may be necessary, such as specifying the field to monitor for Semantic Data Monitors.
Alerts
Alerts are the way that monitors notify you when a condition defined by a rule is met. This could be an email, Pagerduty, a message to a Slack channel, or any other form of notification supported by Streamdal. You can attach one or more alerts to a monitor.
Monitor Efficacy
The effectiveness of monitors in Streamdal depends on the accurate configuration of rules and alerts. Incorrectly configured rules or alerts that are not properly set up can result in missed issues or false alarms.
It’s also essential to remember that monitors are only as good as the data they have access to. Ensuring that data is correctly ingested into Streamdal and that schema changes are accurately reflected in the system will enable monitors to work as expected and provide valuable insights into your data.