Saved Filter Notifications
Alerts and notifications have been part of TrackJS since the very beginning. Our standard notification options reflect our desire to keep things simple. Over time though, our customers have asked to customize their alerts and fine tune them to specific scenarios. To support that use case, we’re releasing a new kind of notification we’re calling “Saved Filter Notifications”.
New Filter Based Alerting
Some customers use TrackJS as an early warning system - if an error with a specific message shows up, it means big trouble and they want to know about it right away. Other customers want to know if their APIs are acting flaky. And some want to know if their most important users are having trouble.
There’s dozens of reasons you might want to know when certain errors show up (or start happening more frequently). To that end we’ve recently launched a new filter based notification system. Now you can tell us which errors you care about, and we can tell you when they start happening!
Saving a Filter
To make use of the new custom notifications, the first thing you need to do is create a filter for the errors you care about. You can filter based on error message, entry type, URL, user and myriad other facets. It’s easy - just click on the global filter bar at the top of the screen and make your selections.
After you’ve filtered to the relevant errors, click the “share” dropdown in the upper right hand corner of the UI. A menu will appear and you can enter a name for your filter. Click the Save button.
Configuring Alerts for a Saved Filter
Once saved, you can click the Manage Alerts link from the same “share” dropdown. You’ll see a list of all your saved filters, and any pre-existing notification channels.
Find the saved filter you want to configure alerts for and click the “+ Add Notification” link.
Customizing Notifications
The new saved filter notifications work using four different strategies (more on that below). Those strategies are based on a Lookback Period and an Error Threshold.
The Lookback Period is how large of a time window to use when considering a filter’s error volume. The Error Threshold is the minimum number of errors that must occur during the lookback period before an alert is sent.
For example, if a filter has a high base level of errors, it might make sense to set the error threshold high, and a short lookback window. This way you’ll need a decent spike in a short period of time to cause an alert.
On the other hand, if there’s a situation where you want an alert any time an error matches a filter - a longer window with a lower threshold will make the alert more sensitive.
And, like our other notifications, you can notify as many channels at a time as you’d like.
How it Works
We use four different strategies when determining whether to alert. Each strategy will decide to trigger or not based on the filter’s error volume over time. Two of the strategies must be triggered in order for a notification to be sent (this cuts down on noise and false positives). We use the following strategies:
- Simple Historical Average: This simple strategy compares the filter’s historical average error volume to the lookback period’s current volume and, if the current volume exceeds a multiple, triggers.
- Standard Deviation: This strategy triggers if a filter’s current error volume is multiple standard deviations higher than the historical volume.
- Moving Average: If the moving average of the error volume in the lookback period is several multiples higher than the historical volume, the strategy is triggered.
- ML Detection Strategy: An ML prediction strategy which finds spikes in time series data based on adaptive kernel density estimations and martingale scores. If the confidence is over 99.9% the strategy triggers.
Conclusion
With the new saved filter based notifications, customers can pick and choose when and why they’re alerted. Filters can be broad, and capture many errors - or very granular and only trigger in rare conditions. No matter your use case, it’s likely that a well chosen filter with proper alert settings will ensure you’re notified when anything goes wrong.
We’re always on the lookout for new ways to improve customer experience, so if you’ve got suggestions please let us know!
