Spike Protection
A spike is a significant, temporary increase in your event volume. Because Sentry bills based on the number of events sent monthly, spikes can quickly consume your
Spike Protection currently applies to errors, transactions, and attachments.
Spike Protection does not apply during trials as category quotas are unlimited for the duration of the trial. For example, new organizations on a trial will not be spike-protected until the trial is over even if projects have Spike Protection enabled.
Notifications for Spike Protection are turned off by default to avoid excessive noise for users. Read the notification section below to learn how to turn them on for key projects.
How Spike Protection Works
Spike Protection can be enabled on a per-
If you're an existing Sentry user with Spike Protection enabled on an org level, you'll be automatically switched to the new per-project level Spike Protection. All your projects will have Spike Protection enabled by default, but you can make adjustments at any time.
How Sentry Detects Spikes
Sentry uses an algorithm to establish a spike threshold for each
Spike thresholds are recalculated every hour for the duration of the spike. This is done to gradually increase thresholds over time to account for new traffic patterns.
Spike Protection will automatically deactivate once the spike event passes, until the next time a spike is detected.
A Dynamic Algorithm
In order to adjust for certain patterns in the number of events your application sends, Sentry's algorithm evaluates your per-
It's done this way so that new projects without 7 days' worth of data can benefit from spike protection. Additionally, the minimum event calculation can be used to minimize false positives in smaller or newer projects so that spikes aren't flagged incorrectly. Read on to learn about our Spike Protection algorithm in detail.
Notifications
While Spike Protection notifications are turned off by default, you can turn them on for key projects you want to monitor in two ways:
- By adding the email “Notification Actions”. This will notify users with Owner, Manager, and Billing roles who are connected with that projectRepresents your service in Sentry and allows you to scope events to a distinct application.via email.
- By adding Slack or Pagerduty “Notification Actions” (if you have these integrations installed). This will notify users who are connected with the project via Slack or Pagerduty.
Managing a Spike
We recommend taking the following steps to manage your spikes:
- Check to see which issues are consuming your quota.
- Set rate limits on the DSNThe Data Source Name (DSN) key tells the Sentry SDK where to send events, ensuring they go to the right project.keys for the projects related to the spike.
- Set up metric alerts for the number of events in a projectRepresents your service in Sentry and allows you to scope events to a distinct application..
- If a specific release version has caused the spike, add the version identifier to the project's inbound filters to avoid accepting events from that release.
- Set up an on-demand budget to make sure you have time to adjust your volume in the event of a future spike.
- Set up spend allocations to ensure your critical projects are guaranteed a portion of your reserved volume, even if there are spikes in other projects.
To review the events that were dropped to save your
To ensure accuracy, your time window on the Stats page must be between a minimum of 6 hours, to a maximum of 30 days for spikes and a spike threshold to appear. In addition, only spikes that are over an hour long will be viewable.
To see spikes that are less than an hour long, visit Sentry's Audit Log by going to the Settings page and selecting "Audit Log" under the "ORGANIZATION" section. You'll be able to see spikes by filtering for spike-protection.activated, spike-protection.deactivated, spike-protection.disabled, or spike-protection.enabled in the "Select Action" dropdown.
How the Spike Protection Algorithm Works
As mentioned above, our Spike Protection Algorithm is based on the higher of two calculations: a minimum event calculation and a usage-based calculation. Read on to learn the details of how it works.
Minimum Event Calculation
This calculation, which is the first step of our algorithm, identifies a minimum number of events, using your
(3 \* your quota)/(720 \* number of projects)
. The number of projects is capped at 5 and is used to adjust the floor. The equation represents your Usage-Based Calculation
This calculation, which is the second step of our algorithm, calculates hourly data from the past seven days to determine spike limit projections for the next seven days. This data is used to calculate weighted averages, which takes into account weekly and hourly seasonality. For example, the weighted average calculated for Monday at 3 pm is more heavily influenced by data points on Monday or the hours around 3 pm. This weighted average is then multiplied by a multiplier that is 5
times the overall standard deviation of the past week — this multiplier is bounded between 3
and 6
.
Example
In this example, the
In the following graph, we can see a zoomed in perspective of the 12-hour period of the spike, along with a line indicating the spike limit as it's being recalculated over the course of the spike:
Throughout the spike, the recalulating limit has the following effect:
- 1st hour: 6k events ingested, limit is recalculated to 2083, 3917 events dropped
- 2nd hour: 34k events ingested, limit is recalculated to 2873, 31217 events dropped
- 3rd hour: 55k events ingested, limit is recalculated to 5452, ~49k events dropped
- 4th hour: 49k events ingested, limit is recalculated to 7628, ~41k events dropped
- 5th hour: 41k events ingested, limit is recalculated to 9371, ~31k events dropped
For this particular example:
- Org quotaThe monthly number of events that you pay Sentry to track.: 500k
- Events ingested during the spike: ~478k
- Events accepted overall: ~157k
Here's an example of spike limit projections for a week, taking into account seasonality:
These regular differences in event ingestion don't cause a spike to occur.
Bursty Projects
There may be instances where a
If this is expected behavior for a given project in your organization, you may want to consider turning off spike protection in the project settings to ensure necessary events aren't dropped.
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").