diff --git a/develop-docs/application-architecture/dynamic-sampling/extrapolation.mdx b/develop-docs/application-architecture/dynamic-sampling/extrapolation.mdx index 35b8994b47aef7..1494b9c70181b9 100644 --- a/develop-docs/application-architecture/dynamic-sampling/extrapolation.mdx +++ b/develop-docs/application-architecture/dynamic-sampling/extrapolation.mdx @@ -3,34 +3,39 @@ title: Extrapolation sidebar_order: 5 --- -Dynamic sampling reduces the amount of data ingested, for reasons of both performance and cost. When configured, a fraction of the data is ingested according to the specified sample rate of a project: if you sample at 10% and initially have 1000 requests to your site in a given timeframe, you will only see 100 spans in Sentry. Without making up for the sample rate, any metrics derived from these spans will misrepresent the true volume of the application. When different parts of the application have different sample rates, there will even be a bias towards some of them, skewing the total volume towards parts with higher sample rates. This bias especially impacts numerical attributes like latency, reducing their accuracy. To account for this fact, Sentry uses extrapolation to smartly combine the data to account for sample rates. +[Dynamic sampling](/application-architecture/dynamic-sampling) reduces the amount of data ingested, to help with both performance and cost. When configured, a fraction of the data is ingested according to the specified sample rates within a project. For example, if you sample 10% of 1000 requests to your site in a given timeframe, you will see 100 spans in Sentry. -### Accuracy & Expressiveness -What happens during extrapolation, how does one handle this type of data, and when is extrapolated data accurate and expressive? Let's start with some definitions: +Without accounting for the lower request volume due to the sample rate, any metrics derived from these spans will misrepresent the true volume of the application. Perhaps more importantly, when different parts of your application have different sample rates, attention may be skewed with a bias towards parts with higher sample rates. This bias especially impacts numerical attributes like latency, reducing their accuracy. To account for this, Sentry uses extrapolation to a) derive a "true" volume of each part of your application and b) combine the extrapolated data from different parts of the application to provide a more wholistic view of the application's performance. + +### Accuracy & Usefulness +What happens during extrapolation? how does Sentry handle this type of data? And when is extrapolated data accurate and useful? Our goal is to make data _accurate_ and _useful_ when reviewing metrics and alerts. Let's define these terms: - **Accuracy** refers to data being correct. For example, the measured number of spans corresponds to the actual number of spans that were executed. As sample rates decrease, accuracy also goes down because minor random decisions can influence the result in major ways. -- **Expressiveness** refers to data being able to express something about the state of the observed system. Expressiveness refers to the usefulness of the data for the user in a specific use case. +- **Usefulness** refers to data being able to express something about the state of the observed system, and the value of the data for the user in a specific use case. For example, a metric that shows the P90 latency of your application is useful for understanding the performance of your application, but a metric that shows the P90 latency of different endpoints in your application sampled at 10%, 1%, and 5% is not as useful because it is not a complete picture. ### Modes -Given these properties, there are two modes that can be used to view data in Sentry: default mode and sample mode. +Given these objectives, there are two modes that can be used to view data: default mode and sample mode. -- **Default mode** extrapolates the ingested data as outlined below. -- **Sample mode** does not extrapolate and presents exactly the data that was ingested. +- **Default mode** extrapolates the ingested data as outlined below - targeting usefulness. +- **Sample mode** does not extrapolate and presents exactly the data that was ingested - targeting accuracy, especially for small datasets. -Depending on the context and the use case, one mode may be more useful than the other. Generally, default mode is useful for all queries that aggregate on a dataset of sufficient volume. As absolute sample size decreases below a certain limit, default mode becomes less and less expressive. There are scenarios where the user needs to temporarily switch between modes, for example, to examine the aggregate numbers first and dive into the number of samples for investigation. In both modes, the user may investigate single samples to dig deeper into the details. +Depending on the context and the use case, one mode may be better suited than the other. Generally, default mode is useful for all queries that aggregate on a dataset of sufficient volume. As absolute sample size decreases below a certain limit, default mode becomes less and less useful. There are scenarios where you may need to temporarily switch between modes, for example, to examine the aggregate numbers first and dive into the number of samples for investigation. In both modes, you may investigate single samples to dig deeper into the details. ### Benefits of Extrapolation -At first glance, extrapolation may seem unnecessarily complicated. However, for high-volume organizations, sampling is a way to control costs and egress volume, as well as reduce the amount of redundant data sent to Sentry. Why don't we just show the user the data they send? We don't just extrapolate for fun; it actually has some major benefits for the user: +At first glance, extrapolation may seem unnecessarily complicated. However, for high-volume organizations, sampling is a way to control costs and reduce volume, as well as reduce the amount of redundant data sent to Sentry. Here are some of the benefits of extrapolation: + +- **The numbers correspond to the real world**: When data is sampled, there is some math you need to do to infer what the real numbers are, e.g., when you have 1000 samples at 10% sample rate, there are 10000 requests to your application. With extrapolation, you don't have to know your sample rate to understand what your application is actually doing. Instead, while viewing charts, you see the real behavior without additional knowledge or math required on your end. -- **The numbers correspond to the real world**: When data is sampled, there is some math you need to do to infer what the real numbers are, e.g., when you have 1000 samples at 10% sample rate, there are 10000 requests to your application. With extrapolation, you don't have to know your sample rate to understand what your application is actually doing. Instead, you get a view of the real behavior without additional knowledge or math required on your end. +- **Steady timeseries when sample rates change**: Whenever you change sample rates, both the count and possibly the distribution of the values will change in some way. When you switch the sample rate from 10% to 1% for whatever reason, there will be a sudden change in all associated metrics. Extrapolation corrects for this, so your graphs are steady, and your alerts track on the same data, regardless of the sample rate. -- **Steady timeseries when sample rates change**: Whenever you change sample rates, both the count and possibly the distribution of the values will change in some way. When you switch the sample rate from 10% to 1% for whatever reason, there will be a sudden change in all associated metrics. Extrapolation corrects for this, so your graphs are steady, and your alerts don't fire when this happens. -- **Combining different sample rates**: When your endpoints don't have the same sample rate, how are you supposed to know the true p90 when one of your endpoints is sampled at 1% and another at 100%, but all you get is the aggregate of the samples? +- **Combining different sample rates**: When your endpoints don't have the same sample rate, how are you supposed to know the true p90 when one of your endpoints is sampled at 1% and another at 100%, but all you get is the aggregate of the samples? Extrapolation calculates the true p90 by combining the data from all endpoints, weighted by the sample rate. ## How Does Extrapolation Work? +![extrapolation =1000x](./images/extrapolated_data_chart.png) + ### Aggregates -Sentry allows the user to aggregate data in different ways - the following aggregates are generally available, along with whether they are extrapolatable or not: +Sentry allows you to aggregate data in different ways - the following aggregates are generally available, along with whether they are extrapolatable or not: | **Aggregate** | **Can be extrapolated?** | | --- | --- | @@ -42,12 +47,12 @@ Sentry allows the user to aggregate data in different ways - the following aggre | max | no | | count_unique | no | -Each of these aggregates has its own way of dealing with extrapolation, due to the fact that e.g., counts have to be extrapolated in a slightly different way from percentiles. While `min` and `max` are technically percentiles, we currently do not offer extrapolation due to the decreased stability of extreme aggregates when sampling. For example, the `p50` will also be more stable than the `p99`; the `min` and `max` are just extreme cases. +Each of these aggregates has its own way of dealing with extrapolation, due to the fact that, for example, counts have to be extrapolated in a slightly different way from percentiles. While `min` and `max` are technically percentiles, we currently do not offer extrapolation due to the decreased stability of extreme aggregates when sampling. For example, the `p50` will also be more stable than the `p99`; the `min` and `max` are just extreme cases. ### Extrapolation for different aggregates To extrapolate, sampling weights are calculated as `1/sample rate`. The sampling weights of each row are then used in the following ways: -- **Count**: Calculate a sum of the sampling weight +- **Count**: Calculate a sum of the sampling weight. Example: the query `count()` becomes `round(sum(sampling_weight))`. - **Sum**: Multiply each value with `sampling_weight`. Example: the query `sum(foo)` becomes `sum(foo * sampling_weight)` @@ -56,7 +61,7 @@ Example: the query `avg(foo)` becomes `avgWeighted(foo, sampling_weight)` - **Percentiles**: Calculate the weighted percentiles with sampling weight. Example: the query `percentile(0.95)(foo)` becomes `weightedPercentile(0.95)(foo, sampling_weight)`. -As long as there are sufficient samples, the sample rate itself does not matter as much, but due to the extrapolation mechanism, what would be a fluctuation of a few samples may turn into a much larger absolute impact, e.g., in terms of the view count. Of course, when a site gets billions of visits, a fluctuation of 100,000 via the noise introduced by a sample rate of 0.00001 is not as critical. +As long as there are sufficient samples, the sample rate itself does not matter as much, but due to the extrapolation mechanism, a fluctuation of a few samples in a small sample set may turn into a much larger absolute impact. When a site gets billions of visits, a fluctuation of 100,000 via the noise introduced by a sample rate of 0.00001 is not as critical. ## How to deal with extrapolation in the product? @@ -73,19 +78,19 @@ In new product surfaces, the question of whether to use extrapolated vs. non-ext - Some scenarios, like visualizing metrics over time, are based on aggregates, whereas a case of debugging a specific user's problem hinges on actually seeing the specific events. The best mode depends on the intended usage of the product. ### Switching to Sample Mode -Sample mode is designed to help users investigate specific events. Here are two common scenarios where it makes the most sense: +Sample mode is designed to help you investigate specific events. Here are two common scenarios where it makes the most sense to use: -1. **When both sample rate and event volume are low**: Extrapolation becomes less reliable in these cases. You can either increase your sample rate to improve accuracy, or switch to sample mode to examine the actual events - both are valid approaches depending on the user's needs. +1. **When both sample rate and event volume are low**: Extrapolation becomes less reliable in these cases. You can either increase your sample rate to improve accuracy, or switch to sample mode to examine the actual events - both are valid approaches depending on your needs. 2. **When you have a high sample rate but still see low event volumes**: In this case, increasing the sample rate won't help capture more data, and sample mode will give you a clearer picture of the events you do have. ### Opting Out of Extrapolation -Users may want to opt out of extrapolation for different reasons. It is always possible to set the sample rate for specific events to 100% and therefore send all data to Sentry, implicitly opting out of extrapolation and behaving in the same way as sample mode. Depending on their configuration, users may need to change Dynamic Sampling settings or their SDK's traces sampler callback for this. +You may want to opt out of extrapolation for different reasons. It is always possible to set the sample rate for specific events to 100% and therefore send all data to Sentry, implicitly opting out of extrapolation and behaving in the same way as sample mode. Depending on your configuration, you may need to change Dynamic Sampling settings or your SDK's trace sampler callback for this. ### Confidence -When users filter on data that has a very low count but also a low sample rate, yielding a highly extrapolated but low-sample dataset, developers and users should be careful with the conclusions they draw from the data. The storage platform provides confidence intervals along with the extrapolated estimates for the different aggregation types to indicate when there is elevated uncertainty in the data. These types of datasets are inherently noisy and may contain misleading information. When this is discovered, the user should either be very careful with the conclusions they draw from the aggregate data or switch to non-default mode for investigation of the individual samples. +When you filter on data that has a very low count but also a low sample rate, yielding a highly extrapolated but low-sample dataset, you should be careful with the conclusions you draw from the data. The storage platform provides confidence intervals along with the extrapolated estimates for the different aggregation types to indicate when there is lower confidence in the data. These types of datasets are inherently noisy and may contain misleading information. When this is discovered, you should either be very careful with the conclusions you draw from the aggregate data or switch to sample mode to investigate the individual samples. ## **Conclusion** - Extrapolation offers benefits in many parts of the product but brings some inherent complexity. -- Some aggregates can be extrapolated, others cannot - we may add the capability to additional aggregates in the future. -- A lot of care should be taken about how to expose extrapolation and especially switching of the modes to the user. \ No newline at end of file +- Some aggregates can be extrapolated, others cannot. +- A lot of care should be taken about how to expose extrapolation in the product, and especially in offering switching modes to a user. \ No newline at end of file diff --git a/develop-docs/application-architecture/dynamic-sampling/images/extrapolated_data_chart.png b/develop-docs/application-architecture/dynamic-sampling/images/extrapolated_data_chart.png new file mode 100644 index 00000000000000..c4315d674fc592 Binary files /dev/null and b/develop-docs/application-architecture/dynamic-sampling/images/extrapolated_data_chart.png differ