Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 50 additions & 1 deletion docs/checks/freshness-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,56 @@ A **Freshness Check** monitors when a dataset (table, file, or view) was last up
* **Pass** → Data is updated within the allowed time.

* **Fail** → Data is older than the limit, triggering an **alert**.


## How Freshness is Measured Across Platforms

Qualytics performs volumetric and freshness measurements differently depending on the data platform and container type (table, view, or computed).

### Snowflake

#### Tables

Freshness and volumetrics are retrieved directly from the `INFORMATION_SCHEMA`. (System-level metadata is used — not the Incremental Field.)

#### Views and Computed Objects

- **Volumetrics:** `SELECT COUNT(*) FROM <object>`

- **Freshness:** `SELECT MAX(<incremental_identifier>) FROM <object>` (Here, the Incremental Field defined in Table Settings determines freshness.)

### Oracle

#### Tables

- **Volumetrics:** Retrieved from the `information_schema`

- **Freshness:** Determined using `SELECT MAX(<incremental_identifier>)`

#### Views and Computed Objects

- **Volumetrics:** `SELECT COUNT(*) FROM <object>`

- **Freshness:** `SELECT MAX(<incremental_identifier>) FROM <object>`

### S3

- **Freshness:** Based on the file modification time.

- **Volumetrics:** Depends on file format.
- CSV, Excel → file load + row count.
- Delta or structured formats → row counts extracted from metadata.

### BigQuery

Same behavior as Snowflake:

- **Tables:** Retrieved from `INFORMATION_SCHEMA`.
- **Views/Computed:** `SELECT COUNT(*)` and `SELECT MAX(<incremental_identifier>)`.

!!! note
- For regular tables, freshness is evaluated from `INFORMATION_SCHEMA` — the configured Incremental Field is not used.
- For views or computed tables, freshness is derived from the Incremental Field defined in Table Settings.

## Configuring Freshness Check

**Step 1:** Log in into your Qualytics account and select the **datastore** from the left menu on which you want to add a volumetric check.
Expand Down
5 changes: 4 additions & 1 deletion docs/observability/freshness.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,7 @@ This measures the timeliness of data by monitoring when new data was last added
| 14. | **Maximum Age** | Displays the maximum recorded age of data in milliseconds. |
| 15. | **Last Asserted** | Shows the latest date when the data was validated or checked. |
| 16. | **Edit Maximum Age** | Edit **Maximum Age** lets users set custom limits for data freshness, allowing control over when alerts are triggered based on the age of the data. |
| 17. | **Graph Visualization** | Graph illustrates consistent data patterns over time, with sudden anomalies marked by spikes in red. It reflects changes in data freshness and highlights when the data was last updated. |
| 17. | **Graph Visualization** | Graph illustrates consistent data patterns over time, with sudden anomalies marked by spikes in red. It reflects changes in data freshness and highlights when the data was last updated. |

!!! note
For more information please refer to the [**Freshness Check**](../checks/freshness-check.md) documentation..
4 changes: 2 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,9 +177,9 @@ nav:
- Overview: observability/observability.md
- Observability Categories:
- Measures: observability/measures.md
- Metric: observability/metric-check.md
- Volumetric: observability/volumetric.md
- Freshness: observability/freshness.md
- Metric: observability/metric-check.md
- Freshness: observability/freshness.md
- Manage Observability:
- Overview: observability/manage-observability.md
- Edit Check: observability/edit-check.md
Expand Down