Pipeline Checks - Freshness rule

The "Freshness" rule is a type of Pipeline Check that ensures the timeliness and reliability of data within data processing workflows. It verifies data quantities, assesses data quality and performance, and monitors if data is present or updated within a specified time frame.

In simple terms, the Freshness Check monitors whether data is available or updated within a certain timeframe. For instance, in a weather forecasting app, if data was initially fetched at "2023-09-30 08:00:00" and the next execution timestamp is "2023-09-30 14:30:00," resulting in a time difference of 6 hours, but the defined threshold is 3 hours, then the Freshness Check would alert because the difference exceeds the defined threshold.

This rule enhances the flow, quality, and efficiency of data processing by ensuring that data remains current and relevant, thus improving the reliability of the overall system.

For further information about the rule detail page including scope, threshold, notifications, etc., please see the detailed article Rule Detail page.

Detailed Steps:

Step 1: Log in to DvSum, proceed to the Data Dictionary tab, and select the relevant Data source and Table Name.

 

Step 2: Select the table name then select the Data Quality tab and choose Available Rules.

Step 3: Open the Settings Reference page and enter the Load Profile & Metric Time 

                                      Note: The Load Profile should be set to Incremental Data

Step 4: Select the "⊕ Add Rule" button, then choose the "Pipeline Checks" category. From the list of options, click on "Freshness"

Step 5: Basic Input

In the Rule Wizard's Basic Input section, you need to fill in the Rule Description and Threshold Hours corresponding to the table in which you want to search that data is not present or updated by a certain time.

Step 6: Validate

After saving the rule, you'll see its definition. Click "Run" to execute and test the rule.

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk