bigfunctions > get_latest_partition_timestamp
get_latest_partition_timestamp¶
Call or Deploy get_latest_partition_timestamp
?
✅ You can call this get_latest_partition_timestamp
bigfunction directly from your Google Cloud Project (no install required).
- This
get_latest_partition_timestamp
function is deployed inbigfunctions
GCP project in 39 datasets for all of the 39 BigQuery regions. You need to use the dataset in the same region as your datasets (otherwise you may have a function not found error). - Function is public, so it can be called by anyone. Just copy / paste examples below in your BigQuery console. It just works!
- You may prefer to deploy the BigFunction in your own project if you want to build and manage your own catalog of functions. This is particularly useful if you want to create private functions (for example calling your internal APIs). Discover the framework
Public BigFunctions Datasets:
Region | Dataset |
---|---|
eu |
bigfunctions.eu |
us |
bigfunctions.us |
europe-west1 |
bigfunctions.europe_west1 |
asia-east1 |
bigfunctions.asia_east1 |
... | ... |
Description¶
Signature
get_latest_partition_timestamp(fully_qualified_table)
Description
Return the maximum of the partition column of fully_qualified_table
Examples¶
call bigfunctions.eu.get_latest_partition_timestamp("my_project.my_dataset.my_table");
select * from bigfunction_result;
call bigfunctions.us.get_latest_partition_timestamp("my_project.my_dataset.my_table");
select * from bigfunction_result;
call bigfunctions.europe_west1.get_latest_partition_timestamp("my_project.my_dataset.my_table");
select * from bigfunction_result;
+----------------------------+
| latest_partition_timestamp |
+----------------------------+
| 2023-01-10 |
+----------------------------+
Need help using get_latest_partition_timestamp
?
The community can help! Engage the conversation on Slack
For professional suppport, don't hesitate to chat with us.
Found a bug using get_latest_partition_timestamp
?
If the function does not work as expected, please
- report a bug so that it can be improved.
- or open the discussion with the community on Slack.
For professional suppport, don't hesitate to chat with us.
Use cases¶
Let's say you have a BigQuery table called my_project.my_dataset.event_logs
that is partitioned by day on a column named event_date
. You want to find the date of the most recent events logged in that table. You can use get_latest_partition_timestamp
for this:
SELECT * FROM bigfunctions.us.get_latest_partition_timestamp('my_project.my_dataset.event_logs');
SELECT * FROM bigfunction_result; -- To see the result
This will return a table with one row and one column, latest_partition_timestamp
, containing the latest date for which a partition exists.
Other Use Cases:
- Data ingestion pipelines: Determine if new data has arrived since the last pipeline run. You can compare the latest partition timestamp with the timestamp of the last processed data.
- Report generation: Automatically filter reports to include only the most recent data. For example, a daily report could use this function to determine the date to filter on.
- Monitoring: Track the recency of data in different tables. Regularly call this function on key tables and alert if the latest partition timestamp is older than expected.
- Data quality checks: Verify that partitions are being created as expected. For instance, if you expect daily partitions, you can check if the latest partition timestamp is within the last 24 hours.
Important Considerations:
- Partitioning Column: The table must be partitioned, and the function will return the maximum value of the partitioning column.
- Data Type: The partitioning column's data type will determine the returned timestamp format. If the partitioning column is DATE, the function will return a date. If it's TIMESTAMP, it will return a timestamp.
- Empty Partitions: The function returns the latest partition regardless of whether that partition contains data. An empty partition will still be considered the latest.
- Region: Remember to use the correct BigFunctions dataset for your region (e.g.,
bigfunctions.us
for US,bigfunctions.eu
for EU). See the documentation for the full list of regions.
Spread the word¶
BigFunctions is fully open-source. Help make it a success by spreading the word!