Skip to content

bigfunctions > list_bigquery_resources_in_current_project

list_bigquery_resources_in_current_project

Call or Deploy list_bigquery_resources_in_current_project ?

✅ You can call this list_bigquery_resources_in_current_project bigfunction directly from your Google Cloud Project (no install required).

  • This list_bigquery_resources_in_current_project function is deployed in bigfunctions GCP project in 39 datasets for all of the 39 BigQuery regions. You need to use the dataset in the same region as your datasets (otherwise you may have a function not found error).
  • Function is public, so it can be called by anyone. Just copy / paste examples below in your BigQuery console. It just works!
  • You may prefer to deploy the BigFunction in your own project if you want to build and manage your own catalog of functions. This is particularly useful if you want to create private functions (for example calling your internal APIs). Discover the framework

Public BigFunctions Datasets:

Region Dataset
eu bigfunctions.eu
us bigfunctions.us
europe-west1 bigfunctions.europe_west1
asia-east1 bigfunctions.asia_east1
... ...

Description

Signature

list_bigquery_resources_in_current_project()

Description

List BigQuery resources in current Google Cloud project enriched with their BigQuery usage in current project.

BigQuery resources include tables, views, columns, datasets, users.

For each data asset, a popularity score is computed. For example, for a table, it is equal to the number of distinct users which read the table in the latest 30 days using execution_projects.

The result is written into bigfunction_result temporary table which has the following columns:

  • id: unique identifier of the data asset
  • type: asset type such as tables, views, datasets, users, etc.
  • name: asset name
  • description: asset description
  • popularity: popularity as float64 (the higher the more popular)
  • details: a json blob with asset additional details such as the tables of a dataset or the users of a table.

Examples

call bigfunctions.eu.list_bigquery_resources_in_current_project('you-bigquery-project');
call bigfunctions.us.list_bigquery_resources_in_current_project('you-bigquery-project');
call bigfunctions.europe_west1.list_bigquery_resources_in_current_project('you-bigquery-project');
Need help using list_bigquery_resources_in_current_project?

The community can help! Engage the conversation on Slack

For professional suppport, don't hesitate to chat with us.

Found a bug using list_bigquery_resources_in_current_project?

If the function does not work as expected, please

  • report a bug so that it can be improved.
  • or open the discussion with the community on Slack.

For professional suppport, don't hesitate to chat with us.

Use cases

A use case for the list_bigquery_resources_in_current_project function is to analyze BigQuery resource usage and identify popular or underutilized data assets within a project.

Imagine a large organization with numerous datasets, tables, and views in their BigQuery project. They want to understand:

  • Which datasets are most actively used: This can inform decisions about data retention, access control, and resource allocation. The popularity score, reflecting recent usage by distinct users, highlights heavily used datasets.
  • Identify unused or rarely used tables: These might be candidates for deletion or archiving to save storage costs and simplify data governance. Low popularity scores indicate underutilization.
  • Understand data lineage and dependencies: The details field can reveal relationships between datasets and tables, helping to visualize data flow and assess the impact of potential changes. For example, you could see which tables are referenced by a particular view.
  • Track user activity: The function can identify users interacting with different BigQuery resources, providing insights into data access patterns and potential security risks.
  • Automate data discovery and documentation: The output can be used to generate reports or dashboards summarizing key information about BigQuery resources, including descriptions and usage metrics. This assists in data discovery and documentation efforts.

Example Scenario:

A data engineering team needs to optimize their BigQuery costs. They suspect that many tables are no longer being used. By calling list_bigquery_resources_in_current_project, they can get a ranked list of tables by popularity. Tables with a popularity of zero are prime candidates for deletion. This allows them to reclaim storage space and reduce costs. Further, they can examine the details for the popular tables to ensure appropriate access controls and optimize performance for frequently accessed data.

Spread the word

BigFunctions is fully open-source. Help make it a success by spreading the word!

Share on Add a on