refresh_powerbi¶
refresh_powerbi(dataset_id, workspace_id, tenant_id, app_id, token_secret, custom_refresh_param)
Description¶
Refresh a Power BI dataset (semantic model)
by it's id dataset_id
.
Use case:
After model refresh, launch Power BI dataset (semantic model) refresh from BigQuery in SQL
- in a dbt post-hook
- in a dataform post_operations
- in a SQL Mesh post-statements
- in your favorite orchestration tool
Optionnal:
on premium capacity, you can pass json argument (xmla like) to launch a custom refresh (ex: Full refresh only a given table )
Docs:
- Microsoft's doc for service principal creation, token, security group affectation & Fabric API to activate
- Microsoft's doc of refresh api
Encrypt your secrets
We advise NOT TO write your token in plain text in token_secret
argument.
Otherwise, they will be stored in plain text in your BigQuery logs for months.
Instead, you can use the following snippet to generate an encrypted version of token_secret
that you can copy safely as token_secret
argument.
This public bigfunction (deployed on bigfunctions GCP project) will be able to decrypt it. But no one else can.
Usage¶
Call or Deploy refresh_powerbi
?
Call refresh_powerbi
directly
The easiest way to use bigfunctions
refresh_powerbi
function is deployed in 39 public datasets for all of the 39 BigQuery regions.- It can be called by anyone. Just copy / paste examples below in your BigQuery console. It just works!
- (You need to use the dataset in the same region as your datasets otherwise you may have a function not found error)
Public BigFunctions Datasets
Region | Dataset |
---|---|
eu |
bigfunctions.eu |
us |
bigfunctions.us |
europe-west1 |
bigfunctions.europe_west1 |
asia-east1 |
bigfunctions.asia_east1 |
... | ... |
Deploy refresh_powerbi
in your project
Why deploy?
- You may prefer to deploy
refresh_powerbi
in your own project to build and manage your own catalog of functions. - This is particularly useful if you want to create private functions (for example calling your internal APIs).
- Get started by reading the framework page
Deployment
refresh_powerbi
function can be deployed with:
pip install bigfunctions
bigfun get refresh_powerbi
bigfun deploy refresh_powerbi
Requirements
refresh_powerbi
uses the following secrets. Get them by reading the documentation link and store them in Google Secret Manager in the project where you deploy the function (and give Accessor role to the service account of the function):
name | description | documentation to get the secret |
---|---|---|
refresh_powerbi_rsa_private_key |
RSA Private Key used to decrypt secrets | doc |
Examples¶
1. Refresh of a dataset
select bigfunctions.eu.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", null)
select bigfunctions.us.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", null)
select bigfunctions.europe_west1.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", null)
+----------+
| response |
+----------+
| ok |
+----------+
2. custom refresh (xmla like) - premium capacity only
select bigfunctions.eu.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", json '{ "type": "Full", "objects": [ { "table": "table_name" } ] }')
select bigfunctions.us.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", json '{ "type": "Full", "objects": [ { "table": "table_name" } ] }')
select bigfunctions.europe_west1.refresh_powerbi("xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "xxx-xxx-xxx", "ENCRYPTED_SECRET(GvVm...)", json '{ "type": "Full", "objects": [ { "table": "table_name" } ] }')
+----------+
| response |
+----------+
| ok |
+----------+
Use cases¶
A common use case for the refresh_powerbi
function is automating the refresh of a Power BI dataset after data in its connected BigQuery tables has been updated.
Scenario: Imagine you have a BigQuery data warehouse that is used as a source for a Power BI dashboard. You have a daily ETL process that updates several tables in BigQuery. After this process completes, you want to ensure that the Power BI dataset is refreshed so that the dashboard reflects the latest data.
Implementation: You could use an orchestration tool like Airflow, Cloud Composer, or Cloud Functions to schedule the ETL process and the subsequent Power BI dataset refresh. After the ETL tasks have successfully completed, a final task would call the refresh_powerbi
function. This function would trigger the refresh of the Power BI dataset using the provided credentials and parameters.
Example (using Airflow):
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from datetime import datetime
with DAG(
dag_id="refresh_powerbi_example",
start_date=datetime(2023, 10, 26),
schedule_interval="@daily",
catchup=False,
) as dag:
# ETL tasks (e.g., loading data into BigQuery)
etl_task_1 = BigQueryInsertJobOperator(
task_id="etl_task_1",
configuration={
"query": {
"query": "your_etl_query_1",
"useLegacySql": False,
}
},
)
etl_task_2 = BigQueryInsertJobOperator(
task_id="etl_task_2",
configuration={
"query": {
"query": "your_etl_query_2",
"useLegacySql": False,
}
},
)
# Refresh Power BI dataset after ETL completes
refresh_powerbi_task = BigQueryInsertJobOperator(
task_id="refresh_powerbi",
configuration={
"query": {
"query": f"""
SELECT bigfunctions.{your_region}.refresh_powerbi(
'{your_dataset_id}',
'{your_workspace_id}',
'{your_tenant_id}',
'{your_app_id}',
'ENCRYPTED_SECRET({your_encrypted_token})',
NULL
);
""",
"useLegacySql": False,
}
},
)
[etl_task_1, etl_task_2] >> refresh_powerbi_task
Replace the placeholder values with your actual configuration. This setup ensures that the Power BI dataset is automatically refreshed after the ETL process finishes, keeping the dashboard up-to-date. This automation simplifies data management and provides users with the most current insights.
Need help or Found a bug?
Get help using refresh_powerbi
The community can help! Engage the conversation on Slack
We also provide professional suppport.
Report a bug about refresh_powerbi
If the function does not work as expected, please
- report a bug so that it can be improved.
- or open the discussion with the community on Slack.
We also provide professional suppport.