post¶
post(url, data, headers)
Description¶
POST data
to url
.
response
is a json formatted as {status_code, content}
Usage¶
Call or Deploy post
?
Call post
directly
The easiest way to use bigfunctions
post
function is deployed in 39 public datasets for all of the 39 BigQuery regions.- It can be called by anyone. Just copy / paste examples below in your BigQuery console. It just works!
- (You need to use the dataset in the same region as your datasets otherwise you may have a function not found error)
Public BigFunctions Datasets
Region | Dataset |
---|---|
eu |
bigfunctions.eu |
us |
bigfunctions.us |
europe-west1 |
bigfunctions.europe_west1 |
asia-east1 |
bigfunctions.asia_east1 |
... | ... |
Deploy post
in your project
Why deploy?
- You may prefer to deploy
post
in your own project to build and manage your own catalog of functions. - This is particularly useful if you want to create private functions (for example calling your internal APIs).
- Get started by reading the framework page
Deployment
post
function can be deployed with:
pip install bigfunctions
bigfun get post
bigfun deploy post
Keep the secrets safe!
Do NOT write secrets in plain text in your SQL queries!
Otherwise, anyone with access to your BigQuery logs can read and use them.
Instead, generate an encrypted version that you can safely share:
- Enter a secret value below along with the emails of the users who are authorized to use it (separated by commas).
- Click on
Encrypt Secret
. - The browser (no server is called) will generate an encrypted version and copy it in the clipboard
- Paste the encrypted secret into the arguments of your function exactly like if you passed the plain text version.
- The bigfunction will decrypt it and check that the calling user is authorized.
More on secret encryption
Technically, this encryption system uses the same encryption mechanism used to transfer data over the internet. It uses a pair of a public and private keys.
The public key (contained in this web page) is used to encrypt a text. The corresponding private key is the only one who is able to decrypt the text. The private key is stored in a secret manager and is only accessible to this function. Thus, this function (and this function only) can decrypt it.
Moreover, the function will check that the caller of the function belong to the list of authorized users
that you gave at encryption time.
Thanks to this:
- Nobody but this function will be able to decrypt it.
- Nobody but
authorized users
can use the encrypted version in a function. - No function but the function
post
can decrypt it.
Examples¶
Without headers
select bigfunctions.eu.post("https://httpbin.org/post", json '{"hello": "world"}', null)
select bigfunctions.us.post("https://httpbin.org/post", json '{"hello": "world"}', null)
select bigfunctions.europe_west1.post("https://httpbin.org/post", json '{"hello": "world"}', null)
+-----------------------------------------------+
| response |
+-----------------------------------------------+
| {
"content": {...},
"status_code": 200
}
|
+-----------------------------------------------+
Use cases¶
This post
BigQuery function could be used in several scenarios:
1. Sending Data to a Webhook:
Imagine you have a BigQuery table that tracks user sign-ups. You could use the post
function to send real-time notifications to a Slack channel or other messaging platform via a webhook every time a new user registers. The data
parameter would contain the user information you want to send in the notification.
SELECT bigfunctions.us.post('YOUR_WEBHOOK_URL', TO_JSON_STRING(new_users), NULL)
FROM project.dataset.new_users;
2. Interacting with an API:
You could use post
to interact with REST APIs from within BigQuery. For example, you might want to enrich your data with information from a third-party service. After performing some transformations on your data in BigQuery, you could use the post
function to send the transformed data to the API endpoint, receive the response, and then process it further within BigQuery.
SELECT bigfunctions.us.post('https://api.example.com/data', TO_JSON_STRING(t), NULL)
FROM (
SELECT user_id, SUM(order_value) as total_spent
FROM project.dataset.orders
GROUP BY user_id
) AS t;
3. Triggering Actions in External Systems:
Suppose you have a BigQuery table that monitors key performance indicators (KPIs). If a KPI falls below a certain threshold, you could use the post
function to trigger an action in an external system. This could be anything from sending an alert email to initiating a process in a workflow automation tool.
SELECT bigfunctions.us.post('https://api.example.com/alert', TO_JSON_STRING(t), NULL)
FROM (
SELECT *
FROM project.dataset.kpis
WHERE kpi_value < threshold
) AS t;
4. Sending Data to a Real-time Dashboard:
If you are using a real-time dashboarding tool, you could use the post
function to send data updates directly from BigQuery. This would allow you to keep your dashboards up-to-date with the latest information without needing to build complex data pipelines.
SELECT bigfunctions.us.post('https://api.dashboard.com/update', TO_JSON_STRING(t), NULL)
FROM (
SELECT COUNT(*) AS active_users
FROM project.dataset.users
WHERE last_seen > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
) AS t;
Key Considerations:
- Data Format: The
data
parameter must be a valid JSON string. You can use theTO_JSON_STRING
function in BigQuery to convert your data into the required format. - Headers: The
headers
parameter allows you to set custom HTTP headers for your request. This can be useful for authentication or setting content types. PassNULL
if no headers are needed. - Error Handling: You should implement proper error handling to ensure that your queries are resilient to network issues or API errors. Check the
status_code
in the response to determine if the request was successful. - Rate Limiting: Be mindful of rate limits imposed by the API you are interacting with. You might need to implement retry mechanisms or introduce delays to avoid exceeding these limits.
- Security: If you are sending sensitive data, ensure that the connection to the API is secure (HTTPS) and consider using appropriate authentication methods.
By leveraging the post
function, you can extend the functionality of BigQuery and seamlessly integrate it with other systems and services. This opens up a wide range of possibilities for automating tasks, enriching data, and building more dynamic data-driven applications.
Need help or Found a bug?
Get help using post
The community can help! Engage the conversation on Slack
We also provide professional suppport.
Report a bug about post
If the function does not work as expected, please
- report a bug so that it can be improved.
- or open the discussion with the community on Slack.
We also provide professional suppport.