Skip to content

export_to_datastore

export_to_datastore(datastore_path, key, data)

Description

Exports data to Datastore (Firestore in Datastore mode) to key at datastore_path (like project/database/namespace/kind)

💡 For this to work, bigfunction@bigfunctions.iam.gserviceaccount.com must have datastore user role in your project.

Usage

Call or Deploy export_to_datastore ?
Call export_to_datastore directly

The easiest way to use bigfunctions

  • export_to_datastore function is deployed in 39 public datasets for all of the 39 BigQuery regions.
  • It can be called by anyone. Just copy / paste examples below in your BigQuery console. It just works!
  • (You need to use the dataset in the same region as your datasets otherwise you may have a function not found error)

Public BigFunctions Datasets

Region Dataset
eu bigfunctions.eu
us bigfunctions.us
europe-west1 bigfunctions.europe_west1
asia-east1 bigfunctions.asia_east1
... ...
Deploy export_to_datastore in your project

Why deploy?

  • You may prefer to deploy export_to_datastore in your own project to build and manage your own catalog of functions.
  • This is particularly useful if you want to create private functions (for example calling your internal APIs).
  • Get started by reading the framework page

Deployment

export_to_datastore function can be deployed with:

pip install bigfunctions
bigfun get export_to_datastore
bigfun deploy export_to_datastore

Examples

1. Export data to default database, to default namespace with auto-generated key.

select bigfunctions.eu.export_to_datastore("your-project/default/default/users", null, json '{"name": "Marc Harris", "email": "marc@harris.com"}')

select bigfunctions.us.export_to_datastore("your-project/default/default/users", null, json '{"name": "Marc Harris", "email": "marc@harris.com"}')

select bigfunctions.europe_west1.export_to_datastore("your-project/default/default/users", null, json '{"name": "Marc Harris", "email": "marc@harris.com"}')

+------------------+
| key              |
+------------------+
| 4503604769587200 |
+------------------+

2. Export data with email as key.

select bigfunctions.eu.export_to_datastore("your-project/default/default/users", "marc@harris.com", json '{"name": "Marc Harris"}')

select bigfunctions.us.export_to_datastore("your-project/default/default/users", "marc@harris.com", json '{"name": "Marc Harris"}')

select bigfunctions.europe_west1.export_to_datastore("your-project/default/default/users", "marc@harris.com", json '{"name": "Marc Harris"}')

+-----------------+
| key             |
+-----------------+
| marc@harris.com |
+-----------------+

Use cases

This export_to_datastore function is useful for scenarios where you need to move or synchronize data from BigQuery to Datastore (Firestore in Datastore mode). Here are a few use cases:

  • Materialized Views in Datastore: You might have complex analytical queries in BigQuery that produce aggregated data. Instead of recomputing these queries every time you need the results, you could use export_to_datastore to periodically store the aggregated data in Datastore. This makes accessing these aggregations much faster for applications that don't need the full power of BigQuery.

  • Data Synchronization for Microservices: Imagine a microservice architecture where one service uses BigQuery for analytics and another service relies on Datastore for operational data. You can use this function to keep the relevant data synchronized between the two data stores. For example, BigQuery might store a user's purchase history, while Datastore stores their profile information. You can use export_to_datastore to update the Datastore profile with aggregated purchase statistics calculated in BigQuery.

  • Creating Lookups for Real-time Applications: BigQuery is great for analytical workloads, but not ideal for low-latency lookups. If your application needs to quickly retrieve data based on a key, you can use export_to_datastore to create a lookup table in Datastore. For instance, you might have product information stored in BigQuery, and you want to quickly retrieve product details by their SKU. You could export the SKU and relevant product details to Datastore for faster retrieval by your application.

  • Simplifying Data Access for Non-technical Users: Datastore often provides a simpler interface for accessing data compared to BigQuery, especially for users who are not familiar with SQL. You can use export_to_datastore to make specific datasets available in Datastore, allowing non-technical users to access and manipulate data more easily.

  • Backup and Restore: While not a primary backup solution, export_to_datastore could be used in conjunction with other backup methods to create a copy of specific BigQuery data in Datastore, particularly for smaller, critical datasets.

Example: Building a Real-time Product Catalog

Let's say you have product information stored in a BigQuery table called products. You want to display this information on your website, which requires low-latency data access. Here's how you could use export_to_datastore:

  1. BigQuery Query: Write a query that selects the relevant product information (e.g., product_id, name, price, description) from the products table.
  2. export_to_datastore Function: Use the function within your BigQuery query to export the results to Datastore. You would use the product_id as the key and the remaining product information as the data.
SELECT bigfunctions.us.export_to_datastore(
    'your-project',
    null,  -- Use default namespace
    'product_catalog', -- Kind for product data
    CAST(product_id as STRING),  -- Product ID as key
    TO_JSON_STRING(STRUCT(name, price, description))  -- Product details as JSON
)
FROM `your_project.your_dataset.products`;
  1. Website Integration: Your website's backend can now efficiently retrieve product information from Datastore using the product_id as the key, providing a fast and responsive user experience.

This is just one example. The versatility of the export_to_datastore function allows you to bridge the gap between BigQuery's analytical capabilities and Datastore's operational strengths in a variety of situations.

Need help or Found a bug?
Get help using export_to_datastore

The community can help! Engage the conversation on Slack

We also provide professional suppport.

Report a bug about export_to_datastore

If the function does not work as expected, please

  • report a bug so that it can be improved.
  • or open the discussion with the community on Slack.

We also provide professional suppport.

Show your ❤ by adding a ⭐ on