Solved: Replicate data from MySQL to BigQuery

Nikita_G · 03-22-2024 12:03 AM

I want to replicate data from MySQL to BigQuery by utilizing DataStream and Pub/Sub in the data transfer process. I aim to perform stream processing of the data from DataStream using Cloud Function before replicate it into BigQuery. anyone suggest if this is possible to achieve without using GCS and Dataflow? I don't find any implementation article regarding this.

MySQL ----> DataStream ----> Pub/Sub ----> Cloud Function ----> BigQuery

nceniza

Is your MySQL is residing in Cloud SQL in GCP ? Also at the moment DataStream to Pub/Sub is not supported, You can track the progress in this issue tracker here: https://issuetracker.google.com/278842503 and a Google community post here: https://www.googlecloudcommunity.com/gc/Serverless/Use-Pub-Sub-as-a-Datastream-destination/m-p/45423...

If you still need advice for orginizing a process using Google Cloud, I would suggest to contact TAM(Technical Account Manager) for your use case.

View solution in original post

nceniza

Is your MySQL is residing in Cloud SQL in GCP ? Also at the moment DataStream to Pub/Sub is not supported, You can track the progress in this issue tracker here: https://issuetracker.google.com/278842503 and a Google community post here: https://www.googlecloudcommunity.com/gc/Serverless/Use-Pub-Sub-as-a-Datastream-destination/m-p/45423...

If you still need advice for orginizing a process using Google Cloud, I would suggest to contact TAM(Technical Account Manager) for your use case.

Nikita_G

Great! Thank you for the solution. Can you also suggest if we can migrate specific rows from Cloud SQL to a BigQuery table using DataStream? Does DataStream provide functionality to filter out specific rows during migration?