Copy SQL Server data to BigQuery without CDC

  • Export table to disk
  • upload CSV file to cloud storage
  • Load to BigQuery and autodetect schema
  • It automatically generates a BigQuery schema for you, so you dont have to rely on autodetect
  • It works well with serverless where you have no disk, since it does not use local disk as temp storage.
  • It has a basic understanding of state, which is stored in GCS. Meaning it is quite fast to access compared to BQ.
  • It tries to be smart on large tables and partition them into chunks, and only reload the changed chunks from source (since this is usually the bottleneck)

Some example code



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store