Copy SQL Server data to BigQuery without CDC

  • Export table to disk
  • upload CSV file to cloud storage
  • Load to BigQuery and autodetect schema
  • It automatically generates a BigQuery schema for you, so you dont have to rely on autodetect
  • It works well with serverless where you have no disk, since it does not use local disk as temp storage.
  • It has a basic understanding of state, which is stored in GCS. Meaning it is quite fast to access compared to BQ.
  • It tries to be smart on large tables and partition them into chunks, and only reload the changed chunks from source (since this is usually the bottleneck)

Some example code

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store