Copy SQL Server data to BigQuery without CDC

  • Export table to disk
  • upload CSV file to cloud storage
  • Load to BigQuery and autodetect schema
  • It automatically generates a BigQuery schema for you, so you dont have to rely on autodetect
  • It works well with serverless where you have no disk, since it does not use local disk as temp storage.
  • It has a basic understanding of state, which is stored in GCS. Meaning it is quite fast to access compared to BQ.
  • It tries to be smart on large tables and partition them into chunks, and only reload the changed chunks from source (since this is usually the bottleneck)

Some example code

--

--

--

Software developer and cloud enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

In the mind of a Hackathon Winner

Sharing files across kubernetes clusters — the easy way.

Hosting multiple sites in Firebase

Archery Game Series: Learning Zapworks Studio

Hibernate: Don’t use show_sql to log SQL queries

Py-game module introduction

Release Notes — Kuiper 0.9.1

Step by step: .Net Core and Azure Search

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anders Elton

Anders Elton

Software developer and cloud enthusiast

More from Medium

Evaluate arithmetic expressions without values using BigQuery

Automatically Generate Type information from BigQuery schema

HOW TO LOAD DATA INTO GOOGLE BIG QUERY FROM PYTHON PANDAS

How to install Apache HBase on Ubuntu?