Troubleshooting cloud composer

As you can see the top node is doing everything. After the fix is applied (center), it balances more evenly.

Fortunately this is something we can fix!

First connect to the kubernetes cluster.

spec:
template:
spec:
containers:
- name: airflow-worker
resources:
requests:
memory: 6Gi # half of node capacity so it never gets two of this!!!
limits:
memory: 10Gi
- name: gcs-syncd
resources:
requests:
cpu: 10m # default from composer
memory: 512Mi
limits:
cpu: 10m
memory: 512Mi
AIRFLOW_WORKER_NS=$(kubectl get namespaces | grep composer | cut -d ' ' -f1)
kubectl patch deployment airflow-worker -n ${AIRFLOW_WORKER_NS} --patch "$(cat patch.yaml)"
If you see this, your node pool doesnt have enough resources, or… all the workers are running on a single machine.. ;)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store