Troubleshooting cloud composer

As you can see the top node is doing everything. After the fix is applied (center), it balances more evenly.

Fortunately this is something we can fix!

spec:
template:
spec:
containers:
- name: airflow-worker
resources:
requests:
memory: 6Gi # half of node capacity so it never gets two of this!!!
limits:
memory: 10Gi
- name: gcs-syncd
resources:
requests:
cpu: 10m # default from composer
memory: 512Mi
limits:
cpu: 10m
memory: 512Mi
AIRFLOW_WORKER_NS=$(kubectl get namespaces | grep composer | cut -d ' ' -f1)
kubectl patch deployment airflow-worker -n ${AIRFLOW_WORKER_NS} --patch "$(cat patch.yaml)"
If you see this, your node pool doesnt have enough resources, or… all the workers are running on a single machine.. ;)

--

--

--

Software developer and cloud enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Sphero: A toy for motivation

Circuit breakers: don't let your dependencies bring you down

Static Library in C

Azure Functions — Validate Azure active directory tokens using your own custom binding

Bitlab — HackTheBox

Writeup for Rivest Shamir Adleman (Crypto)

Kick-start of my LFX Mentorship with Litmus Chaos

Chaos engineering steps

How to find out if your sprint was a success

How to find out if your sprint was a success

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anders Elton

Anders Elton

Software developer and cloud enthusiast

More from Medium

Docker with Terraform on Clouds GCP and AWS

Useful gcloud cli commands wrapped in functions

Transferring files directly to an EC2 instance via WinSCP from our local machine

Terraform with IBM Cloud Event Notifications