Hi,

Thanks for your feedback!

This response is a bit of guesswork since i havent tried airflow2 yet.

I understand you have already changed these config properties in airflow (viewable in the airflow ui):

As long as your airflow config is ok, the kubernetes cluster is your best bet for debugging.

Have you looked at the resource allocation per worker pod in your cluster, if that allows for more than 15 worker pods? or if that maxes it out? The autoscaler might have some log hints like “i wanted to spawn a task, but didnt find a node”.

The yaml config of the worker spawned should have some details on this, and there should also be something in the kubernetes UI.

Good luck!

--

--

--

Software developer and cloud enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anders Elton

Anders Elton

Software developer and cloud enthusiast

More from Medium

Blog Post 2

LYST Clone

🌞DANDELION WINE 🌞

Daily Technical Analysis BTC & ETH — 7th Mar 2022