Hi,

Thanks for your feedback!

This response is a bit of guesswork since i havent tried airflow2 yet.

I understand you have already changed these config properties in airflow (viewable in the airflow ui):

parallelism = xx
dag_concurrency = xx
max_active_runs_per_dag = xx

As long as your airflow config is ok, the kubernetes cluster is your best bet for debugging.

Have you looked at the resource allocation per worker pod in your cluster, if that allows for more than 15 worker pods? or if that maxes it out? The autoscaler might have some log hints like “i wanted to spawn a task, but didnt find a node”.

The yaml config of the worker spawned should have some details on this, and there should also be something in the kubernetes UI.

Good luck!