The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence.
As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this.
I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.
> Obviously it is also a good fit for low scale use cases.
Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.
I'm from the Cadence team.