Automation is combination of both, accuracy and accuracy of confidence scores.
Good way to think about automation is recall at high precision which is what you need for true automation where you don't worry about documents that are very likely to have correct results and focus on manually correcting documents likely to have errors.
The reason accuracies are tighly grouped but not the automation is because these models are trained to be accurate but not necessarily predictable, where there is no real way to get confidence score calibrated.
Good way to think about automation is recall at high precision which is what you need for true automation where you don't worry about documents that are very likely to have correct results and focus on manually correcting documents likely to have errors.
The reason accuracies are tighly grouped but not the automation is because these models are trained to be accurate but not necessarily predictable, where there is no real way to get confidence score calibrated.
Couldn't find the benchmark mistral used as well