This release introduces two new commands for measuring the inter-annotator agreement for document and token level annotations.
For binary, multiclass and multilabel document level annotations, we report Krippendorf’s Alpha, Gwet’s AC2 and the simple percent agreement.
For token level annotations, it's a pairwise F1 score and a confusion matrix. We've added an extensive documentation section that explains the usage and the input data format assumptions.
Since it's a brand new feature, as always, your feedback would be greatly appreciated 
With this release we also introduce Prodigy Plugins!
Prodigy Plugins are open-source add-ons that can be installed separately from the main Prodigy library. They are meant to write recipes using third party libraries, which opens up lots of possibilities to write custom recipes without worrying about Prodigy dependencies. With this release we have delivered the first 3 plugins (and there's more on the way!):
Prodigy-PDF is a recipe for annotating PDF files using Prodigy image_manual UI:
Prodigy-ANN (approximate near neighbors) and Prodigy-LUNR, leverage distributional semantics and indexing for prioritizing annotation tasks:
We invite you to learn all the details in our new Plugins section in Prodigy docs.
Finally, we'd like to offer a 20% discount on Personal licenses and extensions 
! Use LLM20 at the checkout (valid through October 
)