✨ Prodigy v1.14.3 is out! ✨

This release introduces two new commands for measuring the inter-annotator agreement for document and token level annotations.
For binary, multiclass and multilabel document level annotations, we report Krippendorf’s Alpha, Gwet’s AC2 and the simple percent agreement.
For token level annotations, it's a pairwise F1 score and a confusion matrix. We've added an extensive documentation section that explains the usage and the input data format assumptions.

Since it's a brand new feature, as always, your feedback would be greatly appreciated :pray:

With this release we also introduce Prodigy Plugins!
Prodigy Plugins are open-source add-ons that can be installed separately from the main Prodigy library. They are meant to write recipes using third party libraries, which opens up lots of possibilities to write custom recipes without worrying about Prodigy dependencies. With this release we have delivered the first 3 plugins (and there's more on the way!):

  1. :page_facing_up:Prodigy-PDF
  2. :houses: Prodigy-ANN
  3. :full_moon: Prodigy-LUNR

Prodigy-PDF is a recipe for annotating PDF files using Prodigy image_manual UI:

Prodigy-ANN (approximate near neighbors) and Prodigy-LUNR, leverage distributional semantics and indexing for prioritizing annotation tasks:

We invite you to learn all the details in our new Plugins section in Prodigy docs.

Finally, we'd like to offer a 20% discount on Personal licenses and extensions :money_with_wings:! Use LLM20 at the checkout (valid through October :slight_smile: )

1 Like

Hi, where can might I be able to find the recipe code for the new command metric.iaa.span?

Run prodigy stats and find your Location: path. Open that folder, and look for recipes/metric.py. FYI this works for any of the built-in recipes provided within Prodigy (e.g., in the recipes folder you can find other built-in recipes). Hope this helps!

1 Like

Thank you!

1 Like