what to do if train-curve shows slight decrease in last sample

[ryanwesslen]

Hi @dave-espinosa!

Excellent questions!

@dave-espinosa

What is the metric obtained in train-curve? (accuracy, precision, F1...)

This is from spaCy scorer. You can see the code/formulas from here:

Specific formula for F score, precision and recall NER

Hi! If you're training from manually created annotation, the evaluation all happens within spaCy and doesn't depend on Prodigy. spaCy uses a very standard NER evaluation. If you're working with spaCy v2.x, you can view the code here: For spaCy v3.x, it's here: If you want to do a comparative evaluation, you can also just run both your models over your evaluation data and then calculate the accuracy however you want to, and consistently for both evaluations. Some thing to keep in mind her…

FYI if you're interested, there are ways to add custom evaluation metrics (this uses textcat but should be similar for ner):

Additional metrics (recall, precision, accuracy F1) in textcat.train-curve

Hi-- I was wondering whether it is possible at all to include (and do you have plans to include) additional performance metrics in the output of textcat.train-curve. Accuracy is not always the most useful when dealing with unbalanced classes (as I am). Are additional metrics in the pipeline for textcat.train-curve, and do you suggest any workarounds in the meantime? (I guess apart from manually splitting up the data in various sizes and then running textcat.batch-train on them). Cheers!

@dave-espinosa

where are those annotation guidelines located?

Excellent question! I searched Prodigy/spaCy documentation and found there isn't documentation on creating annotation schemes. I suspect @SofieVL meant annotation schemes in general in terms of carefully defining what each entity means. This made me realize I think there's a lot of potential opportunity with guidelines to help users.

The closest documentation I know of is Matt's 2018 PyData talk. Around 8 minutes into it, he goes through the "Applied NLP Pyramid of Greatness" where he discusses the role of defining an annotation scheme. He then goes through an example of an inadequate annotation scheme for ner. Hopefully this will give you a bit of an idea on why annotation schemes can be important.

Building new NLP solutions with spaCy and Prodigy - Matthew Honnibal

Let me know if you have any thoughts or questions! We greatly appreciate your feedback/questions so please keep them coming