# what to do if train-curve shows slight decrease in last sample **URL:** \ **Category:** Uncategorized\ **Tags:** usage, best-practices, training\ **Created:** [June 8, 2021, 7:28am UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306 "2021-06-08T07:28:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mumud123](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/mumud123/32/1914_2.png) [@mumud123](https://support.prodi.gy/u/mumud123)\ **Post date:** [June 8, 2021, 7:28am UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/1 "2021-06-08T07:28:46Z")
I'm creating a new dataset, and so far I made about 500 annotations. I ran the train-curve command, and the score decreased from 0.3 to 0.29 in the last sample. What should I do at this point to make sure I don't annotate a dataset that won't work? Are there some strategies like going back to make sure that the annotation was more consistent, or troubleshoot and find a root cause if possible? Should I just keep annotating and hope that it improves? Thank you! ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 9, 2021, 9:15am UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/2 "2021-06-09T09:15:36Z")
Hi! It's always difficult to provide generic advice for these kind of ML questions, as a lot depends on the data (size) and how the accuracy table actually looks like in detail. In general, a 1-percentage-point drop isn't necessarily something to worry about, but it depends on the larger trend. It might be that you've come to a sort of a "plateau", where your performance might not necessarily improve anymore if you continue annotating. If you're suspecting some kind of inconsistency in your later annotations, you might also consider using the [`review`](https://prodi.gy/docs/recipes#review) recipe to double-check that and enforce consistency. But you could also just annotate another 50/100 and see what happens first, to determine whether the trend continues, or whether the 1pp drop was just a "fluke"? ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 9, 2021, 9:18am UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/3 "2021-06-09T09:18:39Z")
If you do notice that you're stuck around 30% with your challenge, perhaps it's worth reconsidering the annotation guidelines for your approach? I don't know any details, so it's difficult to give any generic advice though. ---
**Author:** ![mumud123](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/mumud123/32/1914_2.png) [@mumud123](https://support.prodi.gy/u/mumud123)\ **Post date:** [June 11, 2021, 8:07pm UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/4 "2021-06-11T20:07:05Z")
thank you. I did a bit more annotations as you suggested and it is working better now! ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 12, 2021, 8:22am UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/5 "2021-06-12T08:22:44Z")
Happy to hear it! ---
**Author:** ![dave-espinosa](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/dave-espinosa/32/2628_2.png) [@dave-espinosa](https://support.prodi.gy/u/dave-espinosa)\ **Post date:** [June 7, 2022, 5:56pm UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/6 "2022-06-07T17:56:58Z")
Hello @SofieVL I was checking this thread, because it's exactly my own case at the moment. What I am doing is training a **custom NER model, which recognizes 4 different labels** (there are a bit more details [here](https://support.prodi.gy/t/recommended-text-length-for-training-ner-models-in-spacy/5615)). Following [Ines Montani introductory video](https://youtu.be/59BKHO_xBPA), I started off with 500 texts (80-20 split), just to see how things are going, and I got [this result](https://support.prodi.gy/t/train-curve-ends-in-error-module-plotext-has-no-attribute-colorless/5611/3). As you can see, [train-curve](https://prodi.gy/docs/recipes#train-curve) suggested to "add more labeled samples to training set", which I did: by adding 500 samples more (summing up a total of 1000 samples; 80-20 split), I got the following plot: ![1500_samples](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/5/525579aa95fd4eb579a22ad3289b3149664b6af1.png) Comparing those results with the first test, 2 quick observations can be obtained: - The metric increase was not much significant. - Again, adding more samples could improve the model's perfomance. This time, I decided to increase the amount of added samples a bit more: 4000 labeled texts were used this time (80-20 split), to obtain the following results: ![4000_samples](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/a/a6427c17f18c1044d1cf202f2ef2005c18c12e99.png) Sadly this time, the performance worsened. With these results, I have the following questions: 1. **What is the metric obtained in** `train-curve`? **(accuracy, precision, F1...)** 2. Intuitively, it seems that this model is _overfitting_ ; however if less samples are used, I see that the "metric" never surpasses "0.6" (which it is a bit worrying, as Ines in the video mentioned before, obtained a metric of \> "0.7" in her first try). [Here](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/3) you mentioned "_it's worth reconsidering the annotation guidelines for your approach_"; **where are those annotation guidelines located**? Sorry if they are too many questions, please let me know If I should open different cases for each. Thank you! ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [June 8, 2022, 4:57pm UTC](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306/7 "2022-06-08T16:57:56Z")
Hi @dave-espinosa! Excellent questions! > [@dave-espinosa](#): > > **What is the metric obtained in** `train-curve`? **(accuracy, precision, F1...)** This is from [spaCy scorer](https://spacy.io/api/scorer). You can see the code/formulas from here: > [@Specific formula for F score, precision and recall NER](https://support.prodi.gy/t/specific-formula-for-f-score-precision-and-recall-ner/4422/2): > > Hi! If you're training from manually created annotation, the evaluation all happens within spaCy and doesn't depend on Prodigy. spaCy uses a very standard NER evaluation. If you're working with spaCy v2.x, you can view the code here: For spaCy v3.x, it's here: If you want to do a comparative evaluation, you can also just run both your models over your evaluation data and then calculate the accuracy however you want to, and consistently for both evaluations. Some thing to keep in mind her… FYI if you're interested, there are ways to add custom evaluation metrics (this uses `textcat` but should be similar for `ner`): > [@Additional metrics (recall, precision, accuracy F1) in textcat.train-curve](https://support.prodi.gy/t/additional-metrics-recall-precision-accuracy-f1-in-textcat-train-curve/2046): > > Hi-- I was wondering whether it is possible at all to include (and do you have plans to include) additional performance metrics in the output of textcat.train-curve. Accuracy is not always the most useful when dealing with unbalanced classes (as I am). Are additional metrics in the pipeline for textcat.train-curve, and do you suggest any workarounds in the meantime? (I guess apart from manually splitting up the data in various sizes and then running textcat.batch-train on them). Cheers! > [@dave-espinosa](#): > > **where are those annotation guidelines located**? Excellent question! I searched Prodigy/spaCy documentation and found there isn't documentation on creating annotation schemes. I suspect @SofieVL meant annotation schemes in general in terms of carefully defining what each entity means. This made me realize I think there's a lot of potential opportunity with guidelines to help users. The closest documentation I know of is Matt's 2018 PyData talk. Around 8 minutes into it, he goes through the "[Applied NLP Pyramid of Greatness](https://twitter.com/pmbaumgartner/status/1530160343292465152?s=20&t=O97UZNXMD0566vTl8Tz1Lg)" where he discusses the role of defining an annotation scheme. He then goes through an example of an inadequate annotation scheme for `ner`. Hopefully this will give you a bit of an idea on why annotation schemes can be important. [![](https://img.youtube.com/vi/jpWqz85F_4Y/maxresdefault.jpg "Building new NLP solutions with spaCy and Prodigy - Matthew Honnibal") ](https://www.youtube.com/watch?v=jpWqz85F_4Y&t=375) Let me know if you have any thoughts or questions! We greatly appreciate your feedback/questions so please keep them coming 🙂