Hello, this shubham. i am using prodigy v1.10. I am training ner with custom label. i have some confusion related to traning ner.
I train my model in this following sequence:
first I, annotate using ner.manual with blank:en with only 30 senetence and train using en_core_web_md and save model with version1.
then I , annotate new 30 sentence with version1 model but using ner.correct and then train it save with new model version2 and continue this sequence .. and got very good result?
now, my doubt is how it work on 30 sentence only and giving gud result on the following sequence. but why it is not giving gud result when i am giving large amount of annotated sentence and train only one model?
There can be many explanations for this, but one would be: How are you evaluating your results? If you train on 30 sentences at a time and you're holding back 20% for evaluation, your training and evaluation set are very small. It's going to be almost impossible to get reliable results from it. So the accuracy you're seeing isn't very meaningful. A single prediction can result in a big difference in accuracy.
but when i use the last model on my real project apart from the dataset which i use for training, on that sentence which has never seen the model still able to give good result.. I use this approach aproximately 7-8 time on different labels and every time i get gud result. so i am confuse whether approach is a fluke or its wrk like that only.
As @ines says, I'm a bit wary of interpreting based on the small experiments. But are you trying to train a model that has the original entities from en_core_web_md and also additional entities? If so, it might make sense that the extra examples you have don't include those original entities and so you're losing that knowledge.
@honnibal sir, I think u haven't understood my doubt. I try to explain it again.
First I train blank en model on a set of 30 annotation sentences with a new label, for annotation I used prodigy ner.manual dataset1 blank: en source file1 --- label x, y, z, and soon. X, y, z, can be any entity and save this model with version 1.
After achieving the v1 model I used to train a new v2 model with another 30-40 annotated sentence which was correct using the v1 model. I used the following syntax: prodigy net.correct dataset2 v1 model source file2.
And continue this process until I not getting a good baseline. And it was seriously working on my real project.
for eval split I used 30 percent
My question is how it is learning on very few datasets, and if I give more sentences approx 400 then the result is not good?
@shubhum_jain That does sound confusing, and I'm still not 100% sure I understand the procedure. It doesn't sound like a bug in Prodigy or spaCy, but rather some counter-intuitive effect of the learning dynamics?
I'm not really sure what to suggest. Often machine learning development is an iterative process where you need to work on a specific dataset and problem to understand what's going on. It's not always possible to see a general prescription.