Hello,
I have a transformer model with a F-score of 0.78 that I would like to attempt to increase while using ner.teach.
I have thus been busy gathering a large amount of raw text and now that I did that I started trying to use ner.teach.
For the first 50 annotations or so it was working well, getting most things right and a few things wrong, but after the first 50 annotations, it stopped to load for a few seconds then started suggesting every punctuation and some other random words as my custom "PERIOD" entity, and nothing else. I also noticed that from this point onwards all the suggestions had a score of "1.00" whereas before they where in the 0.4-0.6 range.
This is weird, because period is an entity that has a high F-SCORE in my model but ner.teach keeps suggesting random annotations like the one on the picture. I didn't get any suggestion of any other label for over 500 annotations now.
So what is the best approach here? Should I keep declining the suggestions and then retrain the model? Or is this an "error" because I have a heavy transformer model and ner.teach is running on CPU? Or something else?
Update : the problem seems to have something to do with the loading of new data. The initial 50 suggestions always appear as expected and after the first "loading pause" of prodigy the suggestions become completely random.