questions on Multi NERs Annotation & Training at Once in a Sentence

[ryanwesslen]

hi @ruiyeNLP!

@ruiyeNLP

Hey, thanks a lot for your detailed reply, the helpful links, and for offering to coach. I would reply every night so that you could check my questions and updates on the second day. How do you like this style of coaching?

I can't guarantee any response turn-around but feel free to keep posting on this chain. We'll answer as we can.

@ruiyeNLP

How do you think of the NER model trained on 100 sentences? Do you have any suggestions for the training and for a solid project, like data argumentation, or setting a smaller batch size?

Typically, we'd advocate for more many sentences. For example, our NER workflow recommends to start to with at least 1,000 unlabeled sentences at the very beginning

While there's no set, we typically recommend models training with about 500 sentences at minimum with evaluation datasets of at least a few hundred. Therefore, if you followed that you wouldn't have enough for training or even data augmentation. Is there anyway to get more sentences?

Hopefully you can as I would be skeptical on how far you can go with only 100 sentences.

If you did want to use data augmentation, augmenty could work.

If you can overcome the few sentences, down the road you should consider running train-curve. This will give you an idea of how your accuracy changes with more annotations. Look for documentation on the recommendations.

what to do if train-curve shows slight decrease in last sample

I'm creating a new dataset, and so far I made about 500 annotations. I ran the train-curve command, and the score decreased from 0.3 to 0.29 in the last sample. What should I do at this point to make sure I don't annotate a dataset that won't work? Are there some strategies like going back to make sure that the annotation was more consistent, or troubleshoot and find a root cause if possible? Should I just keep annotating and hope that it improves? Thank you!