NER prodigy train with existing model

Hello @ines
i have a model in production with 1 million dataset. Now i need add extra 10k dataset. So used ner.correct with existing model.

Now its time to train the model with the new datsaet. I don't know how to attach this new 10k model to the existing model so I would like to use below commands to build a new model with the existing model.

prodigy train ner dataset_10k en_spacy_prod-1.0 --n-iter 10 --eval-split 0.2 --dropout 0.2

spacy train en ./ner_model_2.0 train_10k eval_10k --base-model en_spacy_prod-1.0 --pipeline ner --n-iter 30

am i on correct path in terms of train commands?. Please correct me if i am doing wrong.

I think this looks correct, yes!

Do you still have the 1 million dataset, though? Because if you do, you might as well add your extra 10k examples to it and then retrain your model from scratch using the whole corpus. This might give you more reliable results and prevents catastrophic forgetting effects etc.

Hi @ines, thanks. yes i do have 1M. i will do that as well.