@idealley Thanks for updating, and sorry I missed this thread! I actually suspect you might have encountered a bug in a previous version. Are you currently using v1.5.1?
That sounds like a good workflow. One difficult question is always whether to recommend training on top of an existing NER model (such as en_core_web_lg), or whether to recommend starting from a blank one. The existing model might know useful things, but on the other hand it can also be stubborn about the existing entity definitions, and the training data might not correct them. For instance, I think this is why you had that problem with a rare category like WORK_OF_ART. If you never label examples with that label, the model never sees any negative examples of it, so it’s hard for it to learn not to predict it.