KeyError: 'U-STRIKE' when training on a new entitity

[Stephan]

Hello Explosion,

first I want to thank you for the fantastic work. I just started working with spacy and prodigy and I love it so far. Especially the documentation of spacy is outstanding and a pleasure to read.

I currently want to train a new entity and followed the video [1] to do so. It all works fine to the last step. But now I noticed that my new model only knows this new entity and would not detect any of the other entities anymore, which is probably due to the --label STRIKE in prodigy ner.batch-train strikes_ner en_core_web_lg --output strikes-model --label STRIKE --eval-split 0.2 --n-iter 6 --batch-size 8. So I left this parameter out in order to train the model on all entities, as the documentation says:

If not set, all available labels will be used.

But this leads to the above mentioned KeyError: 'U-STRIKE' error. Can you please help me with that?

Best, Stephan

1: Named Entity Recognition · Prodigy · An annotation tool for AI, Machine Learning & NLP

[honnibal (Matthew Honnibal)]

Thanks, glad you like it!

That error is definitely strange. Just to rule out one problem: you’re on Prodigy 1.2 and spaCy 2.0.5, right?

If you add the label manually in the batch_train recipe using nlp.entity.add_label('STRIKE'), does the same problem occur?

[ines (Ines Montani)]

To add to Matt's comment above:

@Stephan

But now I noticed that my new model only knows this new entity and would not detect any of the other entities anymore

What you're seeing here might also be due to the "catastrophic forgetting" problem: as you're teaching the model about your new entity, it's overfitting on the data and "forgetting" what it had previously learned. So if you care about the other entity types, a simple solution is to mix in entities that the model previously recognised correctly. This is pretty easy to do with both Prodigy and spaCy. You can find more details and strategies in this thread:

New entity model ruins other entities

I’m trying to train a new entity of type “Technology”. I started with some seed terms, created patterns and annotated ~300 examples. Then I batch-trained them with en_core_web_lg as a baseline model. Although I don’t have a lot of annotations yet, I wanted to see how the model is doing so far. It seems that the model mislabels a lot of entities and tends to label non-entities as WORK_OF_ART: import spacy nlp = spacy.load('tech-model') doc = nlp('Blockchain is a kind of technology') [(ent.tex…

[Stephan]

Hello you two,

thanks a lot for the quick answers.

@honnibal Yes, I am on the latest versions, I had just installed it yesterday.

(py3) ~: pip freeze | grep 'spacy\|prodigy'
prodigy==1.2.0
spacy==2.0.5

As you suggested I added the line nlp.entity.add_label('STRIKE') to ner.py:327 and the problem still persists.

UPDATE: I added nlp.entity.add_label('STRIKE') to ner.py:333 (as I’m of course not creating a new pipe) and now it works. Thanks a lot.

@ines Thanks a lot for your input too. I’m going to read this article now. In any case I’m a bit confused by the documentation now. It made sense to me that my model only new about one label if I use the --label parameter to only teach it about that one. And that it would not overfit on that one in case I leave the label empty so that it trains on the new one, while not forgetting the old. Or is that a mistake in my thinking and even with leaving out --label it would forget about the old one if I don’t explicitly give it examples of the previously learned entities?