ner.teach starts going wacky about 30 examples in

[ines (Ines Montani)]

Hi! I already replied to your other post and it sounds like one problem might be this:

@ines

@tdauria

That is, it shows me me a record with 1 of 2 entities highlighted. I reject. It then shows me the same record with the other entity highlighted. I'm also rejecting this.

Ah, that's not the correct interpretation of the interface: ner.teach will always ask you about a single entity at a time, and the goal is to give feedback on whether that particular entity is correct. So in the first case, you'd accept if the entity span is correct, and so on. If you're rejecting an entity, the feedback you give the model is "this particular entity span is incorrect, try again". That's not what you want here.

If you're rejecting suggestions that are correct, the model will be updated with the information that "this span is not an entity" and it can get confused and will try to come up with interpretations that match this new information. So it's definitely possible that after a few batches, the model ends up in a weird state where it starts to suggest you completely arbitrary spans.

Also, moving the discussion from the previous thread over:

@tdauria

Thank you, Ines! When I use my own model trained with "prodigy train", using ner.teach only shows scores of 1.00 from the very beginning.

How are you evaluating your model? Do you have a dedicated evaluation set, or are you just evaluating against a held-back percentage of the binary annotations? If you're evaluating against binary annotations and you only have a small set of annotations, you can easily end up with relatively unreliable evaluation scores: you're only evaluating against the spares binary information, so you won't know if any of the other predictions are correct or not. And if some of the evaluation examples are examples with no entities that the model gets correct very reliably, you may end up with an accuracy of 100% that's not actually very representative of the overall performance. So it's usually much better to evaluate against a stable set of gold-standard annotations.