N.B: Merged the other thread with this one, as it should now be resolved
Okay, I think I’ve gotten to the bottom of this. There was a regression introduced in v1.5.1 that wasn’t in v1.5.0, concerning the way matches are scored from the pattern matcher. The matcher allows you to set a prior on the number of correct and incorrect matches, to initialize the scores. v1.5.1 changed the way the patterns are indexed, and this caused a problem with the default scores, making the initial scores from the matcher 0. When there are few pattern matches, and active learning is used, this results in no matches from the pattern matcher being selected.
The behaviours you’ve seen are explained as follows. When you had upper-case labels in both the patterns file and the command line (the correct setting), the 0-scores from the matcher were discarded by the active learning, so only questions from the model were shown. The model had no information to start with, so the questions were arbitrary.
When you set a lower-case label on the command line, the model’s matches were not shown, so only questions from the matcher would be presented. If the matcher is seeking lower-cased labels, you’d be asked questions from the matcher — but the model would never learn.
I believe the batch training problem comes down to the same thing. If you batch train with the label in one casing, but then try to teach with the label in a different casing, the model will not learn from the updates, and performance will steadily degrade.
There’s a simple way to mitigate the bug until we can release the next version. In the ner.recipes.teach
file, add these two lines after the PatternMatcher
is defined, in the teach
recipe:
matcher.correct_scores = defaultdict(lambda: 2.0)
matcher.incorrect_scores = defaultdict(lambda: 2.0)
Here’s how it should look in context:
matcher = PatternMatcher(model.nlp).from_disk(patterns)
matcher.correct_scores = defaultdict(lambda: 2.0)
matcher.incorrect_scores = defaultdict(lambda: 2.0)
log("RECIPE: Created PatternMatcher and loaded in patterns", patterns)
# Combine the NER model with the PatternMatcher to annotate both
# match results and predictions, and update both models.
predict, update = combine_models(model, matcher)
You should see pattern matches now coming in with a score of 0.5. As you mark instances from these patterns as correct or incorrect, the score will be updated. You can set a stronger prior if you prefer instead.