# Problems integrating binary data into my model **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, best-practices\ **Created:** [July 28, 2021, 9:04am UTC](https://support.prodi.gy/t/problems-integrating-binary-data-into-my-model/4499 "2021-07-28T09:04:40Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![lox](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/lox/32/2281_2.png) [@lox](https://support.prodi.gy/u/lox)\ **Post date:** [July 28, 2021, 9:04am UTC](https://support.prodi.gy/t/problems-integrating-binary-data-into-my-model/4499/1 "2021-07-28T09:04:40Z")
Extreme newbie here. Working a classifier to take auction results and extract details from them, Prodigy has been working excellently. My workflow was to start out with a patterns file that I generated from a dataset I had and then some manual training with: `prodigy ner.manual whisky_ner blank:en /datasets/auctions.jsonl --patterns /datasets/patterns.jsonl --label BRAND,DISTILLERY,BOTTLER,STATED_AGE,VINTAGE,BOTTLED,CASK_NUMBER,STRENGTH,VOLUME` I hand classified about 500 items and then ran training: `prodigy train ner whisky_ner en_vectors_web_lg --output whisky_model` After that I did some further manual matching using `ner.correct`: `prodigy ner.correct whisky_ner_correct /models/whisky_model -U --exclude whisky_ner` That worked pretty well, after 500-1000 annotations I have reasonable results: ```python Label Precision Recall F-Score ----------- --------- ------ ------- STRENGTH 95.111 94.690 94.900 DISTILLERY 92.727 97.143 94.884 VOLUME 75.824 76.667 76.243 BOTTLED 96.040 94.175 95.098 VINTAGE 90.678 94.690 92.641 BOTTLER 81.250 79.592 80.412 STATED_AGE 98.726 98.726 98.726 CASK_NUMBER 90.566 87.273 88.889 BRAND 47.619 33.333 39.216 ``` It seems obvious I need to put some extra effort into refining the precision on `VOLUME` and `BRAND`. My first attempt at this was to follow approaches in several of the video tutorials, one of which suggested using `ner.teach` with a single label, which I did like this: `prodigy ner.teach whisky_ner_binary /models/whisky_model /datasets/auctions.jsonl --exclude whisky_ner --label BRAND -U ` I ran through about 1000 examples there until the % was \> 90%. After that I tried a few things, none of which resulted in a model that was better: 1. I tried applying the binary annotations directly to the model with `prodigy train ner whisky_ner_binary /models/whisky_model --output /models/whisky_model_post_binary --binary`, but the result seemed radically worse (I assume lots of annotations with only the `BRAND` label derailed earlier learnings from the model). 2. Messed around with `ner.silver-to-gold` to create a new data set, which I then applied to the model. I wasn't sure how much of the manual tagging I needed to do on that dataset, it was time consuming as there wasn't any of the other annotations to rely on. The result when trained into the model didn't seem any better. Am I going about this wrong? Any suggestions? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [July 29, 2021, 2:07am UTC](https://support.prodi.gy/t/problems-integrating-binary-data-into-my-model/4499/2 "2021-07-29T02:07:21Z")
> [@lox](#): > > but the result seemed radically worse (I assume lots of annotations with only the `BRAND` label derailed earlier learnings from the model). Hi! I think it's definitely possible that this is what's going on, and that your model ends up overfitting on the new examples. One thing you could try here is including the previously created manual annotations, to remind the model of the other labels and correct predictions. In Prodigy v1.11 (currently available as a [nightly pre-release](https://support.prodi.gy/t/prodigy-nightly-spacy-v3-support-ui-for-overlapping-spans-more/3861)), the `train` workflow supports training from both manual and binary annotations together, which can definitely help with this as well. You can then take advantage of the complete annotations and consider unannotated tokens here as "not an entity", while also including sparse binary feedback from the manual annotations. > [@lox](#): > > It seems obvious I need to put some extra effort into refining the precision on `VOLUME` and `BRAND`. It could also be worth exploring what exactly is different about those two labels. One thing to look at is obviously the frequencies – if it turns out that you only have very few examples of those entities, then that's definitely unideal, and it makes sense to add more examples that include those labels. But it might also be helpful to perform some more in-depth error analysis to find out if there's a more general problem you can pinpoint, or a common error pattern. You could even do this by writing a small Prodigy recipe that streams in your evaluation data, processes the examples with your trained model and filters for all examples where the result for `BRAND` is different from the correct annotation in the evaluation data. You could then add some options that let you annotate what the problem was, e.g. wrong label, incomplete span etc. (I'm doing something pretty similar in the second error analysis recipe in [this video tutorial](https://www.youtube.com/watch?v=zlyq9z7hdUA)). Maybe it turns out that the model mostly struggles with the distinction between `DISTILLERY`, `BOTTLER` and `BRAND`, if those end up looking fairly similar. In that case, you could experiment with an approach that combines these categories and then uses additional logic for the final distinction. Or maybe you'll find that your evaluation data isn't actually representative and ended up with a disproportionate amount of ambiguous or "weird" examples for a particular label. This can happen with a random split, especially if your dataset is fairly small. With a small evaluation set, even 5 "weird" or messy examples can easily cost you 5% in accuracy. P.S.: This is a fun use case! I was never much of a whisky drinker but I kinda got into it during the lockdowns 🥃 ---
**Author:** ![lox](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/lox/32/2281_2.png) [@lox](https://support.prodi.gy/u/lox)\ **Post date:** [July 29, 2021, 10:13am UTC](https://support.prodi.gy/t/problems-integrating-binary-data-into-my-model/4499/3 "2021-07-29T10:13:00Z")
Thanks for the detailed answer @ines! I was also thinking about going down the path of merging `DISTILLERY` , `BOTTLER` and `BRAND`. I'll watch the video you suggested, and I've also signed up for the nightly program. Cheers! 🥃 ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [July 29, 2021, 11:43pm UTC](https://support.prodi.gy/t/problems-integrating-binary-data-into-my-model/4499/4 "2021-07-29T23:43:27Z")
Cool, let us know how you go! 🥃 You could also consider adding an `EntityRuler` or `Matcher`-based component that handles the most common brands and distilleries. What you can take advantage of here is that a) popular brands are likely to also be most common in your data because they're talked about more and b) whisky is something people take quite seriously, it's regulated and probably well-catalogued. So maybe there's even an existing database you can query for this. Your model will obviously still have to deal with unknown brands, spelling variations and edge cases, but there might be a lot of low-hanging fruit that you can cover with a simple dictionary approach to boost accuracy.