Losing spancat labels when training after using prodigy db-merge

[ryanwesslen]

Thanks for the update.

Just to make sure, have you seen these recent posts?

Disappearing spans when using data-to-spacy

As an aside for extra context all base English models inside of spaCy use the same tokeniser under the hood. So the tokens from nlp = spacy.blank("en") should be the same as those from spacy.load("en_core_web_sm"). These tokens are all determined by the same rule based system. But yeah, it does sound like there's a mismatch. One avenue to explore is to retokenize everything by using this spaCy method which comes with a alignment_mode parameter that should allow you to wiggle around minor charac…

data-to-spacy losing annotations

Thank you! I had to modify merge function to deal with the fact that not every example has a span for a label: def merge(examples): key_values = {} for ex in examples: key = ex['_input_hash'] if key in key_values: if 'spans' in ex: if 'spans' in key_values[key]: key_values[key]['spans'].extend(ex['spans']) else: key_values[key]['spans'] = ex['spans'] else: key_values[k…

One other debugging tip - did you know you can view db-merge or data-to-spacy recipes (or any other built-in recipes)? This way you can see exactly what they're doing and debug/modify them.

Just run prodigy stats, then look in the folder with your Location:, and find the recipes folder. For db-merge, it'll be in the commands.py script and data-to-spacy is in the train.py.

Hope this helps!