mismatched structure when using tranformers model to train textcat (en_core_web_trf)

I am using Ubuntu Linux version 20.14, Python 3.9, Prodigy 1.11.6, Spacy 3.2.0. I also downloaded the latest version of en_core_web_trf.

I created my model using prodigy the following command:

prodigy train ./output -tc verbatim_claims -es .20 --base-model en_core_web_trf --label-stats --verbose --gpu-id 0

Training runs well on my Nvidia RTX-3090 and the final output of the training run is:

759   19000           0.00          5.89       99.07  4894.11    0.99
800   20000           0.00          6.24       99.07  4889.38    0.99
✔ Saved pipeline to output directory
output/model-last

=========================== Textcat F (per label) ===========================

               P       R       F
CLAIM      98.78   99.79   99.28
NO_CLAIM   99.74   98.44   99.08


======================== Textcat ROC AUC (per label) ========================

           ROC AUC
CLAIM         1.00
NO_CLAIM      1.00

I attempt to load the model using:

nlp = spacy.load(name='./output/model-best')

It throws the following exception:

Traceback (most recent call last):
  File "/snap/pycharm-professional/260/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-professional/260/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/.../ClaimsModel/main.py", line 161, in <module>
    nlp_claims = spacy.load(name="./verbatim_claims/output/model-last")
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/__init__.py", line 51, in load
    return util.load_model(
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/util.py", line 422, in load_model
    return load_model_from_path(Path(name), **kwargs)  # type: ignore[arg-type]
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/util.py", line 489, in load_model_from_path
    return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/language.py", line 2043, in from_disk
    util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/util.py", line 1300, in from_disk
    reader(path / key)
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/spacy/language.py", line 2037, in <lambda>
    deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]
  File "spacy/pipeline/transition_parser.pyx", line 595, in spacy.pipeline.transition_parser.Parser.from_disk
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/thinc/model.py", line 593, in from_bytes
    return self.from_dict(msg)
  File "/.../ClaimsModel/venv/lib/python3.9/site-packages/thinc/model.py", line 610, in from_dict
    raise ValueError("Cannot deserialize model: mismatched structure")
ValueError: Cannot deserialize model: mismatched structure

I found some reports of this same problem, but it appeared that they had been resolved from the messages.

Any guidance would be greatly appreciated.

Thanks,

Michael Wade

Hi! Sorry you've been running into issues with this.

Which version of spacy-transformers do you have installed?

Hi,

I am using spacy-transformers 1.1.2 and spacy 3.2.0. I do have pytorch installed with cu113 support as well (torch 1.10.0+cu113)

FYI, I get a good model if I use the en_core_web_lg as the base model and I can categorize from that model w/o issue.

Thanks,

Michael Wade

I was able to resolve the issue with the training by re-installing spacy-transformers directly from github as follows:

pip install git+https://github.com/explosion/spacy-transformers

I built the model directly using spacy (not prodigy) and I was not able to successfully load the model and use it.

Hi! Happy to hear the issue got resolved by reinstalling.

I built the model directly using spacy (not prodigy) and I was not able to successfully load the model and use it.

Can you elaborate a bit further? Was the model you're trying to load built with the same environment? If not, can you summarize the versions (of spacy & spacy-transformers) you used to build it with, and the version you're trying to load it with? What is the error this is being given?

Hi,@Wade, thanks for your message. I got a question that how did you add the metrics like P, R, F, ROC, and AUC in your output? Should I revise the recipe of the train? Sorry, I am new to Prodigy. Thanks!

Went on vacation after this happen and forgot to check back for messages! I was just using prodigy with the --label-stats option to do the training. I did redo all the training but switch over to using spacy train instead. That doesn't output the smae information but I believe you can use the scorer to output this kind of information if you want.

Hello! I'm experiencing the same issue of not being able to load a trf trained textcat model, but it was not resolved by reinstalling pip spacy-transformers.

I created a transformer-based textcat model using the following command:

python -m prodigy train taking_drug_trf_model --textcat-multilabel taking_drug --eval-split .2 --base-model en_core_web_trf --label-stats --verbose 

The model completed with good results but when I load the model to categorize new data I receive an error

import spacy
nlp = spacy.load(r'path\taking_drug_trf_model\model-best')

Here is the error msg:

Exception has occurred: ValueError
Cannot deserialize model: mismatched structure

I tried this code in two environments to make sure there wasn't an issue with not being up-to-date; neither worked. Here is my pip freeze

blis==0.7.9
catalogue==2.0.8
certifi==2022.9.24
charset-normalizer==2.1.1
click==8.1.3
colorama==0.4.6
confection==0.0.3
cymem==2.0.7
en-core-web-trf @ https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.4.1/en_core_web_trf-3.4.1-py3-none-any.whl
filelock==3.8.0
huggingface-hub==0.11.1
idna==3.4
Jinja2==3.1.2
langcodes==3.3.0
MarkupSafe==2.1.1
murmurhash==1.0.9
numpy==1.23.5
packaging==21.3
pandas==1.5.2
pathy==0.10.0
preshed==3.0.8
pydantic==1.10.2
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.6
PyYAML==6.0
regex==2022.10.31
requests==2.28.1
sentencepiece==0.1.97
six==1.16.0
smart-open==5.2.1
spacy==3.4.3
spacy-alignments==0.8.6
spacy-legacy==3.0.10
spacy-loggers==1.0.3
spacy-transformers==1.1.8
srsly==2.4.5
thinc==8.1.5
tokenizers==0.12.1
torch==1.13.0
tqdm==4.64.1
transformers==4.21.3
typer==0.7.0
typing-extensions==4.4.0
Unidecode==1.3.6
urllib3==1.26.13
wasabi==0.10.1

Any help is most appreciated!

hi @clark!

Thanks for your question and welcome to the Prodigy community :wave:

This issue from spaCy GitHub discussion looks very similar, as it compares spacy train with prodigy train and after reinstalling spacy-transformers. Hopefully this should give you more direction and ideas of how to debug. For more spaCy specific problems like model training/configuration, you'll likely find more help on the spaCy GitHub discussions too .

As you'll see in that issue, as you begin doing more advanced modeling like transformers, you'll likely want to move away from running prodigy train for spacy train instead and customizing your config file. prodigy train is simply a wrapper for spacy train and creates a default config file when run. In Prodigy, you can convert your data to spaCy bin files using data-to-spacy. See the docs for more details. An additional benefit of doing this is that data-to-spacy will also create separate training and dedicated hold out evaluation sets, which is better than doing --eval-split, which will create a new hold out every time.

1 Like

It's wonderful to be here! These forums are so helpful. I've always been able to find the answer here or on y'all's GitHub (but alas the transformers broke me). Thank you so much for your quick response and helpful links! I haven't read that GitHub discussion yet so I'm excited to see if it helps. Appreciate you!