✨ Prodigy nightly: spaCy v3 support, UI for overlapping spans & more

Hello, I am trying to train a NER and I get the following error:

python -m prodigy train --ner ner_dataset --base-model en_core_web_trf model_output_trf

 ⚠ Aborting and saving the final best model. Encountered exception:
TypeError("'FullTransformerBatch' object is not iterable")
Traceback (most recent call last):
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/prodigy/__main__.py", line 54, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 329, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/plac_core.py", line 367, in call
    cmd, result = parser.consume(arglist)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/plac_core.py", line 232, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/prodigy/recipes/train.py", line 244, in train
    return _train(
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/prodigy/recipes/train.py", line 172, in _train
    spacy_train(nlp, output_path, use_gpu=gpu_id, stdout=stdout)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/training/loop.py", line 224, in train_while_improving
    score, other_scores = evaluate()
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/training/loop.py", line 281, in evaluate
    scores = nlp.evaluate(dev_corpus(nlp))
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/language.py", line 1377, in evaluate
    for doc, eg in zip(
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/util.py", line 1488, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "spacy/pipeline/trainable_pipe.pyx", line 79, in pipe
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy/util.py", line 1507, in raise_error
    raise e
  File "spacy/pipeline/trainable_pipe.pyx", line 75, in spacy.pipeline.trainable_pipe.TrainablePipe.pipe
  File "spacy/pipeline/tagger.pyx", line 111, in spacy.pipeline.tagger.Tagger.predict
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/borlogh/anaconda3/envs/fb_spacy31/lib/python3.8/site-packages/spacy_transformers/layers/trfs2arrays.py", line 23, in forward
    for trf_data in trf_datas:
TypeError: 'FullTransformerBatch' object is not iterable

I was able to fix the problem modifing the file spacy_transformers/layers/trfs2arrays.py adding the following code :

def forward(model: Model, trf_datas: List[TransformerData], is_train: bool):
    pooling: Model[Ragged, Floats2d] = model.layers[0]
    grad_factor = model.attrs["grad_factor"]
    outputs = []
    backprops = []

    # NEW CODE - BEGIN
    if not isinstance(trf_datas, list):
        trf_datas = trf_datas.doc_data # FullTransformerBatch -> List[TransformerData]
    # NEW CODE - END

    for trf_data in trf_datas:
        if len(trf_data.tensors) > 0:
            t_i = find_last_hidden(trf_data.tensors)
            tensor_t_i = trf_data.tensors[t_i]
            if tensor_t_i.size == 0:

what do you think about what the real problem might be?

I am using prodigy 1.11.0a8 and python 3.8.10

Hi!

Which spacy version do you have and could you try upgrading? This error msg reminds me of a bug that was fixed earlier. (hopefully)

I tried it with 3.0.6 and 3.1, and I get the same error

I installed the latest nighty for Linux (prodigy-1.11.0a8-cp36.cp37.cp38.cp39-cp36m.cp37m.cp38.cp39-linux_x86_64.whl) in a clean Python 3.8.6 virtualenv.

First the installer reported the following version conflicts

fastapi 0.66.0 requires starlette==0.14.2, but you'll have starlette 0.13.8 which is incompatible.
typer 0.3.2 requires click<7.2.0,>=7.1.1, but you'll have click 8.0.1 which is incompatible.
spacy 3.0.6 requires pydantic<1.8.0,>=1.7.1, but you'll have pydantic 1.8.2 which is incompatible.

In particular click==8.0.1 results in an exception

ModuleNotFoundError: No module named 'click._bashcomplete'

when running the prodigy command.

I resolved the issue by manually installing

pip install click==7.1.*

also for completeness

pip install pydantic==1.7.*

Finally fastapi 0.65.0 is the version which introduced the requirement for starlett==0.14.2 (Release Notes - FastAPI).

prodigy 1.11.0a8 requires starlette<0.14.0,>=0.12.9, but you'll have starlette 0.14.2 which is incompatible.
fastapi 0.66.0 requires starlette==0.14.2, but you'll have starlette 0.13.8 which is incompatible.

I backed this out to

pip install fastapi==0.64.*

It's looks really good! Other than click I'm not sure I needed to downgrade the other packages.

Did you have anything installed in your environment previously? I just updated the pydantic pin of Prodigy to match spaCy's, but aside from that, all dependencies install and resolve fine in isolation in our CI builds. (But with the new pip resolver, it's definitely possible to end up with conflicts if there's something else installed in the environment that depends on other versions of those packages.)

Just released a new nightly v1.11.0a10 that includes the following updates:

  • improved support for updating from binary annotations, especially those created with ner.teach
  • ner.teach will now also ask about texts with no entities – so if a suggestion doesn't include any suggestions, you can accept it if it has no entities and reject it if it does contain entities of the given label(s)
  • support for providing --spancat datasets for training spaCy v3.1's new SpanCategorizer in spacy train (with auto-generated suggester function)
  • support for validating created spans in spans.manual against suggester function
  • support for custom config or base model in prodigy train and data-to-spacy
  • support for providing --textcat and --textcat-multilabel (non-exclusive categories, including binary annotations) separately to prodigy train and data-to-spacy
  • sent.teach and sent.correct recipes for improving a sentence recognizer and support for --senter annotations in prodigy train an data-to-spacy
  • textcat.correct for correcting an existing text classifier
  • "_timestamp" property added to all created annotations reflecting the time the annotation was submitted in the UI
  • progress command for viewing annotation progress over time
  • ARM wheels
  • use the -F flag to pass in one or more comma-separated Python files to import from across all recipes to provide the recipe function, but also custom registered functions for spaC configs (e.g. in prodigy train)
  • fixes for various bugs introduced in the previous nightlies

Btw, after downloading an extracting the zip containing the wheel files, you can also run the following to automatically select the best-matching wheel for your platform:

pip install prodigy -f /path/to/wheels
3 Likes

Did you have anything installed in your environment previously? I just updated the pydantic pin of Prodigy to match spaCy's, but aside from that, all dependencies install and resolve fine in isolation in our CI builds. (But with the new pip resolver, it's definitely possible to end up with conflicts if there's something else installed in the environment that depends on other versions of those packages.)

I'm using a clean environment of Python 3.8.6, but that comes with pip 20.2.1 and the new resolver didn't come in until pip 20.3 as far as I'm aware. So I guess it works - since there is a set of dependencies that satisfies the constraints and my understanding is the old pip isn't overly smart in how it determines what to install.

I'll update pip before installing the latest nightly and see if that does a clean install.

Thanks a lot for the update! I will try ner.teach as soon as I can.
I am a little confused with all the download files at the moment. What is the difference between cp36m-cp37m-cp38-cp39? Does it matter which one I use for installation?

I just found out that it refers to the python version that you are using. I leave it here in case anyone else is unfamiliar with this convention

2 Likes

python versions

I tried the new version of prodigy, and this error keeps happening:

python 3.8.10
spacy 3.1
prodigy 1.11.0a10

  File "/home/borlogh/.local/share/virtualenvs/fb_sio-Kywf3_9j/lib/python3.8/site-packages/spacy_transformers/layers/trfs2arrays.py", line 23, in forward
    for trf_data in trf_datas:
TypeError: 'FullTransformerBatch' object is not iterable

Thanks for trying to update. I've just tried to replicate this and couldn't immediately, but I'm on the latest version (master branch) of spacy-transformers. Do you have 1.0.2 installed? If so, could you try building the repo from source? There might have been a bugfix that hasn't made it to the latest release yet. Sorry for being annoying, but I still think this bug was fixed, so I'm hoping we can find you a setup that works!

I am using spacy-transformers 1.0.3, I tried to downgrade to 1.0.2 but it isn't compatible with spacy 3.1

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. 
This behaviour is the source of the following dependency conflicts.
prodigy 1.11.0a10 requires spacy<3.2.0,>=3.1.0, but you have spacy 3.0.6 which is incompatible.
en-core-web-trf 3.1.0 requires spacy<3.2.0,>=3.1.0, but you have spacy 3.0.6 which is incompatible.
en-core-web-trf 3.1.0 requires spacy-transformers<1.1.0,>=1.0.3, 
but you have spacy-transformers 1.0.2 which is incompatible.

I made it work with the library's modification as before, so it isn't a locking problem for me right now.

Sorry I should have been more clear - 1.0.3 is the correct version of spacy-transformers (downgrading won't help), but there's a slightly newer version available only when you build it from source.

I am not sure what I am doing wrong, but it appears that when I try to fine-tune a custom model with my binary ner.teach annotation, it doesn't seem to work at all. This is what happens when I try to fine-tune a spacy 3.1 model. When using a spacy 3.0.6 model (the same training process, just using previous spacy version), it appears to have a start value of 0.36, and then decreases to 0 as well.

command used : prodigy train -n ner_teach_july -m /temp_data/transformers/model-best -l xx -g 0 /temp_data/transformers/new

It worked! thanks

Happy to hear it, thanks for reporting back! We'll make sure to release a new spacy-transformers soon.

@tristan19954: thanks for sharing these results! It would be good to get to the bottom of this.

It looks like there are 254 instances in the training dataset for 11 NER labels. This might be a bit too few, depending on how close your new annotations are to what the model was originally trained on. You don't have a separate evaluation dataset, so 63 cases were selected at random to do the evaluation on, but these 63 might not be represented by the 254 training instances? Again, this depends a bit on the variability of your training dataset. A way to test this, is to run an artificial experiment with -n ner_teach_july,eval:ner_teach_july, which will effectively train AND evaluate on the same dataset. You typically want to avoid this, but for making sure the training mechanism works as expected, it would be a good check.

An important point is that when training on the ner_teach_july dataset, it might be the case that the model starts "forgetting" about previously learned instances, and starts overfitting on this dataset. With the Prodigy nightly you should be able to prevent this by feeding an additional NER dataset, so you can train on the "teach" dataset and another dataset simultaneously. Ideally you'd have a separate evaluation dataset that you've used both to analyse the original performance as well as the performance after training on the "teach" dataset (rather than the random split used here).

@SofieVL Thank you for the answer!

I stopped arround 300 because after that I was getting similar problems to what I reported in a previous post back in april. I seems to be working way longer/better now, but after 320 it started to just select the whole sentence and highlight that as an entity, which is why I tried to update the model at that point.

I will try out what you suggested and keep you updated! I still have access to my original training and evaluation data so I will try mixing that in too

1 Like

Strange behavior is happening since I update prodigy from 1.11.0a8 to 1.11.0a10, it is using more GPU Memory. I run the exact same command in both environments and I got these results:

python -m prodigy train --ner fbsio-140721 --base-model en_core_web_trf --gpu-id 0 model_output_140721

1.11.0a8

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.84       Driver Version: 460.84       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   76C    P0    41W /  70W |  10146MiB / 15109MiB |     87%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2910      C   python                          10143MiB |
+-----------------------------------------------------------------------------+

1.11.0a10

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.84       Driver Version: 460.84       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   70C    P0    31W /  70W |  15072MiB / 15109MiB |      80%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2451      C   python                          15069MiB |
+-----------------------------------------------------------------------------+

I've detected because I run out of memory:

⚠ Aborting and saving the final best model. Encountered exception:
RuntimeError('CUDA out of memory. Tried to allocate 264.00 MiB (GPU 0; 14.76 GiB
total capacity; 11.06 GiB already allocated; 37.75 MiB free; 11.91 GiB reserved
in total by PyTorch)')

Is it normal this difference in memory consumption?