Train recipe for parser: ValueError: Could not find gold transition

[jaan]

This is using prodigy version 1.11.4 on Ubuntu 20.04 and python 3.9.2.

I have annotated sentences using the rel.manual recipe, and am trying to train a model with the following command.

I have not been able to debug the "Could not find gold transition" error. Does anyone have any ideas?

Thanks so much!

prodigy train --parser annotations
2021-10-12 11:47:07.693324: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-12 11:47:07.693388: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-10-12 11:47:07.724601: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/count
2021-10-12 11:47:07.724681: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/count
2021-10-12 11:47:07.724692: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/api
2021-10-12 11:47:07.724698: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/api
2021-10-12 11:47:07.984034: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/count
2021-10-12 11:47:07.984103: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/count
2021-10-12 11:47:07.984154: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/api
2021-10-12 11:47:07.984187: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/api
ℹ Using CPU

========================= Generating Prodigy config =========================
ℹ Auto-generating config with spaCy
✔ Generated training config

=========================== Initializing pipeline ===========================
[2021-10-12 11:47:09,956] [INFO] Set up nlp object from config
Components: parser
Merging training and evaluation data for 1 components
  - [parser] Training: 13825 | Evaluation: 3456 (20% split)
Training: 132 | Evaluation: 29
Labels: parser (4)
[2021-10-12 11:47:25,505] [INFO] Pipeline: ['tok2vec', 'parser']
[2021-10-12 11:47:25,511] [INFO] Created vocabulary
[2021-10-12 11:47:25,512] [INFO] Finished initializing nlp object
[2021-10-12 11:47:27,010] [INFO] Initialized pipeline components: ['tok2vec', 'parser']
✔ Initialized pipeline

============================= Training pipeline =============================
Components: parser
Merging training and evaluation data for 1 components
  - [parser] Training: 13825 | Evaluation: 3456 (20% split)
Training: 132 | Evaluation: 29
Labels: parser (4)
ℹ Pipeline: ['tok2vec', 'parser']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS PARSER  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
---  ------  ------------  -----------  -------  -------  -------  ------
  0       0          0.00        20.25     8.33     8.33   100.00    0.08
('S', 0, 9000.0)
('L-EXP', 0, 9000.0)
('L-dep', 0, 9000.0)
('R-dep', 0, 9000.0)
('B', 0, 9000.0)
('B-ROOT', 0, 9000.0)
('Gold sent starts?', 0, 0)
Traceback (most recent call last):
  File "/home/jaan/miniconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jaan/miniconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/__main__.py", line 61, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 331, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/plac_core.py", line 367, in call
    cmd, result = parser.consume(arglist)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/plac_core.py", line 232, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/recipes/train.py", line 277, in train
    return _train(
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/recipes/train.py", line 197, in _train
    spacy_train(nlp, output_path, use_gpu=gpu_id, stdout=stdout)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/language.py", line 1122, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy/pipeline/transition_parser.pyx", line 403, in spacy.pipeline.transition_parser.Parser.update
  File "spacy/pipeline/transition_parser.pyx", line 515, in spacy.pipeline.transition_parser.Parser.get_batch_loss
  File "spacy/pipeline/_parser_internals/arc_eager.pyx", line 824, in spacy.pipeline._parser_internals.arc_eager.ArcEager.set_costs
ValueError: Could not find gold transition - see logs above.

[adriane (Adriane Boyd)]

This is probably due to a default setting in spacy that's not great for small datasets. If you export your data and train with spacy you can just change [components.parser.min_action_freq] in the config.

I think in prodigy you'll need to change how it generates its internal default config. I think you'd want to add something like this to prodigy/recipes/train.py around line 550 in generate_default_config():

    if "parser" in config["components"]:
        config["components"]["parser"]["min_action_freq"] = 1

We've brought up setting this by default in prodigy in the future, but the details haven't been finalized yet. (It would be a setting rather than hard-coded like this, but you can test this and see if it works.)

[apatel415]

Hi Adriane,

I attempted this solution but still getting the same error on my relationship training. Do you have any other insight on what I can do to resolve this error?

Also note I have ~950 entries in my dataset so don’t think spacy considers that as small.

Thanks

[adriane (Adriane Boyd)]

Sorry, I don't have any other suggestions here. If you can post a full example causes issues, we can have a look to see if it might be a bug, but it sounds like maybe one label is still very rare (possibly due to projectivization?) despite the number of examples.

If you'd like to do a bit more debugging on the data, you could export it to .spacy and run spacy debug data -V config.cfg --paths.train ... to see more counts for the individual labels.

[shakyawar.developer (Himanshu)]

Hey @adriane,
I am facing the same issue, i have done the annotation using rel.manual recipe(for NER and relation) and when i was exporting the prodigy data to spacy, only --parser component data was exporting.

and when i was trying to train on the exported spacy data,
training fails with output erro " Aborting and saving the final best model. Encountered exception:
ValueError('Could not find gold transition - see logs above.')"

I have done 300 annotation.
Thanks

[ryanwesslen]

hi @shakyawar.developer!

Sorry for the confusion, but if you simply annotated relations and entities with rel.manual, the problem is data-to-spacy doesn't work because spaCy's Doc object doesn't have built-in support for relations.

prodigy data-to-spacy for relation extraction

At the time of writing, spaCy doesn't natively support relation extraction models. The example that we list on our docs here is meant to be a tutorial on how to set up a custom component, not a guide on a feature in spaCy. The crux of the issue is that the Doc object in spaCy currently has no support for relationships. That is also why, in turn, the .spacy object does not support them. The config file that you see can be changed via the --config flag (docs). If this flag is not set, which is y…

I just posted a similar post where I tried to provide steps of how you can modify Sofie's relations extraction video and code (be sure to watch segments where she describes the code):

Training a relation extraction component

Hi @stella! Yes, this is exactly the setup Sofie does. She explicitly says from the beginning she's going to assume she already has a trained ner component. Yes! Sofie used Thinc for training. You can see the training code here and she carefully explains the code in 8:11 to 18:30 the Thinc model script. She then describes around 22:55 an Overview of the TrainablePipe API and how to implement the custom component. You may not need to know all of the details and can luckily leverage a lot of t…

The key is you need to modify the parse_data.py script as mentioned here:

data-to-spacy for rel_component training

Hi @korneliaB , Ok, let us step back for a bit. I realized that since you already have the labeled documents in Prodigy, you can export them into .jsonl using the db-out command, then reuse / modify this parse_data.py script to convert the JSONL files into the spaCy format. The reason why it errored out is because it expects some labels before the component is initialized. You can see this being done in the main function. So you have to do something like: python scripts.parse_data path/to/js…

Hope this helps!