Train recipe for parser: ValueError: Could not find gold transition

This is using prodigy version 1.11.4 on Ubuntu 20.04 and python 3.9.2.

I have annotated sentences using the rel.manual recipe, and am trying to train a model with the following command.

I have not been able to debug the "Could not find gold transition" error. Does anyone have any ideas?

Thanks so much!

prodigy train --parser annotations
2021-10-12 11:47:07.693324: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-12 11:47:07.693388: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-10-12 11:47:07.724601: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/count
2021-10-12 11:47:07.724681: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/count
2021-10-12 11:47:07.724692: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/api
2021-10-12 11:47:07.724698: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/api
2021-10-12 11:47:07.984034: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/count
2021-10-12 11:47:07.984103: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/count
2021-10-12 11:47:07.984154: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/write/api
2021-10-12 11:47:07.984187: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/saved_model/read/api
ℹ Using CPU

========================= Generating Prodigy config =========================
ℹ Auto-generating config with spaCy
✔ Generated training config

=========================== Initializing pipeline ===========================
[2021-10-12 11:47:09,956] [INFO] Set up nlp object from config
Components: parser
Merging training and evaluation data for 1 components
  - [parser] Training: 13825 | Evaluation: 3456 (20% split)
Training: 132 | Evaluation: 29
Labels: parser (4)
[2021-10-12 11:47:25,505] [INFO] Pipeline: ['tok2vec', 'parser']
[2021-10-12 11:47:25,511] [INFO] Created vocabulary
[2021-10-12 11:47:25,512] [INFO] Finished initializing nlp object
[2021-10-12 11:47:27,010] [INFO] Initialized pipeline components: ['tok2vec', 'parser']
✔ Initialized pipeline

============================= Training pipeline =============================
Components: parser
Merging training and evaluation data for 1 components
  - [parser] Training: 13825 | Evaluation: 3456 (20% split)
Training: 132 | Evaluation: 29
Labels: parser (4)
ℹ Pipeline: ['tok2vec', 'parser']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS PARSER  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
---  ------  ------------  -----------  -------  -------  -------  ------
  0       0          0.00        20.25     8.33     8.33   100.00    0.08
('S', 0, 9000.0)
('L-EXP', 0, 9000.0)
('L-dep', 0, 9000.0)
('R-dep', 0, 9000.0)
('B', 0, 9000.0)
('B-ROOT', 0, 9000.0)
('Gold sent starts?', 0, 0)
Traceback (most recent call last):
  File "/home/jaan/miniconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jaan/miniconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/__main__.py", line 61, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 331, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/plac_core.py", line 367, in call
    cmd, result = parser.consume(arglist)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/plac_core.py", line 232, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/recipes/train.py", line 277, in train
    return _train(
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/prodigy/recipes/train.py", line 197, in _train
    spacy_train(nlp, output_path, use_gpu=gpu_id, stdout=stdout)
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/home/jaan/miniconda3/lib/python3.9/site-packages/spacy/language.py", line 1122, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy/pipeline/transition_parser.pyx", line 403, in spacy.pipeline.transition_parser.Parser.update
  File "spacy/pipeline/transition_parser.pyx", line 515, in spacy.pipeline.transition_parser.Parser.get_batch_loss
  File "spacy/pipeline/_parser_internals/arc_eager.pyx", line 824, in spacy.pipeline._parser_internals.arc_eager.ArcEager.set_costs
ValueError: Could not find gold transition - see logs above.

This is probably due to a default setting in spacy that's not great for small datasets. If you export your data and train with spacy you can just change [components.parser.min_action_freq] in the config.

I think in prodigy you'll need to change how it generates its internal default config. I think you'd want to add something like this to prodigy/recipes/train.py around line 550 in generate_default_config():

    if "parser" in config["components"]:
        config["components"]["parser"]["min_action_freq"] = 1

We've brought up setting this by default in prodigy in the future, but the details haven't been finalized yet. (It would be a setting rather than hard-coded like this, but you can test this and see if it works.)