# Training a relation extraction component **URL:** \ **Category:** Uncategorized\ **Tags:** solved, relations, training\ **Created:** [February 20, 2023, 10:19am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376 "2023-02-20T10:19:15Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [April 18, 2023, 12:59pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/62 "2023-04-18T12:59:55Z")
After editing parse\_data\_generic and executing python3 -m spacy project run train\_joint\_cpu , I obtain the following error when executing : ```python File "/home///venv/lib/python3.10/site-packages/thinc/layers/reduce_mean.py", line 19, in forward Y = model.ops.reduce_mean(cast(Floats2d, Xr.data), Xr.lengths) File "thinc/backends/numpy_ops.pyx", line 318, in thinc.backends.numpy_ops.NumpyOps.reduce_mean AssertionError ``` I've made sure numpy is correctly installed. Is it a configuration issue ? On the contrary, train\_cpu is working, but once again, I want to point something really strange about the performance. Here is the ouput of train\_cpu : ```python ℹ Saving to output directory: training ℹ Using CPU =========================== Initializing pipeline =========================== [2023-04-18 14:45:39,983] [INFO] Set up nlp object from config [2023-04-18 14:45:39,989] [INFO] Pipeline: ['tok2vec', 'relation_extractor'] [2023-04-18 14:45:39,991] [INFO] Created vocabulary [2023-04-18 14:45:39,992] [INFO] Finished initializing nlp object [2023-04-18 14:45:40,041] [INFO] Initialized pipeline components: ['tok2vec', 'relation_extractor'] ✔ Initialized pipeline ============================= Training pipeline ============================= ℹ Pipeline: ['tok2vec', 'relation_extractor'] ℹ Initial learn rate: 0.001 E # LOSS TOK2VEC LOSS RELAT... REL_MICRO_P REL_MICRO_R REL_MICRO_F SCORE --- ------ ------------ ------------- ----------- ----------- ----------- ------ 0 0 0.30 2.84 0.85 50.00 1.67 0.02 88 500 1.26 25.69 0.00 0.00 0.00 0.00 208 1000 0.03 15.26 0.00 0.00 0.00 0.00 376 1500 0.04 11.92 0.00 0.00 0.00 0.00 661 2000 0.05 8.87 0.00 0.00 0.00 0.00 1161 2500 0.09 5.40 0.00 0.00 0.00 0.00 1661 3000 0.08 5.39 0.00 0.00 0.00 0.00 2161 3500 0.08 5.39 0.00 0.00 0.00 0.00 2661 4000 0.08 5.38 0.00 0.00 0.00 0.00 3161 4500 0.08 5.25 0.00 0.00 0.00 0.00 3661 5000 0.08 5.24 0.00 0.00 0.00 0.00 4161 5500 0.08 5.25 0.00 0.00 0.00 0.00 4661 6000 0.08 5.25 0.00 0.00 0.00 0.00 5161 6500 0.07 5.24 0.00 0.00 0.00 0.00 5661 7000 0.07 5.24 0.00 0.00 0.00 0.00 6161 7500 0.10 5.25 0.00 0.00 0.00 0.00 6661 8000 0.09 5.24 0.00 0.00 0.00 0.00 7161 8500 0.09 5.24 0.00 0.00 0.00 0.00 7661 9000 0.08 5.24 0.00 0.00 0.00 0.00 8161 9500 0.08 5.24 0.00 0.00 0.00 0.00 8661 10000 0.08 5.24 0.00 0.00 0.00 0.00 ✔ Saved pipeline to output directory training/model-last ``` On the same annotations, training a NER model alone enables to recognize numerous named entities on a text. But a model trained with train\_cpu can't recognize even a single named entity on the same text. I'll post this question on the discussions hub, but I want to make sure that it's not a bug, before. Thanks ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [April 19, 2023, 3:53pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/63 "2023-04-19T15:53:37Z")
Hi Ryan, Any idea ? I should really solve this this week, it's a case of emergency. 😁 Thanks ! ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [April 19, 2023, 4:13pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/64 "2023-04-19T16:13:27Z")
> [@stella](#): > > I obtain the following error when executing : > > ```python > File "/home///venv/lib/python3.10/site-packages/thinc/layers/reduce_mean.py", line 19, in forward > Y = model.ops.reduce_mean(cast(Floats2d, Xr.data), Xr.lengths) > File "thinc/backends/numpy_ops.pyx", line 318, in thinc.backends.numpy_ops.NumpyOps.reduce_mean > AssertionError > > ``` > > I've made sure numpy is correctly installed. Is it a configuration issue ? So this isn't a numpy error - it's an error raised by Thinc in it's `NumpyOps` class which is the class doing numerical operations on CPU. It's complaining about shapes, which suggests it's likely a data issue, e.g. no annotations of something making the internal vector shapes different than what Thinc is expecting. But it's difficult to say more with just that single error snippet. So from this, it's more likely there's an issue with your data. > [@stella](#): > > train\_cpu is working, but once again, I want to point something really strange about the performance. > [@stella](#): > > But a model trained with train\_cpu can't recognize even a single named entity on the same text. Your `train_cpu` was only training `relation_extractor`, see: ```python =========================== Initializing pipeline =========================== [2023-04-18 14:45:39,983] [INFO] Set up nlp object from config [2023-04-18 14:45:39,989] [INFO] Pipeline: ['tok2vec', 'relation_extractor'] ``` That would explain why it's not predicting entities because it's not training a `ner` component. I would suggest posting on spaCy discussions forum with the simplest reproducible example (e.g., `config.cfg` file, Python/SpaCy version numbers, and a couple examples of your data). ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [April 21, 2023, 12:34pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/65 "2023-04-21T12:34:39Z")
Hi Ryan, You were right, my data had an issue, now it works ! Thank you for your help. Please thank Sofie who left a message earlier in the thread, as well as all the team who helped enhance the code. All the best, Stella ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [May 31, 2023, 3:37pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/66 "2023-05-31T15:37:50Z")
Hello again ! I am still having a data issue, probably caused by a relation existing to something that is not labeled as a named entity (might be an error from my side). This time, I don't have few examples to correct but a bigger dataset. Is there a method to check / correct this kind of issue with the annotation tool ? Best, ---
**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\ **Post date:** [June 2, 2023, 8:37am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/67 "2023-06-02T08:37:29Z")
(Ryan is on holiday, which is why I am jumping in) > [@stella](#): > > a relation existing to something that is not labeled as a named entity Just an idea from my end that might help debug: you could run Prodigy with a batch size of one so that the stream just gives one example at a time. That way, when the error occurs ... you should be able to pin-point an example where the error is happening. Once you have such an example it should be relatively easy to figure out a fix. If you know the token that the example is pointing at, I imagine it should be easy to add it as an entity to the example before it is sent for annotation. Does this help? ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 5, 2023, 7:33am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/68 "2023-06-05T07:33:12Z")
Hello, No problem ! I'm not sure it would be a fix as the error is only shown when training the relation model (so after using prodigy recipes). Or do you think it would do the trick ? If so, how should I use this ? I imagined that if a recipe for correcting annotations would exist, it would help, but I didn't find anything in the documentation (except for review, but it only merges two datasets). Thanks ! ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [June 5, 2023, 5:24pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/69 "2023-06-05T17:24:43Z")
hi @stella, > [@stella](#): > > Is there a method to check / correct this kind of issue with the annotation tool ? So the `parse_generic_code.py` Sofie created does have a step that checks and will skip over relations that don't have entities: > This is a more general way to check whether the start of the `child` or `head` of the `relations` are within spans. Another simple way is to check if `rel["head_span"]["label"]` or `rel["child_span"]["label"]` (i.e., the head and child full spans) have labels (i.e., is not `Null`). Recall, that in the `relations` recipe, the output data will look like [this](https://prodi.gy/docs/api-interfaces#relations). Here's a function that would take 1 example (`eg`) and would check for this. ```python def validate_answer(eg): relations = eg.get("relations", []) errors = [] for rel in relations: if rel["head_span"]["label"] is None: errors.append("Head relation is not an entity") if rel["child_span"]["label"] is None: errors.append("Child relation is not an entity") if errors: raise ValueError(" ".join(errors)) ``` What's nice with this function is that with Prodigy's [`validate_answer` callback](https://prodi.gy/docs/custom-recipes#validate_answer), you can create a custom recipe largely based on the `relations` recipe but with this validation. This would enable you to write a check to prevent annotators from accepting situations where the annotator accidentally forgot to make the relation between entities. I've taken the existing `rel.manual` recipe and added the `validate_answer` callback above and put it as a GitHub gist so you can see: > This is what the user will see if they provide an invalid relation (i.e., head or child isn't also an entity). ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/1/15a875660c2ba8a8c4b8486f7912d10d22e1e6a2.jpeg) Hope this helps! ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 14, 2023, 12:09pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/71 "2023-06-14T12:09:50Z")
Hello Ryan, You are right, it seems like the error isn't triggered by an invalid relation. I am still using Sofie's original script. With my actual annotation dataset, the following error is triggered : ```python ==================================== data ==================================== Running command: venv/bin/python3 ./scripts/parse_data_generic.py assets/annotations.jsonl data/train.spacy data/dev.spacy data/test.spacy venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") Traceback (most recent call last): File "nlp/components/rel_component/./scripts/parse_data_generic.py", line 144, in typer.run(main) File "nlp/components/rel_component/./scripts/parse_data_generic.py", line 56, in main span["start"], span["end"], label=span["label"] KeyError: 'start' ``` My annotation dataset does contain start attribute for spans, for each annotation it seems. Do you have any idea ? ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 15, 2023, 9:44am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/72 "2023-06-15T09:44:24Z")
I'll add more information to help you debug this. Upon deleting all examples from annotations.jsonl except one, the keyerror 'start' is not triggered anymore. I am positive that there is no relation between anything else than named entities. I am using Sofie's original script, as I previously said. I use Prodigy webtool to annotate. On the first example, the triggered error is : ```python File "venv/lib/python3.10/site-packages/spacy/pipeline/tok2vec.py", line 216, in initialize assert doc_sample, Errors.E923.format(name=self.name) AssertionError: [E923] It looks like there is no proper sample data to initialize the Model of component 'tok2vec'. To check your input data paths and annotation, run: python -m spacy debug data config.cfg and include the same config override values you would specify for the 'spacy train' command. ``` I've tried to debug the config files. First, rel\_joint.cfg : ```python python3 -m spacy debug data configs/rel_joint.cfg ``` ```python ValueError: [E002] Can't find factory for 'relation_extractor' for language English (en). This usually happens when spaCy calls `nlp.create_pipe` with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator `@Language.component` (for function components) or `@Language.factory` (for class components). Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, lemmatizer, trainable_lemmatizer, entity_linker, ner, beam_ner, entity_ruler, tagger, morphologizer, senter, sentencizer, textcat, spancat, future_entity_ruler, span_ruler, textcat_multilabel, en.lemmatizer ``` Same for rel\_tok2vec.cfg : ```python Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, lemmatizer, trainable_lemmatizer, entity_linker, ner, beam_ner, entity_ruler, tagger, morphologizer, senter, sentencizer, textcat, spancat, future_entity_ruler, span_ruler, textcat_multilabel, en.lemmatizer ``` and for rel\_trf : ```python Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, lemmatizer, trainable_lemmatizer, entity_linker, ner, beam_ner, entity_ruler, tagger, morphologizer, senter, sentencizer, textcat, spancat, future_entity_ruler, span_ruler, textcat_multilabel, en.lemmatizer ``` Is there anything I'm missing ? ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [June 15, 2023, 12:38pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/73 "2023-06-15T12:38:52Z")
Hey @stella! Thanks for your patience. Can you add to the `spacy debug data` `-c ./scripts/custom_functions.py`? You need to pass that `custom_function.py` script which has the factory for `relation_extractor`. > [@Training a relation extraction component](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/22): > > As per the error, it sounds like you may not be passing the tutorial’s config that creates this component. Are you running your command [like](https://github.com/explosion/projects/blob/ab30f61bebe7b39100243c82c28932dc89a008ea/tutorials/rel_component/project.yml#L53) the tutorial does: python -m spacy train configs/rel\_tok2vec.cfg --output training --paths.train train.spacy --paths.dev dev.spacy -c ./scripts/custom\_functions.py I'm definitely curious on what `debug data` will provide. ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 16, 2023, 7:21am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/74 "2023-06-16T07:21:57Z")
Hi Ryan, Here is the command line and its output : ```python python3 -m spacy debug data configs/rel_joint.cfg -c ./scripts/custom_functions.py /venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ✘ Config validation error before_update field required {'seed': 342, 'gpu_allocator': None, 'dropout': 0.1, 'accumulate_gradient': 1, 'patience': 1600000, 'max_epochs': 0, 'max_steps': 10000, 'eval_frequency': 500, 'frozen_components': [], 'dev_corpus': 'corpora.dev', 'train_corpus': 'corpora.train', 'before_to_disk': None, 'annotating_components': ['ner'], 'logger': {'@loggers': 'spacy.ConsoleLogger.v1'}, 'batcher': {'@batchers': 'spacy.batch_by_words.v1', 'discard_oversize': False, 'tolerance': 0.2, 'get_length': None, 'size': {'@schedules': 'compounding.v1', 'start': 100, 'stop': 1000, 'compound': 1.001}}, 'optimizer': {'@optimizers': 'Adam.v1', 'beta1': 0.9, 'beta2': 0.999, 'L2_is_weight_decay': True, 'L2': 0.01, 'grad_clip': 1.0, 'use_averages': False, 'eps': 1e-08, 'learn_rate': 0.001}, 'score_weights': {'ents_f': 0.5, 'ents_p': 0.0, 'ents_r': 0.0, 'ents_per_type': None, 'rel_micro_p': 0.0, 'rel_micro_r': 0.0, 'rel_micro_f': 0.5}} If your config contains missing values, you can run the 'init fill-config' command to fill in all the defaults, if possible: python -m spacy init fill-config configs/rel_joint.cfg configs/rel_joint.cfg ``` I'm using a script to automate the model training, I'm running : ```python #!/bin/bash rm -rf data mkdir data rm -rf training mkdir training python3 -m spacy project assets python3 -m spacy project run data python3 -m spacy project run train_joint_cpu ``` Should I replace the last command by : ```python python -m spacy train configs/rel_tok2vec.cfg --output training --paths.train train.spacy --paths.dev dev.spacy -c ./scripts/custom_functions.py ``` ? Thank you ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [June 16, 2023, 7:17pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/75 "2023-06-16T19:17:24Z")
> [@stella](#): > > ```python > ✘ Config validation error > before_update field required > > ``` Quick glance - you're missing the `before_update` field. Can you try to fill it and and retry? ```python python -m spacy init fill-config configs/rel_joint.cfg configs/rel_joint.cfg ``` ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 19, 2023, 7:11am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/76 "2023-06-19T07:11:20Z")
It is still raising the same error : ```python (venv) rel_component$ python3 -m spacy init fill-config configs/rel_joint.cfg configs/rel_joint.cfg venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "venv/lib/python3.10/site-packages/spacy/ __main__.py", line 4, in setup_cli() File "venv/lib/python3.10/site-packages/spacy/cli/_util.py", line 74, in setup_cli command(prog_name=COMMAND) File "venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "venv/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(**use_params) # type: ignore File "venv/lib/python3.10/site-packages/spacy/cli/init_config.py", line 101, in init_fill_config_cli fill_config(output_file, base_path, pretraining=pretraining, diff=diff) File "venv/lib/python3.10/site-packages/spacy/cli/init_config.py", line 117, in fill_config nlp = util.load_model_from_config(config, auto_fill=True, validate=False) File "venv/lib/python3.10/site-packages/spacy/util.py", line 554, in load_model_from_config nlp = lang_cls.from_config( File "venv/lib/python3.10/site-packages/spacy/language.py", line 1803, in from_config nlp.add_pipe( File "venv/lib/python3.10/site-packages/spacy/language.py", line 786, in add_pipe pipe_component = self.create_pipe( File "venv/lib/python3.10/site-packages/spacy/language.py", line 660, in create_pipe raise ValueError(err) ValueError: [E002] Can't find factory for 'relation_extractor' for language English (en). This usually happens when spaCy calls `nlp.create_pipe` with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator `@Language.component` (for function components) or `@Language.factory` (for class components). Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, lemmatizer, trainable_lemmatizer, entity_linker, ner, beam_ner, entity_ruler, tagger, morphologizer, senter, sentencizer, textcat, spancat, future_entity_ruler, span_ruler, textcat_multilabel, en.lemmatizer ``` Here is rel\_joint.cfg : ```python [paths] train = null dev = null vectors = null init_tok2vec = null [system] seed = 342 gpu_allocator = null [nlp] lang = "en" pipeline = ["tok2vec","ner","relation_extractor"] disabled = [] before_creation = null after_creation = null after_pipeline_creation = null tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"} batch_size = 1000 [components] [components.ner] factory = "ner" incorrect_spans_key = "incorrect_spans" moves = null scorer = {"@scorers":"spacy.ner_scorer.v1"} update_with_oracle_cut_size = 100 [components.ner.model] @architectures = "spacy.TransitionBasedParser.v2" state_type = "ner" extra_state_tokens = false hidden_width = 64 maxout_pieces = 2 use_upper = true nO = null [components.ner.model.tok2vec] @architectures = "spacy.Tok2VecListener.v1" width = 96 upstream = "*" [components.tok2vec] factory = "tok2vec" [components.tok2vec.model] @architectures = "spacy.HashEmbedCNN.v1" pretrained_vectors = null width = 96 depth = 2 embed_size = 2000 window_size = 1 maxout_pieces = 3 subword_features = true [components.relation_extractor] factory = "relation_extractor" threshold = 0.5 [components.relation_extractor.model] @architectures = "rel_model.v1" [components.relation_extractor.model.create_instance_tensor] @architectures = "rel_instance_tensor.v1" [components.relation_extractor.model.create_instance_tensor.tok2vec] @architectures = "spacy.Tok2VecListener.v1" width = ${components.tok2vec.model.width} [components.relation_extractor.model.create_instance_tensor.pooling] @layers = "reduce_mean.v1" [components.relation_extractor.model.create_instance_tensor.get_instances] @misc = "rel_instance_generator.v1" max_length = 100 [components.relation_extractor.model.classification_layer] @architectures = "rel_classification_layer.v1" nI = null nO = null [corpora] [corpora.dev] @readers = "spacy.Corpus.v1" path = ${paths.dev} max_length = 0 gold_preproc = false limit = 0 augmenter = null [corpora.train] @readers = "spacy.Corpus.v1" path = ${paths.train} max_length = 0 gold_preproc = false limit = 0 augmenter = null [training] seed = ${system.seed} gpu_allocator = ${system.gpu_allocator} dropout = 0.1 accumulate_gradient = 1 patience = 1600000 max_epochs = 0 max_steps = 10000 eval_frequency = 500 frozen_components = [] dev_corpus = "corpora.dev" train_corpus = "corpora.train" before_to_disk = null annotating_components = ["ner"] logger = {"@loggers":"spacy.ConsoleLogger.v1"} [training.batcher] @batchers = "spacy.batch_by_words.v1" discard_oversize = false tolerance = 0.2 get_length = null [training.batcher.size] @schedules = "compounding.v1" start = 100 stop = 1000 compound = 1.001 [training.optimizer] @optimizers = "Adam.v1" beta1 = 0.9 beta2 = 0.999 L2_is_weight_decay = true L2 = 0.01 grad_clip = 1.0 use_averages = false eps = 0.00000001 learn_rate = 0.001 [training.score_weights] rel_micro_p = 0.0 rel_micro_r = 0.0 rel_micro_f = 1.0 [initialize] vectors = ${paths.vectors} init_tok2vec = ${paths.init_tok2vec} vocab_data = null lookups = null before_init = null after_init = null [initialize.components] [initialize.tokenizer] ``` custom\_functions.py : ```python from functools import partial from pathlib import Path from typing import Iterable, Callable import spacy from spacy.training import Example from spacy.tokens import DocBin, Doc # make the factory work from scripts.rel_pipe import make_relation_extractor # make the config work from scripts.rel_model import create_relation_model, create_classification_layer, create_instances, create_tensors @spacy.registry.readers("Gold_ents_Corpus.v1") def create_docbin_reader(file: Path) -> Callable[["Language"], Iterable[Example]]: return partial(read_files, file) def read_files(file: Path, nlp: "Language") -> Iterable[Example]: """Custom reader that keeps the tokenization of the gold data, and also adds the gold GGP annotations as we do not attempt to predict these.""" doc_bin = DocBin().from_disk(file) docs = doc_bin.get_docs(nlp.vocab) for gold in docs: pred = Doc( nlp.vocab, words=[t.text for t in gold], spaces=[t.whitespace_ for t in gold], ) pred.ents = gold.ents yield Example(pred, gold) ``` evaluate.py : ```python import random import typer from pathlib import Path import spacy from spacy.tokens import DocBin, Doc from spacy.training.example import Example # make the factory work from rel_pipe import make_relation_extractor, score_relations # make the config work from rel_model import create_relation_model, create_classification_layer, create_instances, create_tensors def main(trained_pipeline: Path, test_data: Path, print_details: bool): nlp = spacy.load(trained_pipeline) doc_bin = DocBin(store_user_data=True).from_disk(test_data) docs = doc_bin.get_docs(nlp.vocab) examples = [] for gold in docs: pred = Doc( nlp.vocab, words=[t.text for t in gold], spaces=[t.whitespace_ for t in gold], ) pred.ents = gold.ents for name, proc in nlp.pipeline: pred = proc(pred) examples.append(Example(pred, gold)) # Print the gold and prediction, if gold label is not 0 if print_details: print() print(f"Text: {gold.text}") print(f"spans: {[(e.start, e.text, e.label_) for e in pred.ents]}") for value, rel_dict in pred._.rel.items(): gold_labels = [k for (k, v) in gold._.rel[value].items() if v == 1.0] if gold_labels: print( f" pair: {value} --> gold labels: {gold_labels} --> predicted values: {rel_dict}" ) print() random_examples = [] docs = doc_bin.get_docs(nlp.vocab) for gold in docs: pred = Doc( nlp.vocab, words=[t.text for t in gold], spaces=[t.whitespace_ for t in gold], ) pred.ents = gold.ents relation_extractor = nlp.get_pipe("relation_extractor") get_instances = relation_extractor.model.attrs["get_instances"] for (e1, e2) in get_instances(pred): offset = (e1.start, e2.start) if offset not in pred._.rel: pred._.rel[offset] = {} for label in relation_extractor.labels: pred._.rel[offset][label] = random.uniform(0, 1) random_examples.append(Example(pred, gold)) thresholds = [0.000, 0.050, 0.100, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99, 0.999] print() print("Random baseline:") _score_and_format(random_examples, thresholds) print() print("Results of the trained model:") _score_and_format(examples, thresholds) def _score_and_format(examples, thresholds): for threshold in thresholds: r = score_relations(examples, threshold) results = {k: "{:.2f}".format(v * 100) for k, v in r.items()} print(f"threshold {'{:.2f}'.format(threshold)} \t {results}") if __name__ == " __main__": typer.run(main) ``` parse\_data\_generic.py (with anonymized labels) : ```python # This script was derived from parse_data.py but made more generic as a template for various REL parsing needs import json import random import typer from pathlib import Path from spacy.tokens import DocBin, Doc from spacy.vocab import Vocab from wasabi import Printer msg = Printer() # TODO: define your labels used for annotation either as "symmetrical" or "directed" SYMM_LABELS = ["D", "E"] DIRECTED_LABELS = ["A", "B", "C"] # TODO: define splits for train/dev/test. What is not in test or dev, will be used as train. test_portion = 0.1#0.2 dev_portion = 0.1#0.3 # TODO: set this bool to False if you didn't annotate all relations in all sentences. # If it's true, entities that were not annotated as related will be used as negative examples. is_complete = False def main(json_loc: Path, train_file: Path, dev_file: Path, test_file: Path): """Creating the corpus from the Prodigy annotations.""" Doc.set_extension("rel", default={}) vocab = Vocab() docs = {"train": [], "dev": [], "test": []} count_all = {"train": 0, "dev": 0, "test": 0} count_pos = {"train": 0, "dev": 0, "test": 0} with json_loc.open("r", encoding="utf8") as jsonfile: for line in jsonfile: example = json.loads(line) span_starts = set() if example["answer"] == "accept": neg = 0 pos = 0 # Parse the tokens words = [t["text"] for t in example["tokens"]] spaces = [t["ws"] for t in example["tokens"]] doc = Doc(vocab, words=words, spaces=spaces) # Parse the entities spans = example["spans"] entities = [] span_end_to_start = {} for span in spans: entity = doc.char_span( span["start"], span["end"], label=span["label"] ) span_end_to_start[span["token_end"]] = span["token_start"] entities.append(entity) span_starts.add(span["token_start"]) if not entities: msg.warn("Could not parse any entities from the JSON file.") doc.ents = entities # Parse the relations rels = {} for x1 in span_starts: for x2 in span_starts: rels[(x1, x2)] = {} relations = example["relations"] for relation in relations: # Ignoring relations that are not between spans (they are annotated on the token level if not relation["head"] in span_end_to_start or not relation["child"] in span_end_to_start: msg.warn(f"This script only supports relationships between annotated entities.") break # the 'head' and 'child' annotations refer to the end token in the span # but we want the first token start = span_end_to_start[relation["head"]] end = span_end_to_start[relation["child"]] label = relation["label"] if label not in SYMM_LABELS + DIRECTED_LABELS: msg.warn(f"Found label '{label}' not defined in SYMM_LABELS or DIRECTED_LABELS - skipping") break if label not in rels[(start, end)]: rels[(start, end)][label] = 1.0 pos += 1 if label in SYMM_LABELS: if label not in rels[(end, start)]: rels[(end, start)][label] = 1.0 pos += 1 # If the annotation is complete, fill in zero's where the data is missing if is_complete: for x1 in span_starts: for x2 in span_starts: for label in SYMM_LABELS + DIRECTED_LABELS: if label not in rels[(x1, x2)]: neg += 1 rels[(x1, x2)][label] = 0.0 doc._.rel = rels # only keeping documents with at least 1 positive case if pos > 0: # create the train/dev/test split randomly # Note that this is not good practice as instances from the same article # may end up in different splits. Ideally, change this method to keep # documents together in one split (as in the original parse_data.py) if random.random() < test_portion: docs["test"].append(doc) count_pos["test"] += pos count_all["test"] += pos + neg elif random.random() < (test_portion + dev_portion): docs["dev"].append(doc) count_pos["dev"] += pos count_all["dev"] += pos + neg else: docs["train"].append(doc) count_pos["train"] += pos count_all["train"] += pos + neg docbin = DocBin(docs=docs["train"], store_user_data=True) docbin.to_disk(train_file) msg.info( f"{len(docs['train'])} training sentences, " f"{count_pos['train']}/{count_all['train']} pos instances." ) docbin = DocBin(docs=docs["dev"], store_user_data=True) docbin.to_disk(dev_file) msg.info( f"{len(docs['dev'])} dev sentences, " f"{count_pos['dev']}/{count_all['dev']} pos instances." ) docbin = DocBin(docs=docs["test"], store_user_data=True) docbin.to_disk(test_file) msg.info( f"{len(docs['test'])} test sentences, " f"{count_pos['test']}/{count_all['test']} pos instances." ) if __name__ == " __main__": typer.run(main) ``` rel\_model.py : ```python from typing import List, Tuple, Callable import spacy from spacy.tokens import Doc, Span from thinc.types import Floats2d, Ints1d, Ragged, cast from thinc.api import Model, Linear, chain, Logistic @spacy.registry.architectures("rel_model.v1") def create_relation_model( create_instance_tensor: Model[List[Doc], Floats2d], classification_layer: Model[Floats2d, Floats2d], ) -> Model[List[Doc], Floats2d]: with Model.define_operators({">>": chain}): model = create_instance_tensor >> classification_layer model.attrs["get_instances"] = create_instance_tensor.attrs["get_instances"] return model @spacy.registry.architectures("rel_classification_layer.v1") def create_classification_layer( nO: int = None, nI: int = None ) -> Model[Floats2d, Floats2d]: with Model.define_operators({">>": chain}): return Linear(nO=nO, nI=nI) >> Logistic() @spacy.registry.misc("rel_instance_generator.v1") def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def get_instances(doc: Doc) -> List[Tuple[Span, Span]]: instances = [] for ent1 in doc.ents: for ent2 in doc.ents: if ent1 != ent2: if max_length and abs(ent2.start - ent1.start) <= max_length: instances.append((ent1, ent2)) return instances return get_instances @spacy.registry.architectures("rel_instance_tensor.v1") def create_tensors( tok2vec: Model[List[Doc], List[Floats2d]], pooling: Model[Ragged, Floats2d], get_instances: Callable[[Doc], List[Tuple[Span, Span]]], ) -> Model[List[Doc], Floats2d]: return Model( "instance_tensors", instance_forward, layers=[tok2vec, pooling], refs={"tok2vec": tok2vec, "pooling": pooling}, attrs={"get_instances": get_instances}, init=instance_init, ) def instance_forward(model: Model[List[Doc], Floats2d], docs: List[Doc], is_train: bool) -> Tuple[Floats2d, Callable]: pooling = model.get_ref("pooling") tok2vec = model.get_ref("tok2vec") get_instances = model.attrs["get_instances"] all_instances = [get_instances(doc) for doc in docs] tokvecs, bp_tokvecs = tok2vec(docs, is_train) ents = [] lengths = [] for doc_nr, (instances, tokvec) in enumerate(zip(all_instances, tokvecs)): token_indices = [] for instance in instances: for ent in instance: token_indices.extend([i for i in range(ent.start, ent.end)]) lengths.append(ent.end - ent.start) ents.append(tokvec[token_indices]) lengths = cast(Ints1d, model.ops.asarray(lengths, dtype="int32")) entities = Ragged(model.ops.flatten(ents), lengths) pooled, bp_pooled = pooling(entities, is_train) # Reshape so that pairs of rows are concatenated relations = model.ops.reshape2f(pooled, -1, pooled.shape[1] * 2) def backprop(d_relations: Floats2d) -> List[Doc]: d_pooled = model.ops.reshape2f(d_relations, d_relations.shape[0] * 2, -1) d_ents = bp_pooled(d_pooled).data d_tokvecs = [] ent_index = 0 for doc_nr, instances in enumerate(all_instances): shape = tokvecs[doc_nr].shape d_tokvec = model.ops.alloc2f(*shape) count_occ = model.ops.alloc2f(*shape) for instance in instances: for ent in instance: d_tokvec[ent.start : ent.end] += d_ents[ent_index] count_occ[ent.start : ent.end] += 1 ent_index += ent.end - ent.start d_tokvec /= count_occ + 0.00000000001 d_tokvecs.append(d_tokvec) d_docs = bp_tokvecs(d_tokvecs) return d_docs return relations, backprop def instance_init(model: Model, X: List[Doc] = None, Y: Floats2d = None) -> Model: tok2vec = model.get_ref("tok2vec") if X is not None: tok2vec.initialize(X) return model ``` rel\_pipe.py : ```python from itertools import islice from typing import Tuple, List, Iterable, Optional, Dict, Callable, Any from spacy.scorer import PRFScore from thinc.types import Floats2d import numpy from spacy.training.example import Example from thinc.api import Model, Optimizer from spacy.tokens.doc import Doc from spacy.pipeline.trainable_pipe import TrainablePipe from spacy.vocab import Vocab from spacy import Language from thinc.model import set_dropout_rate from wasabi import Printer Doc.set_extension("rel", default={}, force=True) msg = Printer() @Language.factory( "relation_extractor", requires=["doc.ents", "token.ent_iob", "token.ent_type"], assigns=["doc._.rel"], default_score_weights={ "rel_micro_p": None, "rel_micro_r": None, "rel_micro_f": None, }, ) def make_relation_extractor( nlp: Language, name: str, model: Model, *, threshold: float ): """Construct a RelationExtractor component.""" return RelationExtractor(nlp.vocab, model, name, threshold=threshold) class RelationExtractor(TrainablePipe): def __init__ ( self, vocab: Vocab, model: Model, name: str = "rel", *, threshold: float, ) -> None: """Initialize a relation extractor.""" self.vocab = vocab self.model = model self.name = name self.cfg = {"labels": [], "threshold": threshold} @property def labels(self) -> Tuple[str]: """Returns the labels currently added to the component.""" return tuple(self.cfg["labels"]) @property def threshold(self) -> float: """Returns the threshold above which a prediction is seen as 'True'.""" return self.cfg["threshold"] def add_label(self, label: str) -> int: """Add a new label to the pipe.""" if not isinstance(label, str): raise ValueError("Only strings can be added as labels to the RelationExtractor") if label in self.labels: return 0 self.cfg["labels"] = list(self.labels) + [label] return 1 def __call__ (self, doc: Doc) -> Doc: """Apply the pipe to a Doc.""" # check that there are actually any candidate instances in this batch of examples total_instances = len(self.model.attrs["get_instances"](doc)) if total_instances == 0: msg.info("Could not determine any instances in doc - returning doc as is.") return doc predictions = self.predict([doc]) self.set_annotations([doc], predictions) return doc def predict(self, docs: Iterable[Doc]) -> Floats2d: """Apply the pipeline's model to a batch of docs, without modifying them.""" get_instances = self.model.attrs["get_instances"] total_instances = sum([len(get_instances(doc)) for doc in docs]) if total_instances == 0: msg.info("Could not determine any instances in any docs - can not make any predictions.") scores = self.model.predict(docs) return self.model.ops.asarray(scores) def set_annotations(self, docs: Iterable[Doc], scores: Floats2d) -> None: """Modify a batch of `Doc` objects, using pre-computed scores.""" c = 0 get_instances = self.model.attrs["get_instances"] for doc in docs: for (e1, e2) in get_instances(doc): offset = (e1.start, e2.start) if offset not in doc._.rel: doc._.rel[offset] = {} for j, label in enumerate(self.labels): doc._.rel[offset][label] = scores[c, j] c += 1 def update( self, examples: Iterable[Example], *, drop: float = 0.0, set_annotations: bool = False, sgd: Optional[Optimizer] = None, losses: Optional[Dict[str, float]] = None, ) -> Dict[str, float]: """Learn from a batch of documents and gold-standard information, updating the pipe's model. Delegates to predict and get_loss.""" if losses is None: losses = {} losses.setdefault(self.name, 0.0) set_dropout_rate(self.model, drop) # check that there are actually any candidate instances in this batch of examples total_instances = 0 for eg in examples: total_instances += len(self.model.attrs["get_instances"](eg.predicted)) if total_instances == 0: msg.info("Could not determine any instances in doc.") return losses # run the model docs = [eg.predicted for eg in examples] predictions, backprop = self.model.begin_update(docs) loss, gradient = self.get_loss(examples, predictions) backprop(gradient) if sgd is not None: self.model.finish_update(sgd) losses[self.name] += loss if set_annotations: self.set_annotations(docs, predictions) return losses def get_loss(self, examples: Iterable[Example], scores) -> Tuple[float, float]: """Find the loss and gradient of loss for the batch of documents and their predicted scores.""" truths = self._examples_to_truth(examples) gradient = scores - truths mean_square_error = (gradient ** 2).sum(axis=1).mean() return float(mean_square_error), gradient def initialize( self, get_examples: Callable[[], Iterable[Example]], *, nlp: Language = None, labels: Optional[List[str]] = None, ): """Initialize the pipe for training, using a representative set of data examples. """ if labels is not None: for label in labels: self.add_label(label) else: for example in get_examples(): relations = example.reference._.rel for indices, label_dict in relations.items(): for label in label_dict.keys(): self.add_label(label) self._require_labels() subbatch = list(islice(get_examples(), 10)) doc_sample = [eg.reference for eg in subbatch] label_sample = self._examples_to_truth(subbatch) if label_sample is None: raise ValueError("Call begin_training with relevant entities and relations annotated in " "at least a few reference examples!") self.model.initialize(X=doc_sample, Y=label_sample) def _examples_to_truth(self, examples: List[Example]) -> Optional[numpy.ndarray]: # check that there are actually any candidate instances in this batch of examples nr_instances = 0 for eg in examples: nr_instances += len(self.model.attrs["get_instances"](eg.reference)) if nr_instances == 0: return None truths = numpy.zeros((nr_instances, len(self.labels)), dtype="f") c = 0 for i, eg in enumerate(examples): for (e1, e2) in self.model.attrs["get_instances"](eg.reference): gold_label_dict = eg.reference._.rel.get((e1.start, e2.start), {}) for j, label in enumerate(self.labels): truths[c, j] = gold_label_dict.get(label, 0) c += 1 truths = self.model.ops.asarray(truths) return truths def score(self, examples: Iterable[Example], **kwargs) -> Dict[str, Any]: """Score a batch of examples.""" return score_relations(examples, self.threshold) def score_relations(examples: Iterable[Example], threshold: float) -> Dict[str, Any]: """Score a batch of examples.""" micro_prf = PRFScore() for example in examples: gold = example.reference._.rel pred = example.predicted._.rel for key, pred_dict in pred.items(): gold_labels = [k for (k, v) in gold.get(key, {}).items() if v == 1.0] for k, v in pred_dict.items(): if v >= threshold: if k in gold_labels: micro_prf.tp += 1 else: micro_prf.fp += 1 else: if k in gold_labels: micro_prf.fn += 1 return { "rel_micro_p": micro_prf.precision, "rel_micro_r": micro_prf.recall, "rel_micro_f": micro_prf.fscore, } ``` ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 19, 2023, 8:13am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/77 "2023-06-19T08:13:50Z")
Hi Stella, Whenever you are working with a config file that has a custom registered function in it, like the `relation_extractor`, you need to add `-c custom_functions.py` to your command. Otherwise, spaCy can't interpret the config and will raise this same error that it can't find the function as you can read in the error message: ```python ValueError: [E002] Can't find factory for 'relation_extractor' ``` `init fill-config` also has this `-c` flag available to you, as you can see here: [Command Line Interface · spaCy API Documentation](https://spacy.io/api/cli#init-fill-config) ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 19, 2023, 9:29am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/78 "2023-06-19T09:29:51Z")
Hi Sofie, Great, now the init fill-config command works. But I think there is still something missing in my workflow. What I'm doing is : Executing a script to get my annotations : ```python #!/bin/bash cd ../../../model_training/prodigy_scripts ./get_annotations.sh cd .. cp prodigy_annotations/datasets/my_dataset.jsonl ../nlp/components/rel_component/assets/annotations.jsonl ``` Executing a script to train the model for relation extraction : ```python #!/bin/bash rm -rf data mkdir data rm -rf training mkdir training python3 -m spacy project assets python3 -m spacy project run data python3 -m spacy project run train_joint_cpu ``` Output of the last script : ```python (venv) rel_component$ ./relation_model_training.sh venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ℹ Fetching 1 asset(s) ✔ Asset already exists: rel_component/assets/annotations.jsonl venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ==================================== data ==================================== Running command: venv/bin/python3 ./scripts/parse_data_generic.py assets/annotations.jsonl data/train.spacy data/dev.spacy data/test.spacy venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ℹ 0 training sentences, 0/0 pos instances. ℹ 0 dev sentences, 0/0 pos instances. ℹ 0 test sentences, 0/0 pos instances. venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ============================== train_joint_cpu ============================== Running command: venv/bin/python3 -m spacy train configs/rel_joint.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py venv/lib/python3.10/site-packages/torch/cuda/ __init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") ℹ Saving to output directory: training ℹ Using CPU =========================== Initializing pipeline =========================== [2023-06-19 11:05:14,161] [INFO] Set up nlp object from config [2023-06-19 11:05:14,169] [INFO] Pipeline: ['tok2vec', 'ner', 'relation_extractor'] [2023-06-19 11:05:14,171] [INFO] Created vocabulary [2023-06-19 11:05:14,172] [INFO] Finished initializing nlp object Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "venv/lib/python3.10/site-packages/spacy/ __main__.py", line 4, in setup_cli() File "venv/lib/python3.10/site-packages/spacy/cli/_util.py", line 74, in setup_cli command(prog_name=COMMAND) File "venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "venv/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(**use_params) # type: ignore File "venv/lib/python3.10/site-packages/spacy/cli/train.py", line 45, in train_cli train(config_path, output_path, use_gpu=use_gpu, overrides=overrides) File "venv/lib/python3.10/site-packages/spacy/cli/train.py", line 72, in train nlp = init_nlp(config, use_gpu=use_gpu) File "venv/lib/python3.10/site-packages/spacy/training/initialize.py", line 84, in init_nlp nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer) File "venv/lib/python3.10/site-packages/spacy/language.py", line 1308, in initialize proc.initialize(get_examples, nlp=self, **p_settings) File "venv/lib/python3.10/site-packages/spacy/pipeline/tok2vec.py", line 216, in initialize assert doc_sample, Errors.E923.format(name=self.name) AssertionError: [E923] It looks like there is no proper sample data to initialize the Model of component 'tok2vec'. To check your input data paths and annotation, run: python -m spacy debug data config.cfg and include the same config override values you would specify for the 'spacy train' command. ``` It looks like -c /scripts/custom\_functions.py is correctly passed. My project.yml file : ```python title: "Example project of creating a novel nlp component to do relation extraction from scratch." description: "This example project shows how to implement a spaCy component with a custom Machine Learning model, how to train it with and without a transformer, and how to apply it on an evaluation dataset." # Variables can be referenced across the project.yml using ${vars.var_name} vars: annotations: "assets/annotations.jsonl" tok2vec_config: "configs/rel_tok2vec.cfg" trf_config: "configs/rel_trf.cfg" joint_config: "configs/rel_joint.cfg" train_file: "data/train.spacy" dev_file: "data/dev.spacy" test_file: "data/test.spacy" trained_model: "training/model-best" # These are the directories that the project needs. The project CLI will make # sure that they always exist. directories: ["scripts", "configs", "assets", "data", "training"] # Assets that should be downloaded or available in the directory. You can replace # this with your own input data. assets: - dest: ${vars.annotations} description: "Gold-standard REL annotations created with Prodigy" workflows: all: - data - train_cpu - evaluate all_gpu: - data - train_gpu - evaluate # Project commands, specified in a style similar to CI config files (e.g. Azure # pipelines). The name is the command name that lets you trigger the command # via "spacy project run [command] [path]". The help message is optional and # shown when executing "spacy project run [optional command] [path] --help". commands: - name: "data" help: "Parse the gold-standard annotations from the Prodigy annotations." script: - "python ./scripts/parse_data_generic.py ${vars.annotations} ${vars.train_file} ${vars.dev_file} ${vars.test_file}" deps: - ${vars.annotations} outputs: - ${vars.train_file} - ${vars.dev_file} - ${vars.test_file} - name: "train_cpu" help: "Train the REL model on the CPU and evaluate on the dev corpus." script: - "python -m spacy train ${vars.tok2vec_config} --output training --paths.train ${vars.train_file} --paths.dev ${vars.dev_file} -c ./scripts/custom_functions.py" deps: - ${vars.train_file} - ${vars.dev_file} outputs: - ${vars.trained_model} - name: "train_joint_cpu" help: "Jointly train the NER and REL model on the CPU and evaluate on the dev corpus." script: - "python -m spacy train ${vars.joint_config} --output training --paths.train ${vars.train_file} --paths.dev ${vars.dev_file} -c ./scripts/custom_functions.py" deps: - ${vars.train_file} - ${vars.dev_file} outputs: - ${vars.trained_model} - name: "train_gpu" help: "Train the REL model with a Transformer on a GPU and evaluate on the dev corpus." script: - "python -m spacy train ${vars.trf_config} --output training --paths.train ${vars.train_file} --paths.dev ${vars.dev_file} -c ./scripts/custom_functions.py --gpu-id 0" deps: - ${vars.train_file} - ${vars.dev_file} outputs: - ${vars.trained_model} - name: "evaluate" help: "Apply the best model to new, unseen text, and measure accuracy at different thresholds." script: - "python ./scripts/evaluate.py ${vars.trained_model} ${vars.test_file} False" deps: - ${vars.trained_model} - ${vars.test_file} - name: "clean" help: "Remove intermediate files to start data preparation and training from a clean slate." script: - "rm -rf data/*" - "rm -rf training/*" ``` ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 19, 2023, 9:35am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/79 "2023-06-19T09:35:04Z")
Hi Stella, This thread has been going on for quite a while now. Often, you can find very useful hints as to what is going on by carefully reading the error messages. I'm afraid we can't be online 24/7 just to help you debug your scripts at every single step, as we want to make sure we also have sufficient time to help others and to work on features & releases. In this particular case, you mention that "you think something is missing", but can't you find more details in the log output & error message? What is going wrong? How can you try to fix this? ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 19, 2023, 9:49am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/80 "2023-06-19T09:49:05Z")
Hey Sofie, Of course I've tried to fix it myself, but unfortunately I didn't succeed. I thought maybe the thread was also useful for the team as it helped you correct the component at some point. I really think there's still something going on, as I'm constantly struggling with data format errors when getting around 30 annotations (and I don't think I'm doing any mistake during the annotation process / when using the web annotation tool). My team has bought a license on my recommendation and counts on me. I understand you don't have the time to answer me "24/7", but I deeply regret that it kinda looks like I'm an inconvenience or I'm somewhat incompetent to debug it myself. Could someone look into it for a little time ? It's the same error from 4 days ago, actually. ---
**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\ **Post date:** [June 19, 2023, 11:54am UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/81 "2023-06-19T11:54:58Z")
Hi Stella, I haven't said you're an "inconvenience", but I think it's reasonable to acknowledge the time and effort our team has already spent on this in the past 4 months. Many of the error messages are in fact self explanatory and we've tried adding additional explanations as you encountered them. I think it's only fair to ask that you try fixing things yourself as well, like adding the `-c` flag as explained earlier and in the documentation. Anyway. To pick up from the error you shared 4 days ago, right before the weekend, Ryan asked about the output of `spacy debug data`, did you actually end up sharing that? From your last logs in today's message, it doesn't look like you're passing any valid data onto the scripts. ---
**Author:** ![stella](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stella](https://support.prodi.gy/u/stella)\ **Post date:** [June 19, 2023, 1:27pm UTC](https://support.prodi.gy/t/training-a-relation-extraction-component/6376/82 "2023-06-19T13:27:27Z")
Sofie, First of all, as I said, your team has benefited from my thread as it greatly helped you correct the component. So your efforts do not benefit only me, but also benefit you, and that's probably why you spent "4 months" on this thread in the first place (actually from end of February to end of April, so 2 months, and then I added few questions 15 days ago and now 5 days ago, so it's not 4 months "24/7" and it's fair to acknowledge that also). I'd like to add that the marketing of Prodigy is to annotate your data easily and train models such a relation extraction model, and it's also implicitly marketed as having this feature "out of the box" : [Dependencies & Relations · Prodigy · An annotation tool for AI, Machine Learning & NLP](https://prodi.gy/features/dependencies-relations) (because why could you annotate data if it was not meant to train a model ?) In reality, you need to use a component that's coded outside the API and configure it, it is quite complex and the video tutorial, as helpful as it is, is (for me at least) incomplete compared to the information that has been made available in this thread. Maybe the integration of the component in the API will be part of the next release (I'd think it would be quite clever to integrate it more naturally in the API) and it will be easier to use (like the NER model training that is very simple to use, even for lazy and dumb people like me). Then, we should not forget that we are customers, and the licence is not free. In a professional context, people can use the tool and not be a data scientist, a dev... so maybe self explanatory things for you are not for others and people buy your solution so it can help them solve a problem, not spending months to configure it. I think I've proven myself to be patient and I've tried to debug many things on my own (not asking anything between end of April and end of May, trying to debug it on my own). Ryan has been very helpful and respectful all along (thanking me for my patience, being available and exhaustive even if maybe it annoyed him). I don't really get why you're implying that it's reasonable to let things not working because you already spent too much time on that. It's a key feature. Maybe because you think I'm not doing any effort or it's not worthy to spend time on my issue ? I should naturally be the one losing patience and I'm not, I'm still giving credit to your solution. Maybe my issue is interesting for you also, because at some point, the annotation dataset looks messy for no particular reason, and I'll say it again, I don't think I'm making any mistake in the annotation process (and I've checked that during that month multiple times). By the way, this -c flag is related to the fill-config command only, I didn't forget it in the relation extraction model training. If my error is self-explanatory, maybe you can point it out. I'll ignore your noticeable annoyance from now on and hope we both don't continue losing time on that. The spacy debug data output is the following : ```python ValueError: [E913] Corpus path can't be None. Maybe you forgot to define it is your .cfg file or override it on the CLI ? ``` The error message may look self-explanatory but I don't get why it's triggered. rel\_joint.cfg is untouched, paths in Ryan's rel\_joint.cfg are null too. Many thanks. [Previous page](https://support.prodi.gy/t/training-a-relation-extraction-component/6376.md?page=3) [Next page](https://support.prodi.gy/t/training-a-relation-extraction-component/6376.md?page=5)