# Training NER and relations extraction (RE) together

**URL:** <https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911>\
**Category:** Uncategorized\
**Tags:** usage, spacy, relations\
**Created:** [February 19, 2021, 9:44pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911 "2021-02-19T21:44:27Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrey](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@Andrey](https://support.prodi.gy/u/Andrey)\
**Post date:** [February 19, 2021, 9:44pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/1 "2021-02-19T21:44:27Z")

</div>

I'm fascinated by the [tutorial of Sofie on RE](https://www.youtube.com/watch?v=8HL-Ap5_Axo). I have a well annotated data set for both NER and RE tasks, but I couldn't find how to train a new model on both tasks.

Any pointers to possibly overlooked examples or documentation? Thanks!

---

<div class="post-metadata">

**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\
**Post date:** [February 22, 2021, 8:42am UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/2 "2021-02-22T08:42:12Z")

</div>

Hi! Happy to hear the REL tutorial was useful to you 🙂

The REL tutorial was meant as an example for implementing your own custom trainable component from scratch, and I think the provided implementation for relation extraction should really be taken as a baseline to start from. I can imagine a realistic application would benefit from additional features or a more complex network architecture.

That said - you can take the code from the [example project](https://github.com/explosion/projects/tree/v3/tutorials/rel_component) and construct a config file that refers to the relation extraction component and the NER at the same time. You can extend the [provided config](https://github.com/explosion/projects/blob/v3/tutorials/rel_component/configs/rel_tok2vec.cfg) with an NER component. If you need inspiration on how to define the NER component, you can run `python -m spacy init config -p "ner" ner_config.cfg` and merge that config with the REL one, so you'd have a pipeline including `tok2vec`, `ner` and `relation_extractor`. You can decide whether they should [shar](https://spacy.io/usage/embeddings-transformers#embedding-layers)e the same `tok2vec` layer or not.

If you feed in data that has both named entities annotated as well as the relations, it should train both simultaneously. You'll need to use the `-c` flag on the [`train` command](https://spacy.io/api/cli#train) to make sure the custom functions and architectures from the REL code are imported, because these are not built-in in spaCy. In the example project, this is accomplished by doing `-c custom_functions.py`.

To define your training data, you could follow the same conventions as the REL example project and store the information in the custom attribute `doc._.rel`, cf here: [projects/parse\_data.py at v3 · explosion/projects · GitHub](https://github.com/explosion/projects/blob/v3/tutorials/rel_component/scripts/parse_data.py#L86). After creating the appropriate `Doc` objects with the gold-standard data (entities + relations), you can serialize them to file with `DocBin` to create the binary `.spacy` files that you can feed into the `spacy train` command.

I think that's pretty much the general overview. Let me know if you run into specific issues!

---

<div class="post-metadata">

**Author:** ![Andrey](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@Andrey](https://support.prodi.gy/u/Andrey)\
**Post date:** [February 22, 2021, 8:40pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/3 "2021-02-22T20:40:11Z")

</div>

Fab as usual! I am experimenting with training both components (NER and REL) independently, but it seems like training both should be beneficial. I will then delve into your example and get back with questions and results.

Thanks Sofie.

---

<div class="post-metadata">

**Author:** ![kbrajwani](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/kbrajwani/32/2359_2.png) [@kbrajwani](https://support.prodi.gy/u/kbrajwani)\
**Post date:** [August 24, 2021, 7:01am UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/4 "2021-08-24T07:01:20Z")

</div>

HI @SofieVL I am trying to combine both ner and rel but it's seems fail.  
i am following following blog.

> **[How to Train a Joint Entities and Relation Extraction Classifier using BERT...](https://towardsdatascience.com/how-to-train-a-joint-entities-and-relation-extraction-classifier-using-bert-transformer-with-spacy-49eb08d91b5c)**
>
> A step-by-step guide on how to train a relation extraction classifier using Transformer and spaCy3

> **[GitHub - walidamamou/relation\_extraction\_transformer](https://github.com/walidamamou/relation_extraction_transformer)**
>
> Contribute to walidamamou/relation\_extraction\_transformer development by creating an account on GitHub.

The github repo provide the sample data in spacy format which include ents and rels.  
Following are the changes i have done in config

```python
%%writefile /content/relation_extraction_transformer/rel_component/configs/rel_trf.cfg
[paths]
train = null
dev = null
raw = null
init_tok2vec = null

[system]
seed = 342
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer", "ner","relation_extractor"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
batch_size = 1000

[components]

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 64
stride = 48

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor]
factory = "relation_extractor"
threshold = 0.5

[components.relation_extractor.model]
@architectures = "rel_model.v1"

[components.relation_extractor.model.create_instance_tensor]
@architectures = "rel_instance_tensor.v1"

[components.relation_extractor.model.create_instance_tensor.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.relation_extractor.model.create_instance_tensor.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.get_instances]
@misc = "rel_instance_generator.v1"
max_length = 20

[components.relation_extractor.model.classification_layer]
@architectures = "rel_classification_layer.v1"
nI = null
nO = null

[initialize]

[initialize.components]

[corpora]

[corpora.dev]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.dev}

[corpora.train]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.train}

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600000
max_epochs = 0
max_steps = 1000
eval_frequency = 100
frozen_components = []
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null
logger = {"@loggers":"spacy.ConsoleLogger.v1"}

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0

```

I have only changed the config file can you please tell me what other changes i have to do?

---

<div class="post-metadata">

**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\
**Post date:** [August 24, 2021, 1:14pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/5 "2021-08-24T13:14:34Z")

</div>

Hi @kbrajwani, if the question is about the repository at [GitHub - walidamamou/relation\_extraction\_transformer](https://github.com/walidamamou/relation_extraction_transformer), it might make more sense to contact the authors there specifically. If you're running into spaCy-specific issues, it would be good to elaborate a bit on what exactly the problem is (what code are you executing and which errors are you getting), and to post a new message detailing that over at [Discussions · explosion/spaCy · GitHub](https://github.com/explosion/spaCy/discussions). If you're running into Prodigy-specific issues, can you elaborate further?

Thanks!

---

<div class="post-metadata">

**Author:** ![kbrajwani](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/kbrajwani/32/2359_2.png) [@kbrajwani](https://support.prodi.gy/u/kbrajwani)\
**Post date:** [August 24, 2021, 2:38pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/6 "2021-08-24T14:38:29Z")

</div>

I am only taking data from that repo.  
i have following error.

```python
================================= train_gpu =================================
Running command: /usr/bin/python3 -m spacy train configs/rel_trf.cfg --output training --paths.train /content/relation_extraction_transformer/relations_training.spacy --paths.dev /content/relation_extraction_transformer/relations_dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-08-24 14:36:12,074] [INFO] Set up nlp object from config
[2021-08-24 14:36:12,083] [INFO] Pipeline: ['transformer', 'ner', 'relation_extractor']
[2021-08-24 14:36:12,087] [INFO] Created vocabulary
[2021-08-24 14:36:12,088] [INFO] Finished initializing nlp object
Downloading: 100% 481/481 [00:00<00:00, 441kB/s]
Downloading: 100% 899k/899k [00:00<00:00, 2.98MB/s]
Downloading: 100% 456k/456k [00:00<00:00, 1.50MB/s]
Downloading: 100% 1.36M/1.36M [00:00<00:00, 4.31MB/s]
Downloading: 100% 501M/501M [00:07<00:00, 63.2MB/s]
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.weight', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2021-08-24 14:36:40,591] [INFO] Initialized pipeline components: ['transformer', 'ner', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner', 'relation_extractor']
ℹ Initial learn rate: 0.0
E # LOSS TRANS... LOSS NER LOSS RELAT... ENTS_F ENTS_P ENTS_R REL_MICRO_P REL_MICRO_R REL_MICRO_F SCORE 
--- ------ ------------- -------- ------------- ------ ------ ------ ----------- ----------- ----------- ------
⚠ Aborting and saving the final best model. Encountered exception:
KeyError("[E900] Could not run the full pipeline for evaluation. If you
specified frozen components, make sure they were already initialized and
trained. Full pipeline: ['transformer', 'ner', 'relation_extractor']")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 281, in evaluate
    scores = nlp.evaluate(dev_corpus(nlp))
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1389, in evaluate
    results = scorer.score(examples)
  File "/usr/local/lib/python3.7/dist-packages/spacy/scorer.py", line 135, in score
    scores.update(component.score(examples, **self.cfg))
  File "/content/relation_extraction_transformer/rel_component/scripts/rel_pipe.py", line 201, in score
    return score_relations(examples, self.threshold)
  File "/content/relation_extraction_transformer/rel_component/scripts/rel_pipe.py", line 211, in score_relations
    gold_labels = [k for (k, v) in gold[key].items() if v == 1.0]
KeyError: (0, 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    " __main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ __main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params) # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 226, in train_while_improving
    score, other_scores = evaluate()
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 283, in evaluate
    raise KeyError(Errors.E900.format(pipeline=nlp.pipe_names)) from e
KeyError: "[E900] Could not run the full pipeline for evaluation. If you specified frozen components, make sure they were already initialized and trained. Full pipeline: ['transformer', 'ner', 'relation_extractor']"

```

---

<div class="post-metadata">

**Author:** ![SofieVL](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sofievl/32/915_2.png) [@SofieVL](https://support.prodi.gy/u/SofieVL)\
**Post date:** [August 24, 2021, 3:59pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/7 "2021-08-24T15:59:45Z")

</div>

Ok, right. Can you open a new issue on the spaCy discussion forum? That's a more suited place as this is not Prodigy related. Thanks!

---

<div class="post-metadata">

**Author:** ![kbrajwani](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/kbrajwani/32/2359_2.png) [@kbrajwani](https://support.prodi.gy/u/kbrajwani)\
**Post date:** [August 25, 2021, 6:05am UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/8 "2021-08-25T06:05:25Z")

</div>

sure. Thanks

---

<div class="post-metadata">

**Author:** ![karndeepsingh](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/karndeepsingh/32/2388_2.png) [@karndeepsingh](https://support.prodi.gy/u/karndeepsingh)\
**Post date:** [September 3, 2021, 6:58pm UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/9 "2021-09-03T18:58:39Z")

</div>

Hi,  
I want to train NER and REL model for my usecase. For training NER and REL model i need to annotate data and save it in spacy binary format. But I have a doubt in annotation of data, I am using Label Studio for annotation and it is doing the annotation for Relation as well as NER but how should I convert its output to .spacy binary format?  
Basically, I want to ask, what is expected structure format of the annotate .txt file. As After getting output from Label studio,I can see lot of unccessary key value pairs as well.  
Please help me to understand the structure or format of annotatd to be fed for conversion into .spacy binary format.

Thanks.

---

<div class="post-metadata">

**Author:** ![IldarAskarov0405](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ildaraskarov0405/32/3023_2.png) [@IldarAskarov0405](https://support.prodi.gy/u/IldarAskarov0405)\
**Post date:** [June 10, 2022, 7:19am UTC](https://support.prodi.gy/t/training-ner-and-relations-extraction-re-together/3911/10 "2022-06-10T07:19:40Z")

</div>

Hi, have you succeeded in your usecase?