Annotate relationships on existing entities

[yllwpr]

I have already done the NER labeling and would now like to label relationships as well. How can I do this on my existing data? Is it possible to change and correct the entities when labeling the relationships? Is there anything I need to consider? (e.g. save in a new dataset).

[ryanwesslen]

hi @yllwpr!

Here's a few links and posts that may help.

@yllwpr

I have already done the NER labeling and would now like to label relationships as well. How can I do this on my existing data?

First, have you seen the Prodigy documentation on Combining Named Entity Recognition with relation extraction? This may answer a lot of your questions.

It also includes a trick that you can use Prodigy datasets as source inputs (instead of files) if you prefix it with dataset: like dataset:ner_rels_ent. Also, you can also use the suffix like reject, accept, or ignore too to the dataset in case you want to review only a subset of that Prodigy dataset. I would except you would only want to use the accept records.

Reviewing Ignored Cases

Hi @spothedog1 ! Are you aware of the :ignore (or could be :accept or :reject) suffix you can add to a dataset name (along with dataset: prefix) to review only ignored examples? For example: python -m prodigy rel.manual ner_ignore_data blank:en dataset:ner_data:ignore --label SUBJECT,OBJECT This will enable you to create a review dataset ner_ignore_data where you only review the ignored records from ner_data. This documentation provides more details. Have you tried to modify the "history_s…

@yllwpr

Is it possible to change and correct the entities when labeling the relationships?

The documentation on rel.manual can show how you can do both ner and rel jointly. Related, here's a post on how to review both (e.g., if you need to have an adjudicator between different annotators):

Review NER + Relation annotations jointly

This is definitely not expected and somehow must come down to the hashes that are generated – because these are used to decide whether two annotations are referring to the same input example (i.e. are different versions of the same annotation) or whether they're different tasks. If you look at the JSON data for the examples generated in the review workflow (especially disagreements on the same example), do they end up with different _input_hash values?

@yllwpr

Is there anything I need to consider? (e.g. save in a new dataset).

As the first link does it, it may be best to save your relationship data into a new a dataset. Also since you're considering multiple trainable components, it may be worth learning/using spacy custom config (config.cfg) files. You can pass a config.cfg file as an argument to your prodigy train function (or alternatively just use spacy train, which is really what prodigy train does). Here's a related post:

Relationship between named entities

This is a difficult question to answer, because it's not really trivial to recommend a generic approach for relation extraction that is bound to work in all domains. However, we did recently create an example implementation for a REL component as a spaCy 3 project. You can find it here: It contains a ML model implemented from scratch in Thinc, as well as a new trainable pipeline component rel that you can add to your pipeline. It requires the annotations to be encoded in the doc._.rel cust…

That post has a link to several open source projects available on our explosion/projects repo, like on ner and rel. I don't think one does both but perhaps can help you on how to think about structuring these projects.

Thanks again for your questions and let us know if you have any further ones!

[yllwpr]

Thank you for your detailed answer!