How to do relation annotation after using bert.mer.manual

[ryanwesslen]

hi @Jiachen!

Thanks for your question and welcome to the Prodigy community

Have you seen this post?

Tokenization compatibility issues in rel.manual

If you're happy to just annotate the BPE tokens and relations between them, and don't care so much about aligning the tokens to spaCy's linguistic tokenization, you could also just load in pre-tokenized text using your tokenizer. Here's an example using a word piece tokenizer for NER annotation that aligns with a transformer model: https://prodi.gy/docs/named-entity-recognition#transformers-tokenizers You don't have to do it within the recipe – you could also use the logic as a preprocessing st…

The idea is to pass in the pre-tokenized text and avoiding spaCy's tokenizer. You'll need to create a custom recipe as rel.manual will assume you're using a spaCy tokenizer -- but as the post mentions, you can use the bert.ner.manual as an example of how to use the BPE tokenizer. Also, you can find the rel.manual recipe (for reference) in your installed Prodigy library. Run prodigy stats, find the Location: path (aka where Prodigy is installed), and then look for recipes/rel.py script.

Hope this helps!