# Can I use rel.manual without tokenization?

**URL:** <https://support.prodi.gy/t/can-i-use-rel-manual-without-tokenization/6167>\
**Category:** Uncategorized\
**Created:** [December 2, 2022, 5:02pm UTC](https://support.prodi.gy/t/can-i-use-rel-manual-without-tokenization/6167 "2022-12-02T17:02:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![shensmobile](https://avatars.discourse-cdn.com/v4/letter/s/7ea924/32.png) [@shensmobile](https://support.prodi.gy/u/shensmobile)\
**Post date:** [December 2, 2022, 5:02pm UTC](https://support.prodi.gy/t/can-i-use-rel-manual-without-tokenization/6167/1 "2022-12-02T17:02:53Z")

</div>

Hi, sorry for the re-post. I accidentally deleted the previous one.

I have used rel.manual to label some data successfully and trained a model directly in HuggingFace. I was looking for a good visualization tool to display the predictions and couldn't find anything as good as Displacy for Entities, so I thought "hey maybe I can just use Prodigy!"

Unfortunately, I am unable to get any of my predictions back into Prodigy. I have done this in the past with rel.manual by labelling it with my NER models from Huggingface and creating a jsonl with the following format in it

{  
"text": text,  
"spans": [{"start": 1  
"end": 5  
"label": label}]  
}

I tried doing the same thing except with relations

{  
"text": text,  
"spans": [{"start": 1,  
"end": 5,  
"label": "label"},  
{"start": 6,  
"end": 10,  
"label": "label"}],  
"relations": [{"head\_span": {"start": 1,  
"end": 5,  
"label": "label"}},  
{"child\_span": {"start": 6,  
"end": 10,  
"label": "label"}},  
"label": "related"}]  
}

Does rel.manual require tokens to function? If so, is there any way for me to skip the tokenization step? I am using a custom HuggingFace tokenizer and don't need to adjust the relations, just need to visualize them for now.

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [December 2, 2022, 7:20pm UTC](https://support.prodi.gy/t/can-i-use-rel-manual-without-tokenization/6167/2 "2022-12-02T19:20:10Z")

</div>

hi @shensmobile!

Prodigy does have [character-level highlighting](https://prodi.gy/docs/named-entity-recognition#highlight-chars) which essentially turns off tokenization:

> [@Fully manual NER annotations without tokeniser](https://support.prodi.gy/t/fully-manual-ner-annotations-without-tokeniser/2881/4):
>
> Just released Prodigy v1.10, which includes a --highlight-chars flag that lets you highlight characters instead. Also [see here](https://prodi.gy/docs/named-entity-recognition#highlight-chars) for details and examples.

As the link mentions:

> When using character-based highlighting, annotation may be slower and there’s no guarantee that the spans you annotate map to actual tokens later on. If your goal is to train a named entity recognizer, you should consider using the **same tokenizer** during annotation, to make sure that your data can be used. Also see the section on [efficient annotation for transformers](https://prodi.gy/docs/named-entity-recognition/#transformers-tokenizers) if you’re training a transformer-based model (e.g. BERT) with subword tokenization.

The only problem is these are more for span annotation (`ner` or `spancat`) and `--highlight-chars` isn't available for `rel.manual`. If you wanted, you could look at the `ner` recipes and try to modify your `rel.manual` to get something like it.

To view the recipes, find your Prodigy location (`python -m prodigy stats` and find the path to `Location:`). Then look for the `recipes/rel.py` (relations recipes) and the `recipes.ner.py` (ner recipes). Perhaps you can see how `--highlight-chars` is used in the `ner` recipes and try to replicate something for the `rel.manual`.

Since you're only visualizing the relations it seems like it could work -- but I'd need to take more time to test to confirm if it would. Let us know if you find anything!

---

<div class="post-metadata">

**Author:** ![shensmobile](https://avatars.discourse-cdn.com/v4/letter/s/7ea924/32.png) [@shensmobile](https://support.prodi.gy/u/shensmobile)\
**Post date:** [December 7, 2022, 12:52am UTC](https://support.prodi.gy/t/can-i-use-rel-manual-without-tokenization/6167/3 "2022-12-07T00:52:40Z")

</div>

Hi Ryan,

I ended up getting bored over the weekend and wrote code to port the BERT tokens into my .jsonl and used the bert.ner.manual.py as inspiration to make my own bert.rel.manual.py.

This was a great exercise in learning about how recipes work though! Next I'll try to port over --highlight-chars for future projects that use various different tokenizers!