# Annotating strings without correct separation

**URL:** <https://support.prodi.gy/t/annotating-strings-without-correct-separation/7431>\
**Category:** Uncategorized\
**Tags:** ner, best-practices\
**Created:** [November 12, 2024, 4:00pm UTC](https://support.prodi.gy/t/annotating-strings-without-correct-separation/7431 "2024-11-12T16:00:11Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [November 13, 2024, 10:34am UTC](https://support.prodi.gy/t/annotating-strings-without-correct-separation/7431/2 "2024-11-13T10:34:00Z")

</div>

Hi @toadle,

If the NER architecture worked well so far I definitely wouldn't try to solve the tokenization issue by switching to the spancat architecture.  
The issue is about data preprocessing not really the modelling technique and that's where we should address it.  
The usual solution here would be to record some of these examples and see if you can fix the tokenization by rules. The best way to implement it would be to modify the default spaCy tokenizer by adding your custom rules so that you could easily integrate it in a spaCy pipeline both for annotation, training and production.  
It does require learning a bit a more about customizing spaCy components but the [documentation](https://spacy.io/usage/linguistic-features#special-cases) is excellent.

Prodigy `ner.manual` has the [character highlighting mode](https://prodi.gy/docs/named-entity-recognition#highlight-chars) that you can switch on and off from the UI. This would allow you to highlight subparts of a token but it wouldn't affect the tokenization, so you'd end up with spans that are misaligned with tokens and these would be rejected as training examples.  
The character-level highlighting is meant for models that predict character-based tags not token-based tags, but you could use it to "record" the mistokenized examples and then use this record to write your custom tokenization rules.

The easiest way to check if your data contains misaligned span annotations is to convert Prodigy annotated example with tokens and spans to a spaCy `doc`.  
If spans do not align with tokens, they are set to `None` and you can check for that in a simple Python script.  
So once you've done your annotation in Prodigy, you could process your data to with a script similar to the one below to fish out the misaligned examples and try to fix them with the custom tokenizer:

```python
def prodigy2spacy_ner(task: dict, nlp: Language) -> None:
    task_hash = task.get("_task_hash")
    tokens = task.get("tokens")
    words = [token["text"] for token in tokens]
    spaces = [token["ws"] for token in tokens]
    doc = Doc(nlp.vocab, words=words, spaces=spaces)
    
    prodigy_spans = task.get("spans", [])
    if prodigy_spans:
        spans = []
        for span in task["spans"]:
            spacy_span = doc.char_span(span["start"], span["end"], span["label"])
            if spacy_span is None:
                print(f"Misaligned span detected in example with task hash {task_hash}")
                print(f"Span: {span}")
                print()

```

Please see the related posts on dealing with similar "agglutinations" of words:

> [@Disable automatic selection of full word when using ner.correct recipe](https://support.prodi.gy/t/disable-automatic-selection-of-full-word-when-using-ner-correct-recipe/7106/2):
>
> Welcome to the forum @fernandorodriguespro , In ner.manual and spans.manual recipes you can add a toggle for switching between character and token-based annotation by calling them with the [--highlight-chars](https://prodi.gy/docs/named-entity-recognition#highlight-chars) flag. Be mindful, though, that this won't affect the tokenization so you'll end up with misaligned tokens and spans. In order to train a model afterwards, you'll need a tokenizer that will be able to recreate this same tokenization (See the important note in the docs referred above) The ty…

---

_[View the full topic](https://support.prodi.gy/t/annotating-strings-without-correct-separation/7431)._