# Good practice for unit separators and training

**URL:** <https://support.prodi.gy/t/good-practice-for-unit-separators-and-training/4668>\
**Category:** Uncategorized\
**Tags:** usage, ner\
**Created:** [September 8, 2021, 6:25am UTC](https://support.prodi.gy/t/good-practice-for-unit-separators-and-training/4668 "2021-09-08T06:25:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Balo](https://avatars.discourse-cdn.com/v4/letter/b/f04885/32.png) [@Balo](https://support.prodi.gy/u/Balo)\
**Post date:** [September 8, 2021, 6:25am UTC](https://support.prodi.gy/t/good-practice-for-unit-separators-and-training/4668/1 "2021-09-08T06:25:40Z")

</div>

Hi there,  
I am training a german NER model (spacy 2.3.7) with law texts as input in prodigy (ner.correct-Recipe). The input sometimes has unit separators in to divide longer german words at the end of a line. (e.g. "Ausla-gen"). This is also how the model will get the input in production. The Tokenizer handles the unit separator as its own token (e.g. "Ausla","\x1f","gen"). In my understanding this makes it more complicated for the model to learn, since "Auslagen" and "Ausla-gen" will be handled completely different. Naturally, "Ausla-gen" will occur only a handful of times. Do you have any advice or good practice how we should handle this? Is there a tokenizer fix that lets the model ignore the unit separators? Of course I could simply filter the unit separators. But - if feasible - I would like to leave the text structure intact , so I can easily map predicted entities back to the original text.  
Thanks in advance!

---

<div class="post-metadata">

**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\
**Post date:** [September 14, 2021, 5:24pm UTC](https://support.prodi.gy/t/good-practice-for-unit-separators-and-training/4668/2 "2021-09-14T17:24:58Z")

</div>

I think this is a case where it makes sense to apply some text normalization, as a preprocess. There's a function for this in textacy, although I haven't used it myself: [Text Preprocessing — textacy 0.11.0 documentation](https://textacy.readthedocs.io/en/latest/api_reference/preprocessing.html#normalize)

---

<div class="post-metadata">

**Author:** ![Balo](https://avatars.discourse-cdn.com/v4/letter/b/f04885/32.png) [@Balo](https://support.prodi.gy/u/Balo)\
**Post date:** [September 15, 2021, 8:10am UTC](https://support.prodi.gy/t/good-practice-for-unit-separators-and-training/4668/3 "2021-09-15T08:10:28Z")

</div>

Thanks a lot for the hint! I will definitely have a look at textacy.