# How to tell SpaCy not to split any intra-hyphen words?

**URL:** <https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456>\
**Category:** Uncategorized\
**Tags:** spacy, solved\
**Created:** [April 30, 2019, 5:49am UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456 "2019-04-30T05:49:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZeeshanAli](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zeeshanali/32/754_2.png) [@ZeeshanAli](https://support.prodi.gy/u/ZeeshanAli)\
**Post date:** [April 30, 2019, 5:49am UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/1 "2019-04-30T05:49:54Z")

</div>

Solutions to the problem that are available so far create other unwanted modifications to the rules to tokenize. For example, some methods would not then split “can’t” like words. Some methods would split text into sentences at every dot. The ideal solution that I need and am looking for would be that which does not split any intra-hyphen words and doesn’t create any other unwanted changes to the split rules whatsoever.

Following is an example of problematic solutions available [here](https://stackoverflow.com/questions/51012476/spacy-custom-tokenizer-to-include-only-hyphen-words-as-tokens-using-infix-regex);

```
import re
import spacy
from spacy.tokenizer import Tokenizer
from spacy.util import compile_prefix_regex, compile_infix_regex, compile_suffix_regex

def custom_tokenizer(nlp):
    infix_re = re.compile(r'''[.\,\?\:\;\...\‘\’\`\“\”\"\'~]''')
    prefix_re = compile_prefix_regex(nlp.Defaults.prefixes)
    suffix_re = compile_suffix_regex(nlp.Defaults.suffixes)

    return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
                                suffix_search=suffix_re.search,
                                infix_finditer=infix_re.finditer,
                                token_match=None)

nlp = spacy.load('en')
nlp.tokenizer = custom_tokenizer(nlp)

doc = nlp(u'Note: Since the fourteenth century the practice of “medicine” has become a profession; and more importantly, it\'s a male-dominated profession.')
[token.text for token in doc]

```

**Output:**

```
['Note', ':', 'Since', 'the', 'fourteenth', 'century', 'the', 'practice', 'of', '“', 'medicine', '”', 'has', 'become', 'a', 'profession', ';', 'and', 'more', 'importantly', ',', 'it', "'s", 'a', 'male-dominated', 'profession', '.']

```

**Question:** How does r’’’[.,?:;…\‘\’`\“\”"’~]’’’ solve the problem of intra-hyphen words when there is no hyphen in there?

The above solution creates, for example, the following problem (incorrect tokenization of “can’t”);

```
doc = nlp("This can't be it.")
print ([token.text for token in doc])
print ([(token.text, token.tag_) for token in doc]) 

```

**Output:**

```
['This', 'can', "'", 't', 'be', 'it', '.']
[('This', 'DT'), ('can', 'MD'), ("'", '``'), ('t', 'NN'), ('be', 'VB'), ('it', 'PRP'), ('.', '.')]

```

**Expected output:**

```
['This', 'ca', "n't", 'be', 'it', '.']
[('This', 'DT'), ('ca', 'MD'), ("n't", 'RB'), ('be', 'VB'), ('it', 'PRP'), ('.', '.')]
```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [April 30, 2019, 10:42am UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/2 "2019-04-30T10:42:45Z")

</div>

The reason your custom tokenizer implementation doesn't split exceptions like `"can't"` anymore is that you're not actually passing in any `rules`. Those are the tokenizer exceptions that define special cases, like contractions in English. Also see [the `Tokenizer` API](https://spacy.io/api/tokenizer#init) for the possible arguments.

> [@ZeeshanAli](#):
>
> **Question:** How does r’’’[.,?:;…\‘\’`\“\”"’~]’’’ solve the problem of intra-hyphen words when there is no hyphen in there?

The infix rules define the rules to determine **how to split** inside a token. If the include a hyphen, spaCy will always split intra-hyphen words. If they only match a hyphen within two capital letters, spaCy will only split hyphens between capital letters. If they include no hyphens, spaCy won't split on hyphens at all.

If you only want to make small modifications to your rules like that, you might find it easier to [add to an existing rule set](https://spacy.io/usage/linguistic-features#native-tokenizer-additions) instead of creating an entirely new tokenizer from scratch.

---

<div class="post-metadata">

**Author:** ![ZeeshanAli](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zeeshanali/32/754_2.png) [@ZeeshanAli](https://support.prodi.gy/u/ZeeshanAli)\
**Post date:** [April 30, 2019, 10:51am UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/3 "2019-04-30T10:51:04Z")

</div>

Can I just take out the rule to split intra-hyphen words from the set of infix rules?

```
for infix_ in nlp.Defaults.infixes:
    if '-' in infix_:
        print (infix_)

```

**Output:**

```
(?<=[0-9])[+\-\*^](?=[0-9-])
(?<=[[[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])\.(?=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])
(?<=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]]),(?=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])
(?<=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])[?";:=,.]*(?:-|–|—|--|---|——|~)(?=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])
(?<=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]"])[:<>=/](?=[[[\p{Lu}&&\p{Latin}]||[ЁА-Я]||[ӘӨҮҖҢҺ]||[Α-ΩΆΈΊΌΏΉΎ]||[\p{Ll}&&\p{Latin}]||[ёа-я]||[әөүҗңһ]||[α-ωάέίόώήύ]||[\p{L}&&\p{Bengali}]||[\p{L}&&\p{Hebrew}]||[\p{L}&&\p{Arabic}]||[\p{L}&&\p{Sinhala}]]])

```

How can I do that then? I can’t find any specific portion from the infixes that I need to remove.

---

<div class="post-metadata">

**Author:** ![ZeeshanAli](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zeeshanali/32/754_2.png) [@ZeeshanAli](https://support.prodi.gy/u/ZeeshanAli)\
**Post date:** [April 30, 2019, 10:53am UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/4 "2019-04-30T10:53:11Z")

</div>

If I add anything to the set of suffix rules, there is gonna be mistagging somewhere. I therefore think it would be great if I could “_ **just take out the rule to split intra-hyphen words from the set of infix rules** _”.

For example;

```
suffixes = nlp.Defaults.suffixes + (r'''\w+-\w+''',)
suffix_regex = spacy.util.compile_suffix_regex(suffixes)
nlp.tokenizer.suffix_search = suffix_regex.search

```

The above code solves the problem of intra-hyphen words. However, I fear, this might have caused issues somewhere. In order to verify, I’ll test it. But I would like to know if there was a better way. ^^

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [April 30, 2019, 4:26pm UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/5 "2019-04-30T16:26:49Z")

</div>

Yeah, the thing with tokenization is that there are always trade-offs. The default rules try to optimise for the best possible compromise of performance and accuracy – but there'll always be cases where it over-segments or under-segments. Rule-based tokenization (like spaCy's) can make it more difficult to handle context-specific cases, but statistical tokenization is often much slower and instead makes mistakes in different places.

> [@ZeeshanAli](#):
>
> I therefore think it would be great if I could “ _ **just take out the rule to split intra-hyphen words from the set of infix rules** _ ”.

The rule you're looking for is on line 4 of the rules you posted. It takes care of splitting various hyphens and punctuation characters if they occur between letters. There are no single rules that _only_ handle single characters, because that's not very efficient. So there's also not a single option to "take out character X".

If there are specific cases that you know will always have to be tokenized a certain way, you could also add a custom pipeline component that [merges or splits](https://spacy.io/usage/linguistic-features#retokenization) them afterwards.

---

<div class="post-metadata">

**Author:** ![tsaksena](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tsaksena](https://support.prodi.gy/u/tsaksena)\
**Post date:** [June 5, 2019, 6:30pm UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/6 "2019-06-05T18:30:06Z")

</div>

> [@ZeeshanAli](#):
>
> import

Hi

I am very new to spacy/prodigy so apologies if I am missing something obvious, but I am unsure how to use a custom tokeniser in prodigy. I also need to ensure my text is not split up on hyphens, and the custom tokeniser similar to above works fine for me on spacy. When I am trying to annotate my text on prodigy, most of my entities that I am trying to label are getting split up into multiple tokens due to embedded hyphens. How can i use a custom tokeniser in prodigy (ner,train), or do I have to tokenise my text in spacy and then import into prodigy?

thanks in advance  
Tushar

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 5, 2019, 8:04pm UTC](https://support.prodi.gy/t/how-to-tell-spacy-not-to-split-any-intra-hyphen-words/1456/7 "2019-06-05T20:04:23Z")

</div>

> [@tsaksena](#):
>
> How can i use a custom tokeniser in prodigy (ner,train), or do I have to tokenise my text in spacy and then import into prodigy?

If you're modifying the tokenization rules in spaCy and then saving the `nlp` object with `nlp.to_disk`, those rules will be saved with the model and loaded back in when you load the model. So once you've created your `nlp` object with custom tokenization rules, you could call `nlp.to_disk('/path/to/model')`. In Prodigy, you could then use `/path/to/model` as the base model, instead of `en_core_web_sm` or whichever model you're using.