# Guidance on how to add tokenizer rule

**URL:** <https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666>\
**Category:** Uncategorized\
**Tags:** spacy, solved\
**Created:** [July 1, 2018, 7:31am UTC](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666 "2018-07-01T07:31:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sooheon](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sooheon/32/351_2.png) [@sooheon](https://support.prodi.gy/u/sooheon)\
**Post date:** [July 1, 2018, 7:31am UTC](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666/1 "2018-07-01T07:31:00Z")

</div>

In my corpus, I’m seeing a lot of tokens for MONEY which look like the following: ![30](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/1X/7693a15ab63a9366c85ed0dd54f25edfeee78950.png)

I’d like to split the text into `['$', '8', '/hour']` and tokenize only the first two as MONEY, and train a new token class for the `/hour`. What is the best way to tell spaCy tokenizer to split ‘8/hour’, when the suffix could be ‘/day’, ‘/week’, ‘/job’, etc.?

---

<div class="post-metadata">

**Author:** ![sooheon](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sooheon/32/351_2.png) [@sooheon](https://support.prodi.gy/u/sooheon)\
**Post date:** [July 2, 2018, 7:13am UTC](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666/2 "2018-07-02T07:13:02Z")

</div>

If anyone has the same question, here’s how I solved it:

spaCy has an argument to the Tokenizer() constructor ([`infix_finditer`](https://spacy.io/api/tokenizer#init)) which can take a regex to split all tokens on internally.

```python
>>> infix_re = re.compile(r'/')
>>> tokenizer = Tokenizer(nlp.vocab, infix_finditer=infix_re.finditer)
>>> [w for w in tokenizer('$8/hour')]
[$8, /, hour]

```

edit:  
updated to not split dates in YYYY/MM/DD format:

```python
>>> suffix_re = spacy.util.compile_suffix_regex(tuple(list(nlp.Defaults.suffixes) + [r'/[A-z]+']))
>>> tokenizer = Tokenizer(nlp.vocab, suffix_search=suffix_re.search)
>>> [w for w in tokenizer('$80/wk starting 2018/06/25')]
[$80, /wk, starting, 2018/06/25]

```

---

<div class="post-metadata">

**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\
**Post date:** [July 2, 2018, 11:58am UTC](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666/3 "2018-07-02T11:58:18Z")

</div>

Thanks for updating!

Just a quick note on your solution though: If you create the tokenizer this way, you won’t be loading the normal language-specific rules, so your tokenization will be worse overall.

What you probably want to do is modify the `English.infixes` class-attribute before loading the model. During construction, the class attribute is read to create the infixes regex when creating the tokenizer.

The `English.infixes` attribute is a tuple of strings, which are built into regular expressions. So something like this should work:

```python
from spacy.lang.en import English

English.Defaults.infixes = English.Defaults.infixes + [r'/[A-z]+]

```

More details here: [https://spacy.io/usage/linguistic-features#section-tokenization](https://spacy.io/usage/linguistic-features#section-tokenization)

---

<div class="post-metadata">

**Author:** ![sooheon](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sooheon/32/351_2.png) [@sooheon](https://support.prodi.gy/u/sooheon)\
**Post date:** [July 3, 2018, 8:36am UTC](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666/4 "2018-07-03T08:36:11Z")

</div>

Thanks, this is much simplified (although it seems to be `English.Defaults.infixes` rather than `English.infixes`)

Again for posterity:

```python
import spacy
from spacy.lang.en import English

English.Defaults.infixes = English.Defaults.infixes + tuple([r'/(?=[A-z]+)'])

nlp = spacy.load('en_core_web_lg')

nlp.to_disk('./models/en_lg_custom')

```