# Custom Tokenizer help

**URL:** <https://support.prodi.gy/t/custom-tokenizer-help/6203>\
**Category:** Uncategorized\
**Tags:** ner, spacy\
**Created:** [December 22, 2022, 4:57pm UTC](https://support.prodi.gy/t/custom-tokenizer-help/6203 "2022-12-22T16:57:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvaro.marlo](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@alvaro.marlo](https://support.prodi.gy/u/alvaro.marlo)\
**Post date:** [December 22, 2022, 4:57pm UTC](https://support.prodi.gy/t/custom-tokenizer-help/6203/1 "2022-12-22T16:57:38Z")

</div>

Hi,

I am creating a custom tokenizer in Spanish to execute the `add_tokens` function in an annotations file with spans completed splitting the tokens as I want. I have already updated the tokenizer of the `blank:es` model to isolate the punctuation marks in different tokens. Examples:

- `Secretario/a` -\> `["Secretario", "/", "a"]`
- `Vitoria-Gasteiz` -\> `["Vitoria", "-", "Gasteiz"]`
- `Hola.` -\> `["Hola", "."]`

I have done this including as prefixes, infixes and suffixes all the symbols available with Regex:

```python
prefixes = nlp.Defaults.prefixes + [r'''[!"\#$%&'()*+,\-./:;<=>?@\[\\\]^_‘{|}~]''']
prefix_regex = spacy.util.compile_prefix_regex(prefixes)
nlp.tokenizer.prefix_search = prefix_regex.search

infixes = nlp.Defaults.infixes + [r'''[!"\#$%&'()*+,\-./:;<=>?@\[\\\]^_‘{|}~]''']
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

suffixes = nlp.Defaults.suffixes + [r'''[!"\#$%&'()*+,\-./:;<=>?@\[\\\]^_‘{|}~]''']
suffix_regex = spacy.util.compile_suffix_regex(suffixes)
nlp.tokenizer.suffix_search = suffix_regex.search

```

**NOTE:** I'm not sure if with this Regex syntax it performs exactly as I want.

But in cases as this text fragment:  
`Secretario/a.`  
It splits the tokens in:  
`["Secretario", "/", "a."]`  
When I want:  
`["Secretario", "/", "a", "."]`  
And with this problem I found future token missmatching errors.

I am pretty sure that this is happening because it interprets `.a` as part of an acronym. So, how can I change that problem at the Tokenizer and obtain the results I want?

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [December 23, 2022, 1:58pm UTC](https://support.prodi.gy/t/custom-tokenizer-help/6203/2 "2022-12-23T13:58:07Z")

</div>

Thanks for your question and thanks for reposting your question on the [spaCy GitHub discussion](https://github.com/explosion/spaCy/discussions). Generally spaCy specific questions are better for that forum as Prodigy Support is best for Prodigy-specific questions.

I'll repost the [answer](https://github.com/explosion/spaCy/discussions/12018#discussioncomment-4481698) from @polm below for anyone interested.

* * *

This is a bit tricky because what you want to do here is remove an existing special case.

First, you can figure out what's going on using the `explain` method:

```python
nlp.tokenizer.explain("Secretario/a.")
# => [('TOKEN', 'Secretario'), ('INFIX', '/'), ('SPECIAL-1', 'a.')]

```

`SPECIAL` refers to a tokenizer exception in this case.

You can remove a tokenizer exception by modifying `tokenizer.rules`, like this:

```python
rules = nlp.tokenizer.rules
del rules["a."]
nlp.tokenizer.rules = rules

```

In your case, it sounds like you might want to do this for every lowercase letter. That should be easy to do in a for loop, but if you want to remove more exceptions then it might be better to make your own language subclass where you control the exceptions directly. See [the guide to language subclassing](https://spacy.io/usage/linguistic-features#language-subclass) or the [Spanish definition](https://github.com/explosion/spaCy/blob/eef3d950b4266ab9545143de8070456ce7967950/spacy/lang/es/ __init__.py#L12) for an example of what that looks like.