# Help with tokenization numbers with units of measure

**URL:** <https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy\
**Created:** [July 30, 2018, 9:28pm UTC](https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718 "2018-07-30T21:28:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![drem](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/drem/32/383_2.png) [@drem](https://support.prodi.gy/u/drem)\
**Post date:** [July 30, 2018, 9:28pm UTC](https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718/1 "2018-07-30T21:28:09Z")

</div>

I have measurement values that I’ll be “learning” from. I’m labelling `Number of units` and `unit of measure`.

`5.1V dc` and `5.1 V dc` should be labelled as:

- `Number of units` = `5.1`
- `Unit of measure` = `V dc`

`0.1mV ac` and `0.1 mV ac` should be labelled as:

- `Number of units` = `0.1`
- `Unit of measure` = `mV ac`

As you can see, the `Number of units` may or may not have whitespace between itself and the `Units of measure`.

Could you please point me in the right direction?

Would I need a customer tokenizer to split them up? If so, any thoughts how best I would do that?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [July 31, 2018, 4:57pm UTC](https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718/2 "2018-07-31T16:57:12Z")

</div>

Hi! I just ran one of your examples through the default English tokenizer and you’re right, `"5.1V dc"`  
is currently split as `['5.1V', 'dc']`. So it probably makes sense to [customise the tokenizer](https://spacy.io/usage/linguistic-features#native-tokenizers) and build your own set of rules specific to your domain, starting with suffix rules that split off alphanumeric characters following numbers. [This comment](https://support.prodi.gy/t/guidance-on-how-to-add-tokenizer-rule/666/2) shows how you can extend the default expressions with your own, and save out a custom model.

The tokenizer rules are serialized with the model, and Prodigy supports loading in spaCy models from packages, links and directories – so you can just pass in the path of your model when you run a recipe:

```bash
prodigy ner.teach your_dataset /path/to/custom-model ...

```

Btw, you might also want to experiment with different combinations of rule-based and statistical approaches. For example, there are only so many units, right? If so, you could use the [matcher](https://spacy.io/usage/linguistic-features#section-rule-based-matching) to find them in your text, and then check whether the previous token has a number-like value. [This thread](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651) and [this thread](https://support.prodi.gy/t/ner-or-phrasematcher/686) both have some more details and examples of this approach.

Alternatively, spaCy’s pre-trained models already have a category `ORDINAL`, which you could try and fine-tune on your data. This might be quicker and more efficient, because you don’t need to teach the model all of this from scratch. The solution you choose ultimately depends on your data, and it’s difficult to predict which one works best. You might want to try out a few approaches and test them on the same evaluation set to find the one that produces the best results. (Prodigy can also help with that, since it lets you do quick binary evaluations.)

---

<div class="post-metadata">

**Author:** ![drem](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/drem/32/383_2.png) [@drem](https://support.prodi.gy/u/drem)\
**Post date:** [August 6, 2018, 2:44am UTC](https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718/3 "2018-08-06T02:44:20Z")

</div>

Hi ines,

Thanks for the reply. I was able to get the tokenization working with your guidance. I saved it as a custom model and then used prodigy to start some labeling. However, prodigy wasnt using the tokenizer that I created. I’m sure I’m missing something obvious.

Here’s what I did:

The code:

```python
import spacy
from spacy.lang.en import English
import re
from spacy.tokenizer import Tokenizer

nlp = spacy.load('en')
prefix_re = re.compile(r'''^[±\-\+0-9.,]+[0-9]+''')
infix_re = re.compile(r'''[/]''')

def my_tokenizer_pri(nlp):
  return Tokenizer(nlp.vocab,
    {},
    prefix_search=prefix_re.search,
    suffix_search=nlp.tokenizer.suffix_search,
    infix_finditer=infix_re.finditer,
    token_match=nlp.tokenizer.token_match)

nlp.tokenizer = my_tokenizer_pri(nlp)

nlp.to_disk('./models/en_custom')

print([w.text for w in nlp(u'0.1mV/km ac')])
print([w.text for w in nlp(u'3 → 12V')])

```

Output:

```bash
['0.1', 'mV', '/', 'km', 'ac']
['3', '→', '12', 'V']

```

I can also confirm the saved model does load and function as expected by doing this:

```python
nlp = spacy.load('./models/en_custom')
print([w.text for w in nlp(u'0.1mV/km ac')])
print([w.text for w in nlp(u'3 → 12V')])

```

Output:

```bash
['0.1', 'mV', '/', 'km', 'ac']
['3', '→', '12', 'V']

```

Then I ran the following:

```bash
prodigy ner.manual volts ./models/en_custom volts.jsonl --label QUANTITY,UOM

```

But… I was presented with this:  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/1X/a8566e630c116ce3ebe59efd9c86cf1423156e76.png)

My tokenizer (from within the code above) would have split this as: `1`, `mV`, `ac`.

What am I missing?

Thanks

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [August 6, 2018, 8:57am UTC](https://support.prodi.gy/t/help-with-tokenization-numbers-with-units-of-measure/718/4 "2018-08-06T08:57:25Z")

</div>

Thanks for the update – this looks really good so far!

And I’m confused about this example, too… Especially since Prodigy isn’t really doing anything magical here – it just uses the provided model to process and tokenize the text. So Prodigy’s tokenization should always match what you’re seeing if you’re using the model with spaCy directly.

Just as a sanity check, did you check that this exact string is tokenized as expected? Maybe there’s some edge case that your custom tokenizer is missing?