# Does term.to-patterns tokenise patterns?

**URL:** <https://support.prodi.gy/t/does-term-to-patterns-tokenise-patterns/1937>\
**Category:** Uncategorized\
**Tags:** enhancement, usage, terms, solved\
**Created:** [August 26, 2019, 4:58pm UTC](https://support.prodi.gy/t/does-term-to-patterns-tokenise-patterns/1937 "2019-08-26T16:58:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![atilla](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/atilla/32/927_2.png) [@atilla](https://support.prodi.gy/u/atilla)\
**Post date:** [August 26, 2019, 4:58pm UTC](https://support.prodi.gy/t/does-term-to-patterns-tokenise-patterns/1937/1 "2019-08-26T16:58:19Z")

</div>

I tried to use term.to-patterns recipe to generate patterns.jsonl file whereas it looks like the recipe does not generate tokenised patterns but rather keep strings as they are.

Example usage is:

`prodigy terms.to-patterns programming_langs patterns.jsonl --label PROG_LANG`

and sample output looks like this:

`{"label":"PROG_LANG","pattern":[{"lower":"Python 3"}]}`

Unfortunately the above pattern does not match "Python 3" in the text. I'm aware that I can manually add a pattern like this:

`{"label": "LANGS", "pattern": [{"LOWER": "python"}, {"NUM_LIKE": True, "OP": "?"}]},`

But this is not very useful in this particular case as my intention is to programmatically add many multi-token strings and generate a patterns file without worrying about different use-cases. So is there are an easy way to generate something like this using terms.to-patterns:

`{"label": "LANGS", "pattern": [{"LOWER": "python"}, {"LOWER": "3"}]},`

The reason for that is to leverage term.to-patterns recipe to automate some pattern generation process.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [August 26, 2019, 7:13pm UTC](https://support.prodi.gy/t/does-term-to-patterns-tokenise-patterns/1937/2 "2019-08-26T19:13:41Z")

</div>

At the moment it doesn't tokenize patterns, no – it also doesn't take any model, just a dataset. The main reason is that the recipe was originally developed for the `terms.teach` recipe, which iterates over the model's vocab and vectors. So we can assume that they're single tokens.

If you do want to create your patterns programmatically, you could just write a little script that does this:

```python
nlp = spacy.blank("en") # or whichever language
terms = ["Python 3", "Python 2"]
patterns = []
for doc in nlp.pipe(terms):
    pattern = [{"LOWER": token.lower_} for token in doc]
    patterns.append(pattern)

```

---

<div class="post-metadata">

**Author:** ![atilla](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/atilla/32/927_2.png) [@atilla](https://support.prodi.gy/u/atilla)\
**Post date:** [August 26, 2019, 8:37pm UTC](https://support.prodi.gy/t/does-term-to-patterns-tokenise-patterns/1937/3 "2019-08-26T20:37:16Z")

</div>

Thanks @ines for the clarification. I'll go with the script approach. I understand that the intention was to compliment terms.teach and it totally make sense now. However I still believe it would have been great to have a built-in recipe that do the tokenize part or a parameter to add this functionality to the current recipe. I quite like your recipe approach and ecosystem. It's so super intuitive, nice and tidy, keep up great work. Thanks