# character-based Matching

**URL:** <https://support.prodi.gy/t/character-based-matching/3213>\
**Category:** Uncategorized\
**Tags:** enhancement, usage, ner, spacy\
**Created:** [July 21, 2020, 2:13am UTC](https://support.prodi.gy/t/character-based-matching/3213 "2020-07-21T02:13:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Infinity](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/infinity/32/1542_2.png) [@Infinity](https://support.prodi.gy/u/Infinity)\
**Post date:** [July 21, 2020, 2:13am UTC](https://support.prodi.gy/t/character-based-matching/3213/1 "2020-07-21T02:13:22Z")

</div>

I use prodigy to annotate Chinese sentences. Does Token Matcher suppose character-based Matching?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [July 21, 2020, 9:32am UTC](https://support.prodi.gy/t/character-based-matching/3213/2 "2020-07-21T09:32:33Z")

</div>

Hi! If you're using `--patterns` (powered by spaCy's `Matcher` under the hood), the spans here will refer to tokens produced by the model/tokenizer. So you couldn't match substrings.

If you're annotating with `ner.manual` and `--highlight-chars`, you can select individual characters and pre-defined spans don't have to map to tokens. So you can add `"spans"` to your incoming examples that refer to characters – you'd just have to implement the matching yourself, e.g. using regex. Each span you add needs a `start` and `end` offset and a `label` – you can see an example of the format [here](https://prodi.gy/docs/api-interfaces#ner_manual).

In your custom recipe, you could then do something like this:

```python
# Your custom regex etc. that adds matched character spans
stream = add_your_character_spans(stream)
# Add "tokens" key to the tasks, either with words or characters
stream = add_tokens(nlp, stream, use_chars=highlight_chars)

```

---

<div class="post-metadata">

**Author:** ![Infinity](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/infinity/32/1542_2.png) [@Infinity](https://support.prodi.gy/u/Infinity)\
**Post date:** [July 28, 2020, 7:45am UTC](https://support.prodi.gy/t/character-based-matching/3213/3 "2020-07-28T07:45:53Z")

</div>

According to your guiding, I solved the problem through my custom recipe, by self-carving the add\_your\_character\_spans function. Thanks a lot for your help.

Now, I have a further question. Can I customize spaCy's Tokenizer class, so that I can use --patterns(with Token Matcher) to get the same result?

If so, It's convenient for the situation that one sentence includes Chinese and English at the same time. Then we can tokenize English by word-based, and Chinese by character-based within a sentence.

---

<div class="post-metadata">

**Author:** ![adriane](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@adriane](https://support.prodi.gy/u/adriane)\
**Post date:** [July 28, 2020, 11:37am UTC](https://support.prodi.gy/t/character-based-matching/3213/4 "2020-07-28T11:37:21Z")

</div>

As long as as you just want to split all Chinese characters (rather than any kind of real word segmentation), I think you can modify the English tokenizer settings to do this.

This will treat any characters in our predefined CJK character class as a single token by finding them as prefixes, suffixes, and infixes:

```python
import spacy
from spacy.lang.char_classes import _cjk

nlp = spacy.blank("en")
# alternative: some English model with existing components/vectors
# nlp = spacy.load("en_core_web_sm")

prefixes = list(nlp.Defaults.prefixes) + [r"[{c}]".format(c=_cjk)]
suffixes = list(nlp.Defaults.suffixes) + [r"[{c}]".format(c=_cjk)]
infixes = list(nlp.Defaults.infixes) + [r"[{c}]".format(c=_cjk)]
nlp.tokenizer.prefix_search = spacy.util.compile_prefix_regex(prefixes).search
nlp.tokenizer.suffix_search = spacy.util.compile_suffix_regex(suffixes).search
nlp.tokenizer.infix_finditer = spacy.util.compile_infix_regex(infixes).finditer

nlp.to_disk("/path/to/en_zhchar_model")

```

Then use `/path/to/en_zhchar_model` as the base model when calling prodigy. You can also provide a different character range here if the `_cjk` one isn't quite what you want.

And for the tokenizer to handle the English tokens like it would in plain English text (mainly in terms of the exceptions), they need to be separated from the Chinese characters by whitespace. If you have something like `can't` without separating whitespace it might not get split into `ca n't`, depending on the context.

To do this more consistently, you would want to identify the spans in each writing system and tokenize them separately. This is totally possible if you'd like to do it, but would require a bit more effort than just modifying the tokenizer as above.