# ner-manual does not use custom tokens

**URL:** <https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473>\
**Category:** Uncategorized\
**Tags:** ner, done, solved\
**Created:** [January 28, 2020, 2:48pm UTC](https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473 "2020-01-28T14:48:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![edu](https://avatars.discourse-cdn.com/v4/letter/e/cdc98d/32.png) [@edu](https://support.prodi.gy/u/edu)\
**Post date:** [January 28, 2020, 2:48pm UTC](https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473/1 "2020-01-28T14:48:45Z")

</div>

Hello! Let's consider the following toy problem:

I have a jsonl data file with 1 line:

```python
{ "text": "a:b c", "tokens": [{"text": "a:b", "start": 0, "end": 3, "id": 0}, {"text": "c", "start": 4, "end": 5, "id": 1}] }

```

and I'm trying to use ner.manual to label it:

```python
prodigy ner.manual testing_2 blank:en blah.jsonl --label FOO,BAR

```

I was hoping it would use my custom tokenization of `["a:b" "c"]`, instead, it still does the standard Spacy tokenization `["a", ":", "b", "c"]`.

Is this expected behaviour? Is there a way to force Prodigy to use my custom tokenization?

Thanks!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [January 28, 2020, 5:52pm UTC](https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473/2 "2020-01-28T17:52:10Z")

</div>

Are you using the latest version? Prodigy's built-in token splitting should accept pre-defined tokenization (there used to be a problem but it should be fixed now) 🤔

In the meantime, you could also just use the `mark` recipe if you already have pre-tokenized data. The recipe will stream in whatever is in the data and render it with a given interface. So your command could look like this:

```bash
prodigy mark your_dataset ./your_data.jsonl --view-id ner_manual --label FOO,BAR

```

---

<div class="post-metadata">

**Author:** ![edu](https://avatars.discourse-cdn.com/v4/letter/e/cdc98d/32.png) [@edu](https://support.prodi.gy/u/edu)\
**Post date:** [January 28, 2020, 11:46pm UTC](https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473/3 "2020-01-28T23:46:45Z")

</div>

I'm on v1.9.5 - would you be able to try out my toy example above and see if it works for you?

I just tried using `mark`, but it seems to just get stuck at "Loading..."

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [January 29, 2020, 5:30pm UTC](https://support.prodi.gy/t/ner-manual-does-not-use-custom-tokens/2473/4 "2020-01-29T17:30:26Z")

</div>

> [@edu](#):
>
> I'm on v1.9.5 - would you be able to try out my toy example above and see if it works for you?

I just tried it and it seems to work fine for me – here's what I see:

 ![Screenshot 2020-01-29 at 18.28.34](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/9/98bb4eada6d676a75c997cac6f91cee3f01babb1.png)

You can set `PRODIGY_LOGGING=basic` to see what it does behind the scenes. If you don't see an error and don't see "No tasks available", and it just gets stuck at loading, double-check that you're passing in the input file correctly? If no input file is provided, Prodigy will try to read from standard input.