# Error while using ner.match for pattern matching

**URL:** <https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874>\
**Category:** Uncategorized\
**Tags:** usage, ner, solved\
**Created:** [October 12, 2018, 12:51am UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874 "2018-10-12T00:51:19Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arul](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@Arul](https://support.prodi.gy/u/Arul)\
**Post date:** [October 12, 2018, 12:51am UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/1 "2018-10-12T00:51:19Z")

</div>

Hi there  
I am trying to match a list of patterns to text. My pattern file looks like this:

{“label”: “Action”, “pattern”: [{“lower”: “RIH”}]}  
{“label”: “Equipment”, “pattern”: [{“lower”: “ps-21”}, {“lower”: “slips”}]}

My input CSV looks like this:

id,text  
65ff62f85f222e98ac292682c4f7eee8,Installed PS21 slips. RU auto pipe handling system.  
50fb94effa05495d39636894121086bc,Installed RST toolstring into lubricator.  
466dd1bf0dd8ed7661ff4bcb4cf5fde9,Installed Riser centralizer in tension deck.  
cabadddd5a88eaf89ba6f134f52fcca2,Installed Slick joint and Diverter.  
256fae0b8ee0410824d15f9d62deb3f4,“Installed Spider. Time Wind speed Wind direction Sea Wave direction Knots deg m deg 21:30 4/8 160° 1,7/2,7 350°”  
093e2f523b1b01d54a731cf725ec5b19,“Installed PS-21 slips. Continued RIH w/ 7"” liner on 5½"" DP from 487m to 3036m, filling every 5th stand. Entered 9 5/8"" liner @ 2404m without any obstructions."

I used this command:  
prodigy ner.match sample\_dataset en\_core\_web\_sm my\_csv.csv --patterns sample\_patterns.jsonl

I end up with these problems:

1. It starts with the last input sentence and tags only “PS-21 slips” for equipment. RIH is not identified even if it exists in the text. (Changed case also and checked it does not come up).
2. it finishes up with that one sentence. and i get “no tasks available”. if i delete that line of the file and run it again, it gives an error. The error message looks like this:

Traceback (most recent call last):  
File “cython\_src/prodigy/core.pyx”, line 55, in prodigy.core.Controller. **init**  
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/toolz/itertoolz.py”, line 368, in first  
return next(iter(seq))  
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 193, in \_run\_module\_as\_main  
“ **main** ”, mod\_spec)  
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 85, in \_run\_code  
exec(code, run\_globals)  
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/prodigy/ **main**.py”, line 259, in   
controller = recipe(\*args, use\_plac=True)  
File “cython\_src/prodigy/core.pyx”, line 178, in prodigy.core.recipe.recipe\_decorator.recipe\_proxy  
File “cython\_src/prodigy/core.pyx”, line 60, in prodigy.core.Controller. **init**  
ValueError: Error while validating stream: no first batch. This likely means that your stream is empty.

I get the same error when i try to use any other files. I am stuck here. Could you help me on how to proceed?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 12, 2018, 12:24pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/2 "2018-10-12T12:24:55Z")

</div>

> [@Arul](#):
>
> It starts with the last input sentence and tags only “PS-21 slips” for equipment. RIH is not identified even if it exists in the text. (Changed case also and checked it does not come up).

By changed case, did you mean you also tested it with `"lower": "rih"`? With your current patterns, it makes sense that the following will never match:

```python
{"label": "Action", "pattern": [{"lower": "RIH"}]}

```

This pattern is looking for a token whose **lowercase form** matches `"RIH"`. This will never be true, since that string is uppercase.

> [@Arul](#):
>
> it finishes up with that one sentence. and i get “no tasks available”. if i delete that line of the file and run it again, it gives an error. The error message looks like this:

Based on the data you posted and the two patterns, this would make sense: Since the first pattern can never match, and the second one (`"PS-21 slips"`) only occurs once in the data, you'll only see one example. If you delete the example containing that span, you'll get no matches at all.

That error message means that there are no examples in your stream that can be sent out for annotation. In your case, that's because `ner.match` will look for pattern matches only, and if there are none, Prodigy will raise an error because the stream is empty and there's nothing to annotate.

> [@Arul](#):
>
> I get the same error when i try to use any other files. I am stuck here. Could you help me on how to proceed?

If you want to use `ner.match`, I'd suggest writing more patterns and trying out different variations and maybe more general token descriptions – maybe using the `SHAPE` or `IS_UPPER` attribute? You know the data and domain best, so maybe you can find attributes that commonly indicate that tokens might be something you're interested in. For example, maybe you'll find that most `ACTION` entities are single tokens consisting of only capital letters, so your pattern could be `[{"IS_UPPER": rrue}]`. It's fine if your patterns produce false positives, since you'll be accepting / rejecting the suggestions anyways.

---

<div class="post-metadata">

**Author:** ![Arul](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@Arul](https://support.prodi.gy/u/Arul)\
**Post date:** [October 12, 2018, 4:54pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/3 "2018-10-12T16:54:25Z")

</div>

Yes. I had tried with “RIH” as a lower case entry too. also the {“upper”:“RIH”} pattern. basically all the ncn combinations 🙂 ) I have more patterns - will try them out too.

What i am basically doing is to do a case insensitive match for a list of terms. I am using PhraseMatcher of spaCy directly after tokenizing. I have a list of 3000 terms. I created Upper, lower, title case, capitalized versions of all the terms and using phrase match. But still terms like “VamTop” do not match because it has a capitalization inbetween the word.

I tried to see if there is a possibility in prodigy to do a case insensitive phrase match. I have a list of 3000 phrases to be matched with case-insensitive. (i created the list of patterns too) the above is just a couple of examples. Can i do the ‘real’ case insensitive match without creating all case possibilities for a single word?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 12, 2018, 5:00pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/4 "2018-10-12T17:00:46Z")

</div>

`upper` won't work, because that's not an existing token attribute. But `"lower": "rih"` should match as expected. You can also test this in the [interactive demo](https://explosion.ai/demos/matcher).

> [@Arul](#):
>
> Can i do the ‘real’ case insensitive match without creating all case possibilities for a single word?

You could do that by using the `lower` token attributes with the lowercased form of the words and the regular rule-based `Matcher`. The only thing to keep in mind is that one dictionary represents **one token** – so if your terminology list includes multi-token phrases, you should ideally use spaCy to tokenize them to make sure they'll match. For example:

```python
nlp = spacy.load('en_core_web_sm')

TERMINOLOGY_LIST = ['term', 'another term'] # etc.
patterns = []

for doc in nlp.pipe(TERMINOLOGY_LIST):
    tokens = [token.lower_ for token in doc]
    pattern = [{'LOWER': token} for token in tokens]
    patterns.append(pattern)

```

---

<div class="post-metadata">

**Author:** ![Arul](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@Arul](https://support.prodi.gy/u/Arul)\
**Post date:** [October 12, 2018, 5:09pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/5 "2018-10-12T17:09:04Z")

</div>

Yah, if you look at my previous example: this is what my multi word token looks like:  
I hope this is ok?

> [@Arul](#):
>
> {“label”: “Equipment”, “pattern”: [{“lower”: “ps-21”}, {“lower”: “slips”}]}

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 12, 2018, 5:09pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/6 "2018-10-12T17:09:47Z")

</div>

Yes, that’s perfect! 👍

---

<div class="post-metadata">

**Author:** ![Arul](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@Arul](https://support.prodi.gy/u/Arul)\
**Post date:** [October 12, 2018, 5:12pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/7 "2018-10-12T17:12:22Z")

</div>

Thank you very much.  
Your responses are real quick and very helpful. I appreciate it 🙂

---

<div class="post-metadata">

**Author:** ![Arul](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@Arul](https://support.prodi.gy/u/Arul)\
**Post date:** [October 12, 2018, 7:45pm UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/8 "2018-10-12T19:45:00Z")

</div>

Yey! This works 🧞‍♀️ . Thank you @ines.  
I have 2 questions following this:

1. is there a way tho write these matches to a jsonl file without getting to the interface? (I do not see a “–output” argument.)

2. I would like to remove the overlapping spans and keep only the longest span. For example on the interface I get “PS-21” suggested once and “PS-21 slips” once and “slips” once. I just want to keep “PS-21 slips”.

From spacy matching i checked the output jsonl file and and removed the overlapping spans with writing a function. is there something that will consider only the longest match?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 13, 2018, 10:37am UTC](https://support.prodi.gy/t/error-while-using-ner-match-for-pattern-matching/874/9 "2018-10-13T10:37:20Z")

</div>

Glad it'S working! Moving to the next thread, answer see here:

> [@ner.match to jsonl without getting into the interface](https://support.prodi.gy/t/ner-match-to-jsonl-without-getting-into-the-interface/879/2):
>
> The main idea of ner.match is to give you an interface so you or your annotators can accept and reject matches to bootstrap training sets with positive and negative examples, and to allow creating training data from patterns that produce false positives and to explore patterns interactively. If you only want to create matches based on patterns, you could just use spaCy's Matcher directly and save the matches as JSONL? If you do want to annotate with Prodigy but with custom match logic (or any o…