UnicodeDecodeError:

[koaning (Vincent D Warmerdam)]

I'm trying to repeat your steps. I started by creating a folder called issue-6037 and moving your files in there with the names news_headlines.jsonl and news_headlines_small.jsonl. From there I started annotating via this recipe:

python -m prodigy ner.manual ner_news_headlines blank:en news_headlines.jsonl --label PERSON,ORG,PRODUCT,LOCATION

This is what that interface looks like:

I annotated six examples and I hit the save button. Next, I ran your terms recipe.

python -m prodigy terms.to-patterns ner_news_headlines --label PERSON,ORG,PRODUCT,LOCATION --spacy-model blank:en > news_pattern.jsonl

This is what my news_pattern.sjonl file looks like:

{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"uber"},{"lower":"\u2019s"},{"lower":"lesson"},{"lower":":"},{"lower":"silicon"},{"lower":"valley"},{"lower":"\u2019s"},{"lower":"start"},{"lower":"-"},{"lower":"up"},{"lower":"machine"},{"lower":"needs"},{"lower":"fixing"}]}
{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"pearl"},{"lower":"automation"},{"lower":","},{"lower":"founded"},{"lower":"by"},{"lower":"apple"},{"lower":"veterans"},{"lower":","},{"lower":"shuts"},{"lower":"down"}]}
{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"how"},{"lower":"silicon"},{"lower":"valley"},{"lower":"pushed"},{"lower":"coding"},{"lower":"into"},{"lower":"american"},{"lower":"classrooms"}]}
{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"women"},{"lower":"in"},{"lower":"tech"},{"lower":"speak"},{"lower":"frankly"},{"lower":"on"},{"lower":"culture"},{"lower":"of"},{"lower":"harassment"}]}
{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"silicon"},{"lower":"valley"},{"lower":"investors"},{"lower":"flexed"},{"lower":"their"},{"lower":"muscles"},{"lower":"in"},{"lower":"uber"},{"lower":"fight"}]}
{"label":"PERSON,ORG,PRODUCT,LOCATION","pattern":[{"lower":"uber"},{"lower":"is"},{"lower":"a"},{"lower":"creature"},{"lower":"of"},{"lower":"an"},{"lower":"industry"},{"lower":"struggling"},{"lower":"to"},{"lower":"grow"},{"lower":"up"}]}

And I think, looking at this file, that the recipe isn't doing what you had hoped it did. Notice how each row has "PERSON,ORG,PRODUCT,LOCATION" as a label? While this isn't the error message that you're experiencing, I'm assuming that it's related. The terms.to-patterns recipe is designed to be used together with the terms.teach recipe, not the ner.manual one.

This Youtube video helps explain how it's meant to be used.

Terms and Seeds - Prodigy Shorts

Custom Recipe

That said, nothing is stopping you from writing a custom script that can turn your previous annotations as terms. Here's a small script that does that.

import srsly 
import prodigy 
from prodigy.components.db import connect

@prodigy.recipe(
    "terms.from-ner",
    ner_dataset=("Dataset loader NER annotations from", "positional", None, str),
    file_out=("File to write patterns into", "positional", None, str)
)
def custom_recipe(ner_dataset: str, file_out: str):
    # Connect to Prodigy database
    db = connect()
    # Load in annotated examples 
    annotated = db.get_dataset(ner_dataset)
    # Loop over examples
    pattern_set = set()
    for example in annotated:
        for span in example.get("spans", []):
            pattern_str = example['text'][span['start']: span['end']]
            # Store into tuple, because sets like that
            tup = (pattern_str, span['label'])
            pattern_set.add(tup)
    patterns = [{"pattern": p, "label": l} for p, l in pattern_set]
    srsly.write_jsonl(file_out, patterns)

If you're curious how to work with patterns and custom code, you may appreciate the guide in the docs here. When I run this locally via:

python -m prodigy terms.from-ner ner_news_headlines patterns.jsonl -F recipe.py

Then the file patterns.jsonl contains this:

{"pattern":"Apple","label":"ORG"}
{"pattern":"Silicon Valley","label":"LOCATION"}
{"pattern":"Uber","label":"ORG"}
{"pattern":"Pearl Automation","label":"ORG"}

I can now use these patterns to do ner.manual.

python -m prodigy ner.manual news_data blank:en news_headlines.jsonl --label PERSON,ORG,PRODUCT,LOCATION --patterns patterns.jsonl

Here's what it looks like:

Note how some entities are pre-labelled but also note that there's now PATTERN metadata in there. This tells you which patterns got activated. I hope this helps!