Providing an image from filesystem for ner.manual tasks

Hi, I need to display an image (from filesystem) along with each of NER tasks in the interface.

Through the docs and a little trial and error, I managed to land on this solution:

recipe.py

# After loading the model...
stream = JSONL(source)
stream = add_tokens(nlp, stream)

img_stream = ImageServer(r"path\to\images")

def mergeStreams(s1, s2):
    for r1, r2 in zip(s1, s2):
        yield {**r1, **r2}

combinedStream = mergeStreams(stream, img_stream)

return {
    "view_id": "blocks",  # Annotation interface to use
    "dataset": dataset,  # Name of dataset to save annotations
    "stream": combinedStream,  # Incoming stream of examples
    "exclude": exclude,  # List of dataset names to exclude
    "config": {  # Additional config settings, mostly for app UI
        "lang": nlp.lang,
        "labels": label,  # Selectable label options
        "blocks": [
            {"view_id":"html", "html_template": "<img src={{image}} />"},
            {"view_id":"ner_manual"},
        ],
    }
}

source.jsonl is of the format: {"text": "NER text here"}

I want to know if this is the correct way to implement this interface? Or is there a better solution that will let me specify image filenames for each task as well?

Yes, that looks good to me :+1: (You could also use "view_id": "image" instead of the HTML block here, but the result of that should be the same, so it doesn't really matter.)

Hi, I have a similar problem except that I stream tasks from a load_data function in a python file using the - source parameter and the pipe |

I thought about structuring my tasks messages this way and I would like to avoid to process and merge two different streams:

{"image": "image_url", "text": "ner text", "meta": {...}}

What should I modify from the ner.manual recipe to correctly process these tasks? I am not very familiar with streams but here a non-working custom recipe.

    # After loading the model...
    nlp = spacy.load(spacy_model)
    
    labels = label  # comma-separated list or path to text file
    if not labels:
        labels = nlp.pipe_labels.get("ner", [])
        if not labels:
            msg.fail("No --label argument set and no labels found in model", exits=1)
        msg.text(f"Using {len(labels)} labels from model: {', '.join(labels)}")
    log(f"RECIPE: Annotating with {len(labels)} labels", labels)
    
    
    # process the text stream for ner
    stream = get_stream(
        source,
        loader=loader,
        rehash=True,
        dedup=True,
        input_key="text",
        is_binary=False,
    )
    if patterns is not None:
        pattern_matcher = PatternMatcher(nlp, combine_matches=True, all_examples=True)
        pattern_matcher = pattern_matcher.from_disk(patterns)
        stream = (eg for _, eg in pattern_matcher(stream))
    # Add "tokens" key to the tasks, either with words or characters
    stream = add_tokens(nlp, stream, use_chars=highlight_chars)
    
    # How process the "image_url" stream without having to combine two streams?
    
    return {
        "view_id": "blocks",  # Annotation interface to use
        "dataset": dataset,  # Name of dataset to save annotations
        "stream": stream,  # Incoming stream of examples
        "exclude": exclude,  # List of dataset names to exclude
        "before_db": remove_tokens if highlight_chars else None,
        "config": {  # Additional config settings, mostly for app UI
            "lang": nlp.lang,
            "labels": labels,  # Selectable label options
            "exclude_by": "input",  # Exclude examples by input text
            "ner_manual_highlight_chars": highlight_chars,
            "auto_count_stream": True,
            "blocks": [
                {"view_id":"image"}
                {"view_id":"ner_manual"},
            ],
        }
    }

Thank you in advance for your help,
Cheers

When writing custom recipes, I recommend keeping this part of the docs open in a tab.

The main reason is that it lists the expectation of each annotation interface. In the case of the image interface it's required that each example has a key called image with a filepath, url or base64 encoded image.

In your case, I think you need to write a function that can fetch the correct image path based on information in your stream and add it on each dictionary in the generator. Could you share one example? How are you planning on merging the images with the text?

Note that you can also choose to do this merge operation upfront in a Jupyter notebook.

2 Likes