Welcome to the forum @PeterFig 
It is actually not necessary to use custom javascript to fulfill the requirements you described. The components of your UI i.e. text for NER and image are valid inputs for Prodigy built-in components that can be combined via blocks interface.
Your blocks would compose of ner_manual and image components. And for that to work you need to make sure that there are required fields on the input task: "text" and "tokens" for NER and "image" for image.
Minimally your task should contain the "text" field and "image" field which could be a path to the file on your local disk (there are many other ways you can input images to Prodigy as you've already noticed). In this example, for simplicity I'll use the local paths.
Here's how your custom recipe could look like:
from typing import List
import prodigy
from prodigy.components.preprocess import add_tokens, fetch_media
from prodigy.components.stream import get_stream
from prodigy.core import Arg
from prodigy.types import StreamType, TaskType
from prodigy.util import get_pipe_labels, set_hashes
from spacy import Language
def rehash(stream: StreamType) -> StreamType:
for eg in stream:
yield set_hashes(eg)
@prodigy.recipe(
"ner.image",
dataset=Arg(help="Dataset to save annotations to"),
nlp=Arg(help="Loadable spaCy pipeline with an entity recognizer"),
source=Arg(help="Data to annotate (file path or '-' to read from standard input)"),
label=Arg(
"--label",
"-l",
help="Comma-separated label(s) to annotate or text file with one label per line",
),
)
def ner_image(dataset: str, nlp: Language, source: str, label: List[str]):
labels = get_pipe_labels(label, nlp.pipe_labels.get("ner", []))
stream = get_stream(source) # reading the input file
stream.apply(add_tokens, nlp=nlp, stream=stream) # adding tokens needed for NER
stream.apply(fetch_media, stream) # encoding local paths into base64-encoded data URIs
stream.apply(rehash, stream) # rehashing
def before_db(examples: List[TaskType]) -> List[TaskType]:
# Remove all data URIs before storing example in the database
for eg in examples:
if eg["image"].startswith("data:"):
eg["image"] = eg.get("path")
return examples
blocks = [{"view_id": "ner_manual"}, {"view_id": "image"}]
return {
"view_id": "blocks",
"dataset": dataset, # Name of dataset to save annotations
"stream": stream,
"before_db": before_db,
"config": {
"blocks": blocks,
"labels": labels,
"exclude_by": "input",
},
}
We are encoding the images because most browser won't permit local file paths for security reasons.
As you can see above, before saving the example to the DB we are stripping these big image encoding strings to avoid bloating the database and we are only keeping the path for the reference (this field is produced by the fetch_media helper. This is done in the before_db callback.
This recipe should result in the UI like this:
If you need to recreate more bult-in ner.manual features such as patterns matching feel free to see the source code for the recipe available in your Prodigy installation folder which you can revise by running prodigy stats and checking the Location there.
Hopefully that's enough to get you started?
You can also get fancier and play with the layout etc. but that will require more custom CSS work. Let us know if you need any further help on it!
