Hi Support Team,
I want to create a user interface for multi-label classification of images. There can be zero, one or several labels for an image. In total, there are 14 labels.
Is there a recipe for this use case? Or if you can help me with this that will be much appreciated.
Thanks in advance
Kind regards,
Bilal
I was able to create a recipe from Prodigy computer vision recipes for the multi-label image annotation using the code below:
@prodigy.recipe(("labels-review-recipe1"))
def data_review_recipe(dataset, images_file):def get_stream(): # Load the directory of images and add options to each task stream = JSONL(images_file) stream = fetch_media(stream, ["image"], skip=True) for eg in stream: eg["options"] = options yield eg # def before_db(examples: List[TaskType]) -> List[TaskType]: # # Remove all data URIs before storing example in the database # for eg in examples: # if eg["image"].startswith("data:"): # eg["image"] = eg.get("path") # return examples return { "view_id": "choice", # "before_db": before_db if remove_base64 else None, "config": { "choice_style": "multiple"}, "dataset": dataset, "stream": get_stream(), }
I use the following command to start the server:
prodigy labels-review-recipe1 labels_review_v1 data/input.json -F fix_labels_prodigy.py
with the data/input.json containing the following information:
{"image": "data/00303.jpg", "labels": "Class A, Class C, Class Z"}
{"image": "data/00495.jpg", "labels": "Class B, Class C"}
Can I check the choices item with default values given in the labels field of json file?
EDIT: Just for clarification on the use case: my goal is to review the labels created by a model (trained in fastai library) in Prodigy.
You should be able to pre-select via:
Thanks for sharing this. Now, I can see the checkboxes pre-selected with options. However, when I deselect an option, it still come as part of the accept key as shown in the highlighted text.
{
"image": "data/train_images_tiny_10_frames_crop/clip_024332/00725.jpg",
"labels": "needle_driver",
"path": "data/train_images_tiny_10_frames_crop/clip_024332/00725.jpg",
"accept": [
"n",
"e",
"d",
"l",
"_",
"r",
"i",
"v",
"monopolar_curved_scissor"
],
"mod_labels": "needle_driver",
"options": [
{
"id": "needle_driver",
"text": "needle_driver"
},
{
"id": "monopolar_curved_scissor",
"text": "monopolar_curved_scissor"
},
{
"id": "force_bipolar",
"text": "force_bipolar"
},
{
"id": "clip_applier",
"text": "clip_applier"
},
{
"id": "tip_up_fenestrated_grasper",
"text": "tip_up_fenestrated_grasper"
},
{
"id": "cadiere_forceps",
"text": "cadiere_forceps"
},
{
"id": "bipolar_forceps",
"text": "bipolar_forceps"
},
{
"id": "vessel_sealer",
"text": "vessel_sealer"
},
{
"id": "suction_irrigator",
"text": "suction_irrigator"
},
{
"id": "bipolar_dissector",
"text": "bipolar_dissector"
},
{
"id": "prograsp_forceps",
"text": "prograsp_forceps"
},
{
"id": "stapler",
"text": "stapler"
},
{
"id": "permanent_cautery_hook_spatula",
"text": "permanent_cautery_hook_spatula"
},
{
"id": "grasping_retractor",
"text": "grasping_retractor"
},
{
"id": "nan",
"text": "nan"
},
{
"id": "blank",
"text": "blank"
}
],
"_input_hash": 1917670788,
"_task_hash": -1067387639,
"_view_id": "blocks",
"config": {
"choice_style": "multiple"
},
"answer": "accept",
"_timestamp": 1670096634
}
Any idea of how to fix it?
I've edited your response so that the JSON renders a bit more nicely.
There are some strange things that I see in your options. There seems to be a needle_driver ID, which appears in your accept as seperate characters. I also see a nan item appear as well as a blank on. This suggests to me that something is going awry in your options variable in your recipe. I'm willing to try and reproduce this locally, but before doing that ... could you confirm nothing strange is happening there?
Could you also share how the options variable is created? I'm referring to the variable that you use here:
for eg in stream:
eg["options"] = options
yield eg
I am populating the accept key with model predictions to preselect checkboxes rendered for options. This is how it is explained in the link you shared. I didn't use an integer for a label in the id of the options and used the label instead to avoid translation code.
Both nan and blank are valid values in the options. The nan is used as a label for images with no surgical instrument and blank for empty images (black pixels showing nothing).
Please find the specification of the options variable below:
tools = [
"needle_driver",
"monopolar_curved_scissor",
"force_bipolar",
"clip_applier",
"tip_up_fenestrated_grasper",
"cadiere_forceps",
"bipolar_forceps",
"vessel_sealer",
"suction_irrigator",
"bipolar_dissector",
"prograsp_forceps",
"stapler",
"permanent_cautery_hook_spatula",
"grasping_retractor",
"nan",
"blank"
]
options = [{"id": t, "text": t} for t in tools]
Besides the definition of the get_steam() is as follows:
def get_stream():
stream=JSONL(images_file)
stream=fetch_media(stream, ["image"], skip=True)
for eg in stream:
labels=predict(eg["image"])
eg["accept"]=labels
eg["mod_labels"]=labels
eg["options"]=options
yield eg
I hope you shall be able to spot the problem.
Thanks for the support.
I have downloaded some cat pictures from this site and moved them into a folder called images. I also have the following examples.jsonl file.
{"image": "images/300.jpg"}
{"image": "images/301.jpg"}
{"image": "images/302.jpg"}
I also wrote the following custom recipe.
import prodigy
from prodigy.components.loaders import JSONL
from prodigy.components.preprocess import fetch_media
tools = [
"needle_driver",
"monopolar_curved_scissor",
"force_bipolar",
"clip_applier",
"tip_up_fenestrated_grasper",
"cadiere_forceps",
"bipolar_forceps",
"vessel_sealer",
"suction_irrigator",
"bipolar_dissector",
"prograsp_forceps",
"stapler",
"permanent_cautery_hook_spatula",
"grasping_retractor",
"nan",
"blank"
]
def add_options(stream):
options = [{"id": t, "text": t} for t in tools]
for ex in stream:
ex['options'] = options
ex['accept'] = ["force_bipolar", ]
yield ex
@prodigy.recipe(
"custom",
dataset=("Dataset to save answers to", "positional", None, str),
examples=("Examples to load from disk", "positional", None, str),
)
def custom(dataset, examples):
stream = JSONL(examples)
stream = fetch_media(stream, ["image"], skip=True)
return {
"dataset": dataset,
"stream": add_options(stream),
"view_id": "blocks",
"config":{
"blocks":[
{"view_id": "choice"}
],
"choice_style": "multiple"
}
}
Note! Pay attention to the line that reads
ex['accept'] = ["force_bipolar", ]. This is relevant for the screenshot in a bit.
When I run this, via:
python -m prodigy custom demodemo examples.jsonl -F recipe.py
Then I get an interface that looks like this:
I think this does what you'd want but I'm not 100% sure what's going wrong exactly in your code. If I had to guess, I'd imagine that this part:
labels=predict(eg["image"])
eg["accept"]=labels
is causing the error. Are you 100% sure that labels is a list and not a string?
You are right. It renders fine for me too as you said but until you start changing and saving the values. Try to change the value to needle_driver, save it, and then check the saved JSON from the prodigy database.
This is what db-out gives me.
{
"image": "data:image/jpeg;base64,...",
"path": "images/300.jpg",
"options": [
{
"id": "needle_driver",
"text": "needle_driver"
},
{
"id": "monopolar_curved_scissor",
"text": "monopolar_curved_scissor"
},
{
"id": "force_bipolar",
"text": "force_bipolar"
},
{
"id": "clip_applier",
"text": "clip_applier"
},
{
"id": "tip_up_fenestrated_grasper",
"text": "tip_up_fenestrated_grasper"
},
{
"id": "cadiere_forceps",
"text": "cadiere_forceps"
},
{
"id": "bipolar_forceps",
"text": "bipolar_forceps"
},
{
"id": "vessel_sealer",
"text": "vessel_sealer"
},
{
"id": "suction_irrigator",
"text": "suction_irrigator"
},
{
"id": "bipolar_dissector",
"text": "bipolar_dissector"
},
{
"id": "prograsp_forceps",
"text": "prograsp_forceps"
},
{
"id": "stapler",
"text": "stapler"
},
{
"id": "permanent_cautery_hook_spatula",
"text": "permanent_cautery_hook_spatula"
},
{
"id": "grasping_retractor",
"text": "grasping_retractor"
},
{
"id": "nan",
"text": "nan"
},
{
"id": "blank",
"text": "blank"
}
],
"accept": [
"force_bipolar",
"needle_driver"
],
"_input_hash": -1817032579,
"_task_hash": 1476352692,
"_view_id": "blocks",
"config": {
"choice_style": "multiple"
},
"answer": "accept",
"_timestamp": 1670846561
}
This is what we'd expect right? The only thing I don't have in my recipe is your predict code.