multilabel image classification on a bounding box

[ines (Ines Montani)]

Hi! A similar(ish) question came up in this thread the other day, and for cases like this, we'd typically recommend making two passes over the annotations for the two objectives: getting the bounding box right and categorising the content labelled by the bounding box:

@ines

If you have pre-defined labels for the second dimension, you could make two passes over the data: first, annotate the objects (person etc.), and review the annotations/data quality if needed. Next, stream in each bounding box and the second layer of labels, e.g. as multiple choice options (choice UI). This also makes it easier to validate and evaluate the two layers separately, and fix mistakes early (because you know that if the bounding box is wrong, any subsequent annotation will most likely be wrong, too).

In terms of implementation, the logic in your recipe could look something like this:

options = [
    {"id": "STAND", "text": "🧍 standing"}, 
    {"id": "POINT", "text": "👉 pointing"}
]

def get_box_classification_stream(stream):
    for eg in stream:
        for span in eg.get("spans", []):  # the bounding boxes
            eg = copy.deepcopy(eg)
            eg["spans"] = [span]
            eg["options"] = options
            eg = prodigy.set_hashes(eg)
            yield eg

Another advantage of this approach is that it also makes it easy to slowly start automating parts of the process – for instance, recognising the actual person (i.e. drawing the box) might be something your model is able to do quite accurately pretty quickly, so you can mix in suggestions from the model and only focus on the classification of each box