Clarification on annotation capabilities

,
1

Hello,

I have a few thousand images that I'd like to annotate (either polygon or segmented) and looking for a tool that can run local on my computer. Does prodigy require previously annotated datasets to help with pre-annotations or does it come ready right out the box? For example, I am annotating people within images, when I load the images, will it already have a label/annotation around people (or segmented every pixel) that I can then correct, fine tune, or further define?

Also, does it allow more attributes connected to the same label? For example, a label of "Person" with attributes like height, weight, etc...

Does it use publicly available models or do I need to load my own model for the assisted annotation capability?

I didn't see a clear answer so wanted to ask for clarification. Thank you.

2

Hi! Pre-annotation in Prodigy works via Python functions, also called recipes – they can use an existing model, an API, a set of rules or any other logic to add annotations to the incoming examples so you can accept/reject or correct them. These models are not built into Prodigy itself, but you can integrate pretty much anything you can load in Python. For text-based and NLP-related workflows, Prodigy provides an out-of-the-box integration with spaCy (since that's easy and a library we develop as well).

So if you're doing computer vision and you have a model that already predicts something useful, you can use it to add bounding boxes. You can see examples of how this works here: Computer Vision · Prodigy · An annotation tool for AI, Machine Learning & NLP

Depending on your use case (and if your categories are pretty generic, like PERSON), you might find an existing pretrained model that's already working well enough to use it in the loop. If your data and/or categories are more specific, you might need to collect a few manual annotations first and then pre-train a model that can help you annotate later on.

This is definitely possible, although we'd recommend doing it in two steps: first, make sure your data includes the bounding boxes, next loop over all bounding boxes one at a time and annotate attributes, e.g. using a multiple choice UI, a free-form input etc. See the docs on custom interfaces for how to combine different blocks: Custom Interfaces · Prodigy · An annotation tool for AI, Machine Learning & NLP

Doing it in two steps means that you can evaluate both tasks separately and only go deeper once you've verified that the bounding boxes are correct and your label scheme works. It also makes it easy to sort annotations by type for faster annotation, lets you automate things and reduces the cognitive load on the annotator and helps focus on one decision at a time (bounding boxes or attributes for a given bounding box).

1 Like
3

Thank you Ines. Very helpful. I'm a newbie, do you all offer to setup the workflow for new users for a fee?

4

From what you've described, it sounds like you'd be most interested in some setup work related to the computer vision model, right? We currently don't offer consulting work around computer vision but you could post on this thread: spaCy/prodigy consultants?

Another thing to keep in mind is that a lot of the custom work you'd be looking for here isn't necessarily that specific to Prodigy – you'd mostly want a Python script that loads a computer vision model and API and outputs the predictions in a given JSON format. And a training workflow that loads annotations in this format. If you have that, the integration will be pretty straightforward. So even if you get someone to help you who hasn't worked with Prodigy before, they can likely still get you set up with the most important things you need :slightly_smiling_face:

5

Hey! I have used prodigy to identify person and now I'd like to add attributes to the person, e.g. height, weight, etc... I figured out how to add choice UI; however, how can I make the selections as an attribute of the bounding box?

example, there is an image with two people identified. I'd like to add attributes for both people in a way that is distinguishable and connected to the person bounded box.

I tried to implement this is prodigy and the choice options loaded and I was able to make selections; however, it isn't linked to the top-level label. Any idea of how it be remain linked to the top level label? Also, if there is more than one person, the script only loads one person per image, not all of them. Any guidance would be appreciated

        for eg in stream:
            for span in eg.get("spans", []):
                if span["label"] == "person":
                    task = copy.deepcopy(eg)
                    task['spans'] = [span]
                    task['options'] = options
            # eg["options"] = options
                yield task

Here is an example of the spans for one image

"spans": [
{"label": "person", "model": "yolov3-opencv", "box": [409, 13, 360, 653], "points": [[409, 13], [769, 13], [769, 666], [409, 666]], "confidence": 0.9993869662284851, "color": "#ff00ff", "qid": "1_person", "person_filters_yolo": [0, 1]}, 
{"label": "person", "model": "yolov3-opencv", "box": [131, 147, 293, 515], "points": [[131, 147], [424, 147], [424, 662], [131, 662]], "confidence": 0.9951488375663757, "color": "#ff00ff", "qid": "2_person", "person_filters_yolo": [0, 1]}, 
{"label": "person", "probability": 0.9997418522834778, "box": [149, 153, 256, 506], "points": [[149, 153], [405, 153], [405, 659], [149, 659]], "model": "facebook/detr-resnet-50", "color": "#ffff00", "qid": "61_person", "person_filters": [61, 98], "person_number": "person_0"}, 
{"label": "person", "probability": 0.9996631145477295, "box": [422, 32, 346, 625], "points": [[422, 32], [768, 32], [768, 657], [422, 657]], "model": "facebook/detr-resnet-50", "color": "#ffff00", "qid": "98_person", "person_filters": [61, 98], "person_number": "person_1"}]
6

hi @c00lcoder!

Thanks for your question.

I'm sorry, I don't understand what you mean by being "distinguishable and connection to the person bounded box". Can you elaborate more on what you want to add?

You mentioned earlier to add "height/weight". I'm not sure I understand that either. Those would be numerical values (100 to 300) and the choice UI only allows discrete categories (A, B, C). Can you explain more of why you were thinking of using the choice UI for those values?

7

Sure. I have a bounding box labeled Person. I am looking to add additional attributes about that person, such as height, weight, etc... These values will not be manually typed but rather selected from range options listed as choices.

When I use the method mentioned earlier, the ui works for choice; however, the results are added as "accept" in the overall jsonl dictionary for that image versus the span where the specific "person" bounding box is located in the jsonl file.

So to clarify, if I have an image with two persons...person A and person B...I want the attributes for person A to be saved in the span with the person A bounding box and likewise for person B. If the "accept" answer is saved in the overall jsonl dictionary, it is associated with the image and not the actual person bounding box.

Essentially, I cannot figure out how to loop through all of the spans and save the accepted answer to that specific span.

8

hi @c00lcoder!

Thanks for the clarification.

The before_db callback could help with the 2nd step to modify the data.

Like described previously, it would need to be a two-step process. First you create your bounding boxes, then add the attributes to the spans by iterating over each span separately and adding the choices with the before_db method to the spans.

It's important to know that before_db callback should be used sparingly and with caution. The docs mention this:

The before_db callback modifies the annotations and Prodigy will place whatever it returns in the database. You should therefore use it cautiously , since a small bug in your code could lead to data loss.

Step 1: Get person bounding boxes

Let's say you start with a .jsonl file that has the path to your images:

# image-sample.jsonl
{"image": "person-image.png"}

You first create the bounding boxes for the person by running:

python -m prodigy image.manual image-sample image-sample.jsonl --loader jsonl --label PERSON

localhost_8080_ (14)
localhost_8080_ (14)1920×1328 97.1 KB

Save your annotations so now your bounding box person annotations are in the Prodigy dataset image-sample.

Step 2: Custom recipe for additional attributes

Here's a custom recipe to provide four choice attributes for each person (e.g., female/male, brown/black hair).

# image-nesting.py
import prodigy
from prodigy.components.db import Database
from prodigy.components.db import connect
from prodigy import set_hashes

@prodigy.recipe(
    "image.nesting",
    dataset=("The dataset to use", "positional", None, str),
    origin_dataset=("The original dataset to get the images from", "positional", None, str),
)
def image_nesting(dataset: str, origin_dataset):    
    db = connect()      
    data = db.get_dataset(name=origin_dataset)
    
    def get_stream():
        for image in data:
            if "spans" in image:
                for span in image["spans"]:
                    image_copy = image.copy()
                    
                    image_copy["spans"] = [span]
                    image_copy["options"] = [
                        {"id": "female", "text": "female"},
                        {"id": "male", "text": "male"},
                        {"id": "brown", "text": "brown hair"},
                        {"id": "black", "text": "black hair"},
                        ]
                    image_copy["id"] = span["id"]
                    yield set_hashes(image_copy, task_keys=("spans", "image", "id"), input_keys=("spans", "image", "id"), overwrite=True)
    
    stream = get_stream()
    
    def before_db(examples):
        for eg in examples:
           if "spans" in eg and "accept" in eg:
               eg["spans"][0]["additional_info"] = eg["accept"]
        return examples
        
    return {
        "dataset": dataset,
        "stream": stream,
        "view_id": "choice",
        "config": {
            "choice_style": "multiple"
        },
        "before_db": before_db,
    }

You can then run this recipe as:

python -m prodigy image.nesting image-nesting image-sample -F image-nesting.py

localhost_8080_ (15)
localhost_8080_ (15)1920×1328 82.9 KB

We can then look at your new annotation:

python3 -m prodigy db-out image-nesting > image-nesting.jsonl

#image-nesting.jsonl
{
  "image": "data:image/png;base64, ...", # removed actual base64 for example
  "_input_hash": 1875068863,
  "_task_hash": -354213639,
  "_is_binary": false,
  "path": "person-image.png",
  "_view_id": "choice",
  "width": 400,
  "height": 267,
  "spans": [
    {
      "id": "5b63d3f8-e4c8-4160-8a7f-d6579ce210ad",
      "label": "PERSON",
      "color": "yellow",
      "x": 56.3,
      "y": 9,
      "height": 247,
      "width": 178,
      "center": [
        145.3,
        132.5
      ],
      "type": "rect",
      "points": [
        [
          56.3,
          9
        ],
        [
          56.3,
          256
        ],
        [
          234.3,
          256
        ],
        [
          234.3,
          9
        ]
      ],
      "additional_info": [
        "female",
        "brown"
      ]
    }
  ],
  "answer": "accept",
  "_timestamp": 1672848904,
  "options": [
    {
      "id": "female",
      "text": "female"
    },
    {
      "id": "male",
      "text": "male"
    },
    {
      "id": "brown",
      "text": "brown hair"
    },
    {
      "id": "black",
      "text": "black hair"
    }
  ],
  "id": "5b63d3f8-e4c8-4160-8a7f-d6579ce210ad",
  "config": {
    "choice_style": "multiple"
  },
  "accept": [
    "female",
    "brown"
  ]
}

Does this solve your problem?

9

I will test it out and let you know soon. I needed something quick so I found a temporary workaround but preference would be to figure this out using prodigy.

I'll let you know soon