Editing annotations derived from model predictions with `image.manual`

,
1

I built up a custom recipe (see the link for the code) which uses predictions from a model to generate spans.

I would like to use this recipe for self-training. At the moment the model isn't super great, but it's workable. Not all the annotations are perfect, and some are missing. When I currently run this recipe, I see all the predicted annotations on the screen (with a new diagonal x-style double line across the annotation(s)).

Things I'm able to do:

  • drag annotations around
  • select annotations (for deleting)
  • make new annotations

The thing I'd really like to be able to do is correct the predicted annotations. I.e. resize the bounding box. I'm not sure if I'm just not pressing the right keys (is this something that requires SHIFT to be pressed?) or maybe it's just not currently allowed to do this...

Does Prodigy's image.manual custom recipe interface allow me to do what I'm trying to do? (i.e. amend annotations that have been predicted)

Thanks!

2

Hi @strickvl !

Not sure if this is what you meant, but if you want to review existing annotations, it should be possible. Prodigy (image.manual ) supports loading images from a directory or a JSONL file containing image URLs. If you have existing bounding box annotations (i.e., the ones predicted by a model or something), you can first convert them into Prodigy's format, and load them in to update their annotations.

Basically. you'd want to load the image data from a JSONL file, and pre-populate the "spans" with the annotated bounding boxes (c.f. Annotation interfaces · Prodigy · An annotation tool for AI, Machine Learning & NLP). You can probably do db-out then reload them back again.

{ "image": "https://images.unsplash.com/photo-1554415707-6e8cfc93fe23?w=400", 
"spans": [{"points": [[155, 15], [305, 15], [305, 160], [155, 160]], "label": "LAPTOP"}] }

You can also check this helper function to convert bounding box measurements to pixel coords.

3

Exactly! You can see in my code that I do exactly all of that.

The problem is that once I have the web UI up and running, I'm not sure if I am able to manually edit the boxes. I see that I can delete boxes, and I can create new boxes, but I was sort of hoping and assuming that I'd also be able to tweak the precise shape of the boxes that were predicted by my model. Currently if a box isn't exactly right, I have to delete the box and make a new one. This feels like a bit of an annotation anti-pattern, at least if I'm trying to save some time by having my model in the middle...

If this isn't already a feature, could it be added to a possible backlog?

(Thanks for your answer.)

4

Are you using the latest version of Prodigy? Maybe you ended up with the old legacy interface that didn't yet support resizing boxes?

In the image_manual UI, you should be able to resize a box by selecting it (via click/tap) and dragging one of the corners:

Screenshot 2021-12-20 at 11.54.33

You should also be able to try it out in the interactive demo here: https://prodi.gy/docs/api-interfaces#image_manual

5

I am using Prodigy v1.11.6, which I believe is the latest version.

I get the interface pictured in your reply when I run a custom recipe that does NOT have a model in the middle, predicting the annotations.

But when I do run this latest custom recipe that does insert eg['spans'] = ... into it, I get this other interface where I can't adjust the box by clicking and dragging a corner.

Not sure if this is a bug or what exactly is going on.

In any case, this is the interface that I see, for example, when running my 'model in the middle' custom recipe:

ScreenShot 2021-12-20 at 18.22.52
ScreenShot 2021-12-20 at 18.22.521346×1974 211 KB

(Code for my recipe was linked above in a gist.)

6

Are you 100% sure you're using the latest version of Prodigy? Based on what you describe, including your other thread, it really sounds like you ended up with the old image_manual UI that didn't yet support resizing boxes and didn't yet generate x/y/width/height.

Maybe double-check that you're definitely on the right version and that you haven't set "image_manual_legacy": true somewhere in your prodigy.json?

7

My local .prodigy.json file contains this:

{
  "theme": "dark",
  "batch_size": 10,
  "image_manual_font_size": 4,
  "image_manual_stroke_width": 1,
  "custom_theme": {
    "labels": {
      "CONTENT": "#00bfff",
      "REDACTION": "#ff6347"
    }
  }
}

My global .prodigy.json file is empty.

I'm in a virtual environment, and when I run pip list, among the lines I find:

prodigy                           1.11.6

Am I missing any checks to cover the version and any configuration weirdnesses for my environment?

8

Okay, sorry, I think I didn't actually look at your screenshot properly :woman_facepalming: It does look like the new interface, so I'm confused what's going on.

Are you able to click on the label of the bounding box in the top-left corner? This should select the bounding box, will shade it in the given colour and should allow you to resize it from the different corners.

9

So I can click the upper left corner of the bounding box. It selects it, does this weird thing where I get this inverse flag pattern. Before clicking:

ScreenShot 2021-12-21 at 14.32.39
ScreenShot 2021-12-21 at 14.32.391336×1900 196 KB

After clicking:

ScreenShot 2021-12-21 at 14.32.47
ScreenShot 2021-12-21 at 14.32.471344×1906 226 KB

But once I've clicked, I am unable to resize the box. If I try (in this example), it actually clicks through to the blue box behind it and starts dragging that blue box.

So AFAICS, there is no way to resize the box. It's take it or leave it in the current implementation.

10

Ohhh are these shapes generated by your model? Based on the highlighting, it looks like the points might be in the wrong order. So the shapes aren't actually rectangles, they're basically this double triangle shape. So the top right left is connected to the bottom right corner, not the top right corner, and so on.

11

Ok thank you! That was the fix. I got my xmax and xmin the wrong way round!

One other piece of feedback about the UI, since we're here on this example. I recorded a short video to show what I wanted to say about the ability to tweak the boxes.

I'm wondering if controlling when the double-arrow resizing is actually in Prodigy's hands at this point; maybe it's just a browser thing. But you can see it's a bit difficult to access the resizing, in any case. Probably in an ideal world what I'd want is for the resizing arrow to be available to me not just at the half-way point and the corners of the rectangle edges, but also along all the lines as well?

13

This appears to have been fixed now. Not sure if it was something not mentioned in the new release notes, but it is mostly working as I'd expect it to be working now. There are still some weird edge cases where I have no idea why the 'resize' double-arrow doesn't appear. To fix I have to deselect then reselect the annotation and then zoom in a lot.

14

Sorry for only getting to this now, I think I missed the previous post! And that's definitely strange because we didn't change anything that could be related to this so I wonder if it's indeed a browser thing or related to a recent browser update :thinking:

15

I can't for the life of me figure out how to predict when I'll be able to access the arrows to change the dimensions. I agree that it sounds like something going on in the browser or whatever is rendering those arrows. A bit annoying from the user perspective, but not sure what to be done about it either.

16

I can confirm, though, that I've tested it in Brave, Chrome & Safari (on a Mac) and they all have the same random behaviour.

17

Ah, interesting, I definitely haven't seen this before :thinking: One random idea: I wonder if it's related to the rescaling for larger images, in which case the bounding boxes are subtly scaled to match the final image size. Maybe we end up with differences of like 1 pixel there which makes the target area for the cursor smaller.

Your images are quite large, right? One thing you could try is resizing the images before loading them into Prodigy (e.g. programmatically) so their original size is closer to the display size. If that resolves it, this could point towards my theory.

18

I think that's a good theory. I'll have to think about how I implement that, because it'd be a pretty big shift in how I'm doing things to suddenly use images that are 1/4 the size etc.