# ✨ Idea: Image segmentation annotation interface

**URL:** <https://support.prodi.gy/t/idea-image-segmentation-annotation-interface/42>\
**Category:** Uncategorized\
**Tags:** enhancement, image\
**Created:** [September 25, 2017, 11:49pm UTC](https://support.prodi.gy/t/idea-image-segmentation-annotation-interface/42 "2017-09-25T23:49:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [September 25, 2017, 11:49pm UTC](https://support.prodi.gy/t/idea-image-segmentation-annotation-interface/42/1 "2017-09-25T23:49:25Z")

</div>

Inspired by [this tweet](https://twitter.com/jongold/status/912432262003662848), I started working on some experiments for an addition to the `image` annotation interface. Here's a first draft – thoughts and feedback appreciated! 😃

https://codepen.io/ines/embed/preview/wrgrMM?height=300&slug-hash=wrgrMM&default-tabs=html,result&host=https://codepen.io

Similar to the `ner` mode, the `image` mode could then also take an optional `spans` property, consisting of the segment coordinates (in px, relative to the original image), and an optional label and colour:

```json
{
    "image": "desk.jpg", 
    "spans": [{
        "points": [[150,80], [270,100], [250,200], [170,240], [100,200]],
        "color": "yellow",
        "label": "LAPTOP"
    }]
}

```

* * *

**Note:** Prodigy currently doesn't come with built-in image models, but they're definitely on our list. We've mostly been focusing on NLP so far, since this is what we know best. In the meantime, you should be able to plug in your own image segmentation model via a custom recipe. All the model needs to do is predict segments and attach scores, and provide a method to update it with annotated examples.

```python
import prodigy
from prodigy.components.loaders import Images
from prodigy.components.sorters import prefer_uncertain

@prodigy.recipe('segment-images')
def segment_images(dataset, image_dir):
    stream = Images(image_dir) # stream in image tasks from a directory
    model = load_my_model() # load model that extracts segments & assigns score
    return {
        'dataset': dataset,
        'view_id': 'image',
        'stream': prefer_uncertain(model(stream)),
        'update': model.update
    }

```

```bash
prodigy segment-images my_dataset /images -F recipe.py

```

---

<div class="post-metadata">

**Author:** ![c00lcoder](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/c00lcoder/32/2426_2.png) [@c00lcoder](https://support.prodi.gy/u/c00lcoder)\
**Post date:** [July 21, 2022, 1:31am UTC](https://support.prodi.gy/t/idea-image-segmentation-annotation-interface/42/2 "2022-07-21T01:31:01Z")

</div>

Image Segmentation would be perfect feature for my computer vision use case. Curious, is semantic/panoptic segmentation visualization supported currently or is there a roadmap that includes support for pixel level annotations like that?

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [July 27, 2022, 1:21pm UTC](https://support.prodi.gy/t/idea-image-segmentation-annotation-interface/42/3 "2022-07-27T13:21:56Z")

</div>

Hi Gerald,

we don't support pixel-level annotations at the moment. I'll keep it in mind for a future version.

For now, you _can_ convert the polygon annotations to pixel annotations with a custom Python script. You might enjoy this answer for some inspiration:

> [@Annotation for binary image segmentation in Prodigy?](https://support.prodi.gy/t/annotation-for-binary-image-segmentation-in-prodigy/5799/2):
>
> I'm currently not aware of Prodigy being able to turn the drawn shapes into image masks. You'd need a separate, custom, Python script for that. That said, I think the [Python image library](https://pillow.readthedocs.io/en/stable/) can be very helpful here. import numpy as np from PIL import Image img = Image.open("path/to/img.png").convert("RBG") arr = np.array(img) This gives you a numpy array arr that represents the images' RGB values. If you have a simple shape, like a square, then you might be able to assign a mask via: mask = …