Customizing Prodigy for End to End Video Annotation

[nlp-guy (Bilal)]

Dear Support Team,

I want to customize Prodigy for endoscopic video annotation. The annotation tasks might include tissue/tools/hands detection/segmentation, key points detection, and phases/action classification. Typical UI shall display the video frame, play/pause/next/previous controls, window-based searching widget, and the ability to annotate multiple objects via the video timeline and track the labels.

Is it possible to customize Prodigy for such kind of video annotation workload? There is an example Audio & Video recipe but I can't see how to customize it to show an image frame with video controls and annotation capability through the video timeline.

Can someone guide me through this customization of Prodigy? Your support will be greatly appreciated.

Many thanks and

Kind regards,
Bilal

I will great su

[ryanwesslen]

hi @nlp-guy!

Unfortunately, most of the functionality you're looking for isn't available out-of-the-box with Prodigy. As you've probably seen, typically we would recommend converting into images for every n-th frame or use some combination/similar functionality from the audio interface . I've included several of related posts where other users have asked for similar features.

I have made an internal issue about your request. Let us know if you have any additional details of what the functionality would look like. This could help us out if we look more out developing these features in the future.

Can you annotate Videos?

Hi! You could probably build an interface like this using custom HTML templates and JavaScript, but Prodigy doesn’t have a built-in interface for playing and labelling videos. If you need to actually play videos, jump around frames etc., there are probably much more specialised tools that will do a much better job at this. That said, in many cases, what you’re annotating in videos isn’t actually the live video, but rather every n-th frame as an image. That’s a workflow you can probably build qu…

Keypoint annotation - Vision

Hi! By keypoint annotation, do you mean annotating specific points on an image? There's no separate annotation mode for just points but depending on the task, one solution could be to use the regular bounding box and enable visualising the center point with show_bounding_box_center: https://prodi.gy/docs/api-interfaces#image_manual-settings You can also set image_manual_from_center to enable drawing from the center if that feels more natural. And you can customise the box to make it less promine…

Video and audio annotation simultaneously

Hi! The audio and video UI support annotating segments in the audio track – there's currently no interface for object tracking annotation in video files. However, if your goal is to just mark the speaker (and not track the speaker's movement across frames etc.), you could probably use a combination of the audio or video interface, and the image_manual interface with a still image from the video. See here for the docs on custom interfaces with blocks. The solution here re…

Object Tracking Implementation

Yay, I like that spirit I haven't worked on object tracking use cases myself, but just to make sure I understand the requirements correctly: your goal is essentially to annotate a given object / bounding box across the frames of your video, right? Since you'd be working on the individual frames, one option could be to use a workflow like image.manual and stream in the selected frames as images, in order. You can probably do this all programmatically in Python as part of …

[nlp-guy (Bilal)]

Thanks for sharing the links and asking for more details.

To begin with, a recipe that would allow users to upload a video file for classification where UI displays the labels at the top, followed by the video (maybe with play, stop, next, prev buttons) at the middle and timeline control at the bottom (like the audio_manual labeling interface but for interacting with frames).