Is it possible to customize annotation UI?

[daniel]

Is it possible to customize built in annotation UI (beyond adding meta keys)?

For example: I’m using the ner_manual UI to annotate youtube video titles, I add the ID of the video to the meta and it is displayed at the bottom. But I’d like to be able to convert this meta data to a link at the bottom of the UI, for quick access to the video (opened in new tab) by a single click.

A more advanced scenario for ner_manual UI would be to automatically convert any selected labels to terms in google search URL that is accessible by a click on a link at the bottom of the UI. I should be able to do this easily if I could inject some javascript to the annotation UI.

This should help me speed up the annotation process since I often can’t figure out the correct annotations from the context and need to look it up. And copy/paste is not working well with the ner_manual UI.

I’m sure this can be useful with other annotation interfaces.

I saw that it is possible to add custom HTML to the HTML UI but it doesn’t seem to work with the other interfaces.

Thanks.

[ines (Ines Montani)]

Not yet – but possibly soon! In v1.4.0, we quietly shipped very experimental support for the features described here:

Custom view templates with scripts

Update: All works pretty smoothly already! : [custom_js_poc] The above example only needed the following html_template and javascript config: <button class="custom-button" onClick="updateText()"> Text to uppercase </button> <br /> <strong>{{text}}</strong> let upper = false; function updateText() { const text = window.prodigy.content.text; const newText = !upper ? text.toUpperCase() : text.toLowerCase(); window.prodigy.update({ text: newText }); upper = !upper; …

The custom scripts are currently only available for the html interface, but if it all works well, we might consider adding an option for custom HTML and scripts to all interfaces. (In the case of ner_manual, we could, for example, allow custom HTML in the area below the text – so you could insert your own markup and links there, and write a JavaScript function that updates the data based on the current selection).

Another idea could be to break your task down into a classification task first – e.g. annotate if the title is about topic X, includes type Y or has context Z. This would let you use a custom HTML interface, or even the choice mode with custom HTML (and even a list of several topic options to choose from). Once you've pre-classified your examples, you can then move on to annotate them manually.

@daniel

But I’d like to be able to convert this meta data to a link at the bottom of the UI, for quick access to the video (opened in new tab) by a single click.

Prodigy should convert all URLs in the task meta data to links, so in the meantime, this could be a simple solution for this particular case (where you actually know the link upfront).

@daniel

This should help me speed up the annotation process since I often can’t figure out the correct annotations from the context and need to look it up.

One quick note on this: Keep in mind that the harder it is for you to make the annotation decisions based on the context, the harder it will be for the model to pick up on context clues and learn the labels you assign. This is especially true if you're relying on mostly manual annotations. So if you can come up with creative ways to pack more text context into the individual task that's displayed, it might help the model learn your labels (and help you annotate them).

[daniel]

@ines

The custom scripts are currently only available for the html interface, but if it all works well, we might consider adding an option for custom HTML and scripts to all interfaces. (In the case of ner_manual, we could, for example, allow custom HTML in the area below the text – so you could insert your own markup and links there, and write a JavaScript function that updates the data based on the current selection).

Great. That is exactly what I had in mind and I think it will be quite useful.

I the meanwhile, I'll hack something with a script in Greasemonkey.

@ines

One quick note on this: Keep in mind that the harder it is for you to make the annotation decisions based on the context, the harder it will be for the model to pick up on context clues and learn the labels you assign. This is especially true if you’re relying on mostly manual annotations. So if you can come up with creative ways to pack more text context into the individual task that’s displayed, it might help the model learn your labels (and help you annotate them).

Yes, I understand but this is just the nature of the data I am working with. Usually, there is a context that can help identify the labels but sometimes it isn't immediately apparent to me without looking it up.

[daniel]

If anyone has a similar requirement, I created a simple script that can be pasted to Greasemonkey/Tampermonkey, here is a link to the gist.

Works with NER manual/make-gold recipes. It extracts a value from the current example’s meta data and creates a link (in my case for youtube), dynamically extracts the currently marked labels and creates a Google search URL for them, then adds these links to a new container added below Prodigy’s display div.

Of course, it may need to be modified for different themes and use cases.

When custom HTML in build-in interfaces will be added, the same technique can be used without Greasemonkey/Tampermonkey.

Note that since the CSS classes in Prodigy’s UI are auto generated, the script will break when they are regenerated (next build?).

[ines (Ines Montani)]

Thanks so much for sharing your script, this looks really elegant! For v1.5.0, we should be able to "officially" allow custom JS and HTML across all interfaces – especially considering it already works well using a solution like this.

@daniel

Note that since the CSS classes in Prodigy’s UI are auto generated, the script will break when they are regenerated (next build?).

Yes, I wouldn't want to promise consistency across versions here, because I'm pretty sure Styled Components regenerates the mangled class names on each build. v.1.4.1, which we've just released, also comes with some refactoring on the front-end, so I'm pretty sure you'll have to update the class names.

For the next version, I could also assign additional human-readable class names to the container elements – e.g. .prodigy-card or something. This will also make it easier to access those elements via custom JS going forward. Prodigy could also dispatch more custom events if needed – I'm open to suggestions here. At the moment, there's only prodigyanswer, but we could easily have more. Your script would then only have to listen to the Prodigy event(s) and won't have to do as much observing.

[daniel]

@ines

Thanks so much for sharing your script, this looks really elegant! For v1.5.0, we should be able to “officially” allow custom JS and HTML across all interfaces – especially considering it already works well using a solution like this.

Good news!

@ines

Yes, I wouldn’t want to promise consistency across versions here, because I’m pretty sure Styled Components regenerates the mangled class names on each build. v.1.4.1, which we’ve just released, also comes with some refactoring on the front-end, so I’m pretty sure you’ll have to update the class names.

Yeah, the class names did change. I will update the gist.

@ines

For the next version, I could also assign additional human-readable class names to the container elements – e.g. .prodigy-card or something. This will also make it easier to access those elements via custom JS going forward. Prodigy could also dispatch more custom events if needed – I’m open to suggestions here. At the moment, there’s only prodigyanswer, but we could easily have more. Your script would then only have to listen to the Prodigy event(s) and won’t have to do as much observing.

Adding fixed class names to key UI elements sounds great. For adding UI elements, the "profile card" container is good to have, so additional UI elements can be added below it (similar to what I did in the user script). Other classes are nice to have if the customization needs to retrieve data from the HTML elements, but if you are adding events (which would be even better) -

For custom JS events, I can think of two:

1. An event that fires each time Prodigy loads a new example. Passing as parameters the text, meta data object and whatever other data that is fixed per example
2. An event that fires each time annotation changes, either when new example is loaded or when user modifies the annotation (e.g. when user marks entities in ner.manual or ner.make-gold), passing whatever data that is relevant to the annotation (e.g. new entities in ner)

Also, there needs to be a good way to specify the custom HTML either per UI view or recipe. Maybe a command line option to specify HTML file? Or make the html_template entry in the config to accept a dictionary that maps view_ids to HTML templates. Because adding custom HTML per example or having just one HTML template specified in the config is not ideal.

I'll be happy to give further feedback or beta test, if needed.