Feature Request: Instructions for Labeling


(Motoki Wu) #1

Hi, I think it would great if we can put some instructions for labeling (say in the sidebar) so it would be a constant reminder on how to label (some labeling tasks can get a little confusing).

(Ines Montani) #2

Sure, that’s a nice idea!

If the instructions are not too long, you could use the dataset’s description, which can be set via --description on the command line when creating a new set. The text you enter here will be displayed in the sidebar, underneath the dataset name etc. The only problem is that it’s not so easy to edit or change the description afterwards and once the set is created.

(Andy Halterman) #3

An NER-specific version of this that I’ve wished for is to display the definition for the named entity type that’s currently being asked about. This would be especially nice for LOC vs. GPE, which annotators mess up all the time, or the more obscure ones like EVENT. Not sure if other people would find that useful or worth the clutter, and it’s only really useful when people are starting out.

(Ines Montani) #4

That’s an interesting idea – maybe we do need some option to display a “help” modal or some other way of showing more information (like the annotation manual) without cluttering the UI.

In the meantime, a simple workaround could be to add the label that occurs in the task to its meta, so it’s displayed underneath the card. But of course, this only really works in ner.teach when there’s only one label per task:

labels_desc = {'GPE': 'geopolitical entity - everything with a governing body!'}
for eg in stream:  # assuming a stream with one span per example
    label = eg['spans'][0]['label']
    if label in labels_desc:
        eg['meta'][label] = labels_desc[label]

(Ines Montani) #5

Quick update: Just implemented this feature and it will be available in the next release :tada: You can then specify a config parameter instructions that takes a path to a text file (both plain text and HTML are supported, so it should be very flexible in terms of presentation). The instructions will then be displayed in a modal, accessible via the little help icon at the top, or by pressing h.


Because it’s a regular config setting, you can either have global or project-based instructions, or add them within the recipe config.

    "instructions": "/path/to/instructions.html"

(Ines Montani) #6

Implemented in v1.4.0 :tada: See my comment above or this section in the docs for details.

(Andy Halterman) #7

I’ve really been liking the instructions feature. Thanks!

I’m running into a strange bug, though. When I run Prodigy on my local Mac machine, the instructions work just fine. When I run it on a server, clicking the instructions blanks out the entire page. The only differences seem to be that it’s on Linux vs. Mac and running on instead of on localhost. Any idea what could be causing it? I can share the page where it’s running (privately) if you’d like to take a look.

(Ines Montani) #8

@andy Thanks, and hmm, that’s strange and shouldn’t be happening! The blank page happens because an error on the front-end isn’t caught correctly (sorry about that – likely forgot to add the boundaries around the modal correctly, since it’s happening outside of the regular annotation area).

Could you check the browser console for the error? (It’s a compiled app, so likely not as meaningful – but it’d still be good to verify.)

Also, can you double-check that the instructions markup that goes out is read in correctly? This happens in the server function in the app.py (which is included with Prodigy), so you can simply add print(config['instructions']) at the end. A possible explanation could be that your two machines produce different results here and the front-end can’t handle one of those and dies.

If none of this gets us any further, then yes, feel free to email me with more details!

(Andy Halterman) #9

@ines Printing out in app.py gives the same instructions locally and on the server. Here’s the error on the server from the console, but it’s unfortunately not that meaningful to me:

bundle.js:1 ReferenceError: translate is not defined
    at t.value (bundle.js:1)
    at u (bundle.js:1)
    at beginWork (bundle.js:1)
    at r (bundle.js:1)
    at i (bundle.js:1)
    at a (bundle.js:1)
    at S (bundle.js:1)
    at x (bundle.js:1)
    at batchedUpdates (bundle.js:1)
    at lt (bundle.js:1)
u @ bundle.js:1
a @ bundle.js:1
S @ bundle.js:1
x @ bundle.js:1
batchedUpdates @ bundle.js:1
lt @ bundle.js:1
$t @ bundle.js:1

(Ines Montani) #10

Thanks – looks like there was possibly a bug in the front-end that caused this. Is it possible that your machines are running different versions of Prodigy? Already fixed the bug and it will be included in the next release. (If you’re blocking on this, let me know and I can send you an updated JS bundle that you can replace!)

(Andy Halterman) #11

Yes, it turns out they’re different versions. The version with the bug is 1.4.2. My local computer without the error is running 1.4.0. If you wouldn’t mind sending the JS, that would be great! We have a few annotators starting a new task today and it would be great to have the instructions up to remind them.

(Raffaele) #12

Hi all,
I am in the same situation as Andy. I got the translate is not defined error with version 1.4.2 and I would like to give clear instructions to the annotators that are gonna start next week.
It would be very helpful to have the temporary bundle.js to use or, if it is not feasible, an ETA on this fix.

Many thanks for your group amazing work.

(Ines Montani) #13

@raffaele Sorry about that bug – I managed to track down the issue and it will be included in the next release (should be coming this week and will include a few other improvements and features). In the meantime, feel free to email me at ines@explosion.ai and I’m happy to send you the patched version.

(Raffaele) #14

Thank you for the quick answer @ines. This is really amazing. I look forward for the the next release.

Many thanks.