Is there a way to intercept recently completed annotations?

, ,
1

Hello there,

I just want to ask if there is any way I can extract annotations that have been recently completed but haven’t been saved to the external database (example, link, dataset). I basically want to extract the annotations the instant they are completed in the interface, rather than wait for them to be saved to the external database (which only happens when a) we press the save button or b) we reach the batch_size specified in prodigy.json)

Thank you and I appreciate your help!

2

Hi! One of the API design aspects that makes this difficult at the moment is that annotations that aren’t submitted yet are only stored on the client and are considered “unconfirmed” – i.e. the user can still undo and edit them (which is also why the app sends the answers back in batches).

Once we extend the currently experimental custom JavaScript support to other annotation interfaces and expose the full annotation task via the prodigyanswer event (fired when the user clicks the accept/reject/ignore button), you could in theory implement logic that’d intercept the answer on the client – but you’d also have to implement your own logic to handle undos and updated answers.

If you want the answers quicker and you’re not using an active learning-powered workflow where batches matter, you could also try setting your batch_size to 1? The recipe’s update method will give you access to the annotations as soon as they’re received – alternatively, you could even edit the /get_questions endpoint in app.py to get the request data as soon as it comes back from the web app.

3

Hello Ines,

Thanks for the quick reply! Yes, we have set the batch size to 1 (even 0, but of course that doesn’t work XD) but we don’t get it immediately. The annotation only appears in our example database two tasks later at least. Same thing with the update method - it only activates when we press the save button or the batch_size is reached (and batch_size, like we said, is already delayed).

Just one more question, have you already extended the currently experimental custom JavaScript support to other annotation interfaces? Or is that something you are currently working on?

Thank you for your help!

4

Ah yes, even with a batch size of 1, it's not going to be instant, because the latest answer is kept on the client to allow the user to undo.

What exactly are you trying to do btw that requires instant access to the latest decision?

Not yet – we wanted to make sure people could test it with custom HTML first. But that's been going well, so I definiely want to extend it to other interfaces in a future release :slightly_smiling_face:

5

Hey Ines,

Sorry that it took a while for me reply as I was on vacation.

What we are trying to do is to copy over annotations from the previously annotated sentence to the current sentence. This is because we are annotating the same sentence multiple times. The annotations are pretty similar each time but a few changes will need to be made.

6

Thanks for the explanation!

I definitely want to roll out the custom JS support for all interfaces with the next release. I think for your use case, exposing more data via the event fired on answer should probably do the trick? At the moment, the custom prodigyanswer event only includes the string value of the answer (and is only fired if you use the experimental JS feature in the HTML view).

If we also exposed the task data, you’d be able to do something like this:

document.addEventListener('prodigyanswer', event => {
    const { task, answer } = event.detail
    if (answer == 'accept') {
        // do something with the task here
    }
})