Couldn't save annotations - cache/retry policy

,
#1

Hi,

I'm a bit confused about Prodigy keeping annotated documents cached until saving occurs.

The situation:

  • A document is being annotated.
  • The connection to the server is cut (for whatever reason; to reproduce you can simply shut it down).
  • The annotator accepts and saves the document, and the error "couldn't connect to server" pops up.
  • The document is "cached" into the history. So far so good. No annotations are lost.
  • Now, when the user clicks the save icon again, the document suddenly disappears from the history, but IS counted in the UI stats.
  • When the connection to the server is restored, and another document is saved, the previous "failed" document isn't in the database (even though it IS in the UI stats).

Is this a bug, or am I overlooking something?

Thanks a lot in advance!

Vincent

#2

Hi! When you hit "save" in the UI or a full batch of annotations is available outside of the history (that still allows you to undo, typically the latest 10 examples), Prodigy will "outbox" the annotations and then try to send them to the server. If that fails, the answers stay in the outbox and Prodigy will try again, either when you hit save again, or as you collect more annotations. If the server replies with a successful response, the outbox is cleared. If not, the answers should stay in the outbox.

So even if they're not in the history anymore, they should still be there and if the connection comes back and you save again, the whole outbox should be sent. (Unless the user closes or refreshes the browser.)

#3

Hey Ines,

Thanks a lot for your reply!

I have narrowed down the issue to the "instant_submit" option which I have set to true. My hypothesis is that the auto-save function doesn't try to send the outbox to the DB. Perhaps that's why the failed documents were never picked up, and kept in the outbox indefinitely.

Would that be possible?

Thanks!
Vincent

#4

Ahh, that's interesting – and this could definitely be an explanation. I'll take a look at this! Prodigy should definitely move failed sent answers to the outbox if instant_submit is enabled.

(At least with this setup, the maximum of examples you could ever lose in the worst case scenario is 1, assuming you stop submitting answers when the error message is displayed.)