Frequent KeyError via /give_answers

,
1

Prodigy Version: 1.8.4
OS: Mac
Python: 3.7.4
File Format Used: JSONL

Command used to start the server:

prodigy textcat.teach support en_core_web_lg data/interim/client_messages.jsonl --label UNIQUE_LABEL --patterns patterns.jsonl

During tagging, if the connection is left stale for more than a few seconds e.g. for a tea break - when resuming the annotation breaks with a KeyError. This key is usually label or spans.

Occasionally, it breaks even when decisions are made too fast. E.g. a quick xxxxx press.

Whenever this error is encountered, the work saved state is unclear. The browser says that it is not saved, but then on Ctrl+Cing the server - some records are written to SQLite. Was all the data saved? Was some of the data saved?

Here is stack-trace for reference:

Exception when serving /give_answers
Traceback (most recent call last):
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/waitress/channel.py", line 336, in service
    task.service()
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/waitress/task.py", line 175, in service
    self.execute()
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/waitress/task.py", line 452, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/hug/api.py", line 451, in api_auto_instantiate
    return module.__hug_wsgi__(*args, **kwargs)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/hug/interface.py", line 789, in __call__
    raise exception
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/hug/interface.py", line 762, in __call__
    self.render_content(self.call_function(input_parameters), context, request, response, **kwargs)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/hug/interface.py", line 698, in call_function
    return self.interface(**parameters)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/hug/interface.py", line 100, in __call__
    return __hug_internal_self._function(*args, **kwargs)
  File "/Users/nirant/miniconda3/envs/toys/lib/python3.7/site-packages/prodigy/_api/hug_app.py", line 282, in give_answers
    controller.receive_answers(answers, session_id=session_id)
  File "cython_src/prodigy/core.pyx", line 137, in prodigy.core.Controller.receive_answers
  File "cython_src/prodigy/util.pyx", line 384, in prodigy.util.combine_models.update
  File "cython_src/prodigy/models/textcat.pyx", line 219, in prodigy.models.textcat.TextClassifier.update
  File "cython_src/prodigy/components/preprocess.pyx", line 292, in prodigy.components.preprocess.convert_options_to_cats
KeyError: 'label'

I don't have a data splice right now which I can share right now. Happy to synthesize some if that is needed to reproduce the issue.

2

Hi @NirantK

I can reproduce this issue, especially using the shortcut keys, and I'm looking for an appropriate fix.

Can you elaborate on these circumstances a bit more? I'm not aware of any code in prodigy that accounts for how long you've been away from the computer. Does anyone use the computer while you're away? Do you lock the screen or put your computer to sleep? Any more information or specific steps to reproduce would be helpful.

3

It seems both of your issues can be attributed to rapid clicking in the UI. If you were to click rapidly and leave for tea, when you come back the invalid examples could still be in the queue to send to the server, and you would see the error (props to @ines for sussing this out) as you started again.

I have a fix proposed for the next release. Thanks for reporting this!

4

Thanks @justindujardin and @ines for looking into this.

To confirm, this is probably the most frequent patterns I have as well:

If you were to click rapidly and leave for tea, when you come back the invalid examples could still be in the queue to send to the server, and you would see the error

Is there an interim code-fix or behaviour change that I can make to avoid this? The entire toolchain becomes brittle because of these specific issues.

1 Like
5

I can't think of an easy workaround that wouldn't require us to ship a fix – but we're about to publish a patch version anyways to fix a few other small issues, so we'll definitely include this.

Edit: Actually, an easy workaround would be to add a hack to the give_answers endpoint that filters out empty tasks received back from the client before they're passed back to the controller.

Also, I'm pretty sure that something must have changed in the latest version (or one of the latest versions) that allowed the client to produce "empty" tasks on rapid clicks. The issue just came up again here and it's never come up before. So it's unlikely to be a long-standing issue and likely a regression introduced in v1.8.4 (which should also help narrow it down).

Edit 2: Found and fixed the issue. It was indeed a regression introduced only in v1.8.4.
Edit 3: Just released v1.8.5, which should resolve the issue! Thanks again for the report :+1:

1 Like
6

Thanks for the fast fix Ines!