textcat.eval crash

Invoked thusly:

prodigy textcat.eval human_eval /tmp/model usertext.jsonl --label HUMAN_OR_BOT --exclude human_v9

And crashed:

Traceback (most recent call last):
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/waitress/channel.py", line 338, in service
    task.service()
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/waitress/task.py", line 169, in service
    self.execute()
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/waitress/task.py", line 399, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/hug/api.py", line 423, in 
api_auto_instantiate
    return module.__hug_wsgi__(*args, **kwargs)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/hug/interface.py", line 793, in __call__
    raise exception
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/hug/interface.py", line 766, in __call__
    self.render_content(self.call_function(input_parameters), context, request, response, **kwargs)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/hug/interface.py", line 703, in call_function
    return self.interface(**parameters)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/hug/interface.py", line 100, in __call__
    return __hug_internal_self._function(*args, **kwargs)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/prodigy/app.py", line 105, in get_questions
    tasks = controller.get_questions()
  File "cython_src/prodigy/core.pyx", line 109, in prodigy.core.Controller.get_questions
  File "cython_src/prodigy/components/feeds.pyx", line 56, in 
prodigy.components.feeds.SharedFeed.get_questions
  File "cython_src/prodigy/components/feeds.pyx", line 61, in 
prodigy.components.feeds.SharedFeed.get_next_batch
  File "cython_src/prodigy/components/feeds.pyx", line 130, in 
prodigy.components.feeds.SessionFeed.get_session_stream
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/toolz/itertoolz.py", line 368, in first
    return next(iter(seq))
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/prodigy/recipes/textcat.py", line 227, in 
add_labels_to_stream
    eg = copy.deepcopy(eg)
UnboundLocalError: local variable 'eg' referenced before assignment

Then crashed again as I tried to exit:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/prodigy/__main__.py", line 261, in <module>
    server(controller, controller.config)
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/prodigy/app.py", line 58, in server
    controller.save()
  File "cython_src/prodigy/core.pyx", line 222, in prodigy.core.Controller.save
  File "/home/tal/.virtualenvs/prodigy/lib/python3.6/site-packages/prodigy/recipes/textcat.py", line 234, in on_exit
    data = dict(model.evaluate(examples))
  File "cython_src/prodigy/models/textcat.pyx", line 230, in prodigy.models.textcat.TextClassifier.evaluate
ZeroDivisionError: float division
1 Like

Thanks for the report!

I’m surprised this hasn’t come up before or in our tests. This looks like a bug, sorry about that! Should be straightforward to fix, though: In line 283 of prodigy/recipes/textcat.py, try changing it to eg = copy.deepcopy(task). To find the location of your Prodigy installation, you can run the following:

python -c "import prodigy; print(prodigy.__file__)"

I’ve already fixed this issue in our codebase so we can include it with the next release.

The second error you’ve seen is likely related, because at the end of the recipe, it’s trying to get the evaluation numbers, but nothing is there, since the recipe didn’t even really get started. So at some point, it ends up dividing by zero. We can probably make Prodigy fail more gracefully here, though!

1 Like