Hi Team,
We are using Prodigy tool to annotate NER - This tool is used by two of our agents.
During the production, the prodigy tool is terminating without leaving any error in the terminal log. I have attached the snapshot below.
I'm using an high end machine for hosting with 32gb ram and 16 core processor, resources free by 70%. I'm Using an company licensed version.
I have no clue why this is terminating! Need an urgent help on this issue?
Hi! How are you serving Prodigy? Are you using a reverse proxy? And how/when does this happen and how often? Also, are you using named multi-user sessions for your annotators, or are you spinning up separate instances?
Hi Ines,
How are you serving Prodigy?
Are you using a reverse proxy?
how/when does this happen and how often?
are you using named multi-user sessions for your annotators, or are you spinning up separate instances?
Okay, that's good to know. Do you have any other details about the setup? Is this running inside a Docker container for instance?
This will help track down what the problem could be. For example, one possible explanation is that when the annotators connect via their session with multiple threads and some error occurs in one of the threads, that error is not surfaced to the main process correctly.
@ines
Do you have any other details about the setup? Is this running inside a Docker container for instance?
I have a same issue snapshot taken today attached below for you reference,
Prodigy.json setup is here
{
"theme": "basic",
"custom_theme": {},
"batch_size": 10,
"port": xxxx,
"host": "xxxxxxx",
"cors": true,
"db": "sqlite",
"db_settings":{"sqlite":
{
"name": "prodigy.db",
"path": "/home/yyyy/prodigy/prodigy2_master_path"
}},
"api_keys": {},
"validate": false,
"auto_create": true,
"auto_exclude_current": true,
"instant_submit": false,
"feed_overlap": false,
"show_stats": true,
"hide_meta": false,
"show_flag": false,
"instructions": false,
"swipe": false,
"split_sents_threshold":5000,
"diff_style": "words",
"html_template": false,
"global_css": null,
"javascript": null,
"writing_dir": "ltr",
"hide_true_newline_tokens": false,
"ner_manual_require_click": false,
"ner_manual_label_style": "list",
"choice_style": "single",
"choice_auto_accept": false,
"darken_image": 0,
"show_bounding_box_center": false,
"preview_bounding_boxes": false,
"shade_bounding_boxes": false
}
I have used named multi session via cmd
PRODIGY_ALLOWED_SESSIONS=minsky1,minsky2
Hey @Prabakaran , thanks for the extra info.
It is indeed very strange that the process is just terminating, not even hanging, without any type of exception or log.
Maybe we could check if there's something else going on with asyncio underneath. Could you please try running it in asyncio debug mode? https://docs.python.org/3/library/asyncio-dev.html#debug-mode
You just have to set the env var PYTHONASYNCIODEBUG=1
Maybe that can help us get more information about what could be happening.
Another idea that you could try is installing https://github.com/Qix-/better-exceptions in your environment, in case that can help to show more info.
And if you have a custom recipe, you might want to add Sentry https://sentry.io/ so that any errors are reported there, sometimes it can give you even better information than what is printed out in the terminal.
And if you have a custom recipe, it might also help while debugging to install devtools: https://python-devtools.helpmanual.io/ and use its debug() function.
