hi @bev.manz!
Thanks for your question and welcome to the Prodigy community
You mentioned this problem wasn't an issue in the past. What changed?
Did you have this same architecture on local machines and only now moved to AWS (since you mentioned EC2)?
Or alternatively, is it textcat.teach
? Did this same problem happen with non-model-in-the-loop recipes like textcat.manual
or ner.manual
? Just as a test, get some sample data and run through a few examples for manual recipes, trying to hit multiple sessions at the same time. If that works, then this would suggest it's something wrong with running the model.
Are your documents large or very long?
Granted you're using textcat.teach
which may not be as intensive as ner.teach
due to beam search.
Any possible issues with memory too?
Are you modifying the port or using the default 8080
port? Any chance there's anything else that runs on that port on your machine?
Have you also started using Prodigy logging?
You can add PRODIGY_LOGGING=basic
or PRODIGY_LOGGING=verbose
to your Prodigy command to add more detailed logs. I typically recommend always running this in development environments.
Related, you can also add /docs
to view the interactive API so you can try to figure out if one endpoint is crashing.
Also, it may not be relevant, could there be any issues with security since you're running on http://
, not https://
?
As a best practice, you may want to avoid having users try different session names. I know you're doing this for now b/c of the crashing as a short term solution.
But I mention this in the future if you need to analyze annotator quality (e.g., track inter-annotator agreement) you'll appreciate having correct session names. In fact, a good practice is setting PRODIGY_ALLOWED_SESSIONS
so that you have a finite choices of what sessions are allowed. For instance, PRODIGY_ALLOWED_SESSIONS=alex,jo
would only allow ?session=alex
and ?session=jo
, and other parameters would raise an error. This prevents fat finger errors with invalid session names.
This is secondary to your key problem so you can ignore for now. Hopefully if some of the ideas above can help identify the crashing problem, this may be something good to build into your process afterwards.
Hope this provides some avenues to explore. Let us know if this works (or doesn't). Also, feel free to provide logs or other details.