I'm running an audio annotation tool on EC2 + S3. I have a setup where a server_media.py serves data from the S3 bucket to the Prodigy tool. The code works well, but sometimes it crashes and doesn't work for a few hours. It magically starts working again with no changes to the environment or the code. I'm unable to reproduce the error, but I'd be happy to jump on a call whenever the team is available. Thanks!
For example, Prodigy starts as usual and points to port 8080, but without the Ctrl + C, it exits (Prodigy crashes) and shows me the custom message that I have created, which checks how many annotations were done in a particular session and says "No new annotations were done" if no annotations were made.
Let's first try to see if there are any Python errors related to Prodigy exiting. Could you rerun the session for PRODIGY_LOGGING=verbose (docs) and see of here are any error messages?
The fact that it "re-starts" itself makes me think that it never really exits but perhaps there are some resource e.g memory / database constraints that makes it idle and waiting on the data server or the database? Could you figure it out anything from the EC2 logs? Checking memory usage would be another thing to do.
You're right! The issue was with the ports. Even though the process ended, those ports remained occupied (or restarted occupying those ports even after being killed), which prevented Prodigy from using those ports again, leading to the issue where Prodigy would abruptly stop when those ports were again occupied by the processes for listening. Sometimes it worked, and sometimes it didn't. I was able to fix this. Thanks again!
