Hello, thank you and thanks for the speedy response!
The only change that has occurred is that we've change EC2 size from one that had no gpu to one with gpu. Basically from a m4.4xlarge to a p3.2xlarge. I'm not sure if this could be related? I've monitored the runtime through htop in terminal though and can't see prodigy using much memory or cpu (max 2GB).
We've only used textcat.teach and when I say we've not had this issue previously, I mean it has just not been so bad, so it used to only crash for shorter periods of time and not as regularly.
They can vary in length, there are definitely texts that are quite long, however it doesn't seem like this is impacting the system as you can label some very long texts very quickly and it can crash on the shortest piece of text. It also crashes as you select the label, for example freezes with a highlighted frame of either red or green around the text depending on your 'accept' or 'reject' the piece of text.
I've not noticed anything memory related.
The only other detail is that I seem to be able to repeat the crash. For example if I have a crashed session on:
http://MYIP:MYPORT/?session=username3
then close the tab, then reopen the same url, it will allow me to smoothly label the same 4 items then crash on the 5th exact piece of text that it crashed on previously.
However if I open a new url say username4, then it will smoothly label for quite a while and can sometimes not crash even on very large pieces of text.
This is a good point I will test it again with logging, hopefully this will highlight some issue.