We can expose the settings for the timeout and that's probably a good idea – although, you typically want to at least have some level of work stealing enabled because otherwise, an annotator opening the app, looking around and then closing it again will lead to a batch not being annotated, which is typically not what you want.
If you enable PRODIGY_LOGGING=basic, you should see a log message FEED: re-adding open tasks to stream whenever tasks are re-added for the next annotators. This should give you an idea whether this happens often in your scenario.