Integrating Prodigy with Custom Platform and adding raw data dynamically without restarting Server

Hi All

I have a custom platform where millions of stories are stored on daily basis after NER and Classification.
For now, I am looking for an annotation only
I want to give a button at every article so that when the analyst clicks on that button, a new tab is open where data can be seen on the Prodigy platform, and their data can be annotated and analyst can also choose if wants to annotate for NER or classification or Relation.

I am not able to understand how to handle the stream of data. How to add text dynamically without restarting the server.
Most of the examples are pseudocode. It would be nice if you can give any concrete example where we can load the data dynamically without the need of restarting the server again and again

Hi! If you're annotating data for different tasks, you typically want to do this in different instances so you can keep state in memory, load a model if needed, save the different annotation types to separate datasets you can run training experiments with etc. We'd also recommend running longer annotation sessions at a time instead of annotating a single example at a time, which often isn't that useful.

So a better approach might be to include a feature to flag an example for annotation, and then periodically annotating the selected examples in batches.

If you want to implement this, you could have a loader that periodically queries from an external source, like an API or your database containing the flagged examples for annotation. Here are some basic examples loading from a file path or a custom source: Loaders and Input Data · Prodigy · An annotation tool for AI, Machine Learning & NLP – only that in your case, you'd make a request to your database or similar, and wrap in in while True: so it keeps looping until new data is available. New examples will then be queued up when you refresh the browser.