I am trying to run Prodigy inside one of my view functions.
This is the error I'm getting:
By the way, prodigy runs when I execute the 'prodigy ner.manual ner_news_headlines blank:en ./news_headlines.jsonl --label PERSON,ORG,PRODUCT,LOCATION' command in terminal.
Hey @akerestecioglu!
Yes, the problem is that Prodigy will run a web server itself for most of the commands.
So you shouldn't run it from inside another web server, because weird things would happen. The first thing is that if you send several requests to the same Flask view, it would try to run several servers at the same time, but I see you are already checking available ports.
The next is, you will have strange issues like this if you try to start an asyncio event loop (done automatically to start the web server) from within a thread, and it seems you are running Flask with threads.
Also, prodigy.serve() will start an infinite loop for the web server, so that line will block there and never return, and the line:
return render_template('welcome_user.html')
...will never really execute.
And then, even if you were able to run it in some way, if the client cancels the request (for example the internet connection goes down for a second), that would cancel the current execution of Prodigy.
And a bunch of other potential issues...
I imagine that your intention is to control Prodigy from a Flask application, as a control dashboard or similar. In that case, it would be better to start prodigy as a complete independent subprocess, without waiting for it to finish and making sure the process is not killed if the Flask application is killed.
But then you would also need a way to manage the running processes, so you might need to store their PID in some place to be able to kill them later, etc.
Now, there's a caveat, Prodigy has a very rich and dynamic command line and it's intended to be used from the command line. So, if you run it from inside something else with some fixed parameters, that will limit what you can do with it to those fixed parameters.
So even if you build tools around to control it from a web server, it might end up being problematic and you could miss functionality.
Now, although it doesn't solve this use case, we are currently working hard to release Prodigy Teams, which has all that setup to control Prodigy with a team of users from a web UI, etc. If that's what you want to achieve, you might want to sign up to be notified when it's ready.