EC2 instance was recently moved from public to private subnet. Since then, the prodigy service is broken.
Login mechanism - login to ec2, port forwarded to port 8080, host the job using the prodigy command, hosting is successful.
When i open the browser and send a request to localhost:8080, it opens the prodigy UI but the content isn't loaded and a message appears - "ERROR:Can't fetch project. Make sure the server is running correctly." Upon checking the browser's developer console - I saw a 403 response from localhost:8080/project.
I am attaching the relevant screenshots. Can someone help here and share any possible fixes or resolutions?
Thanks for your question and welcome to the Prodigy community
Could you check your Network tab in the browser? There might be more info on why the request failed, rather than just the console line.
Typically, a 403 should only happen if there's a JWT error. Have you modified any JWT settings?
If not, one possibility is that JWT cookie can't be reached due to something CORS related, then the token isn't set on the request AUTHORIZATION header so the 403 appears.
But if it were a CORS problem, we'd expect to see a CORS warning and then the failed request.
To test if this is the problem, can you try setting "cors": false in prodigy.json. It's not really recommended to turn that off but they could determine if it is a CORS issue through that test. If this does seem to be the issue, you can set the PRODIGY_CORS_ORIGIN env var with the proper URLS for CORS.
One other possibility is to try to modify your "host" in prodigy.json from "localhost" to either "127.0.0.1" or "0.0.0.0". However, this is usually a fix for a different, but somewhat related error: OSError: [Errno 99] Cannot assign requested address:
Last, another user had a somewhat similar 403 issue with the same Prodigy error message ERROR:Can't fetch project .... Seemed like they forgot to pass a session_id which would be required if you set PRODIGY_ALLOWED_SESSIONS in your prodigy.json. Any chance this could be the problem?
The /project route is a simple handler with two opportunities to return a 403:
@routes.get("/project")
def get_project():
log("GET: /project", get_config())
assert_known_session(None)
config = {}
for key, value in get_config().items():
if srsly.is_json_serializable(value):
config[key] = value
return config
The assert_known_session checks the PRODIGY_ALLOWED_SESSIONS env variable, if this is set and a valid sessionId is not provided matching one of these allowed sessions, you get a 403
If JWT is manually turned on (e.g. with the ENV VAR PRODIGY_JWT=1) and Prodigy can't find the JWT token in the browsers cookies (it's there by default) then you get a 403.
There is not another opportunity in our code to return 403.
At one point FastAPI raised a 403 for HTTP Basic Auth but that likely changed at some point (not sure when).
What version of Prodigy are you using? You can provide the print out of python -m prodigy stats.
Can you also share the message you're seeing by clicking the failing request? It could give a hint.
Prodigy sends some messages back with 403 errors so the response would have an extra hint.
Version 1.11.8
Location /home/ubuntu/miniconda3/envs/prodigy_env/lib/python3.10/site-packages/prodigy
Prodigy Home /home/ubuntu/.prodigy
Platform Linux-4.4.0-1128-aws-x86_64-with-glibc2.23
Python Version 3.10.4
Database Name SQLite
Database Id sqlite
Total Datasets 93
Total Sessions 538
Attached below is the message which I get by clicking the failing request.
Ah - the error is "a registered session id is required for this task".
Yep, that's telling you the problem. You're trying to use named multi-user sessions but not naming your session. It is the same problem that I mentioned earlier that another user had solved.
You need to add ?session=[name your session] to your URL. You could also turn off your config settings (e.g., go back to default of feed_overlap: false, remove PRODIGY_ALLOWED_SESSIONS, etc.) and it should work.
Prodigy Stats ==============================