I am trying to deploy the prodigy instance in aws using kubernetes.
It now uses sqlite database.
It works very fine with the image created when I run it with docker run ...
But, while deploying with kube, I get this in the logs:
Added dataset dataset to database SQLite.
Traceback (most recent call last):
File "__main__.py", line 333, in <module>
server(controller, controller.config)
File "/app.py", line 61, in server
threads=1,
File "/usr/local/lib/python3.7/site-packages/waitress/server.py", line 49, in create_server
adj = Adjustments(**kw)
File "/usr/local/lib/python3.7/site-packages/waitress/adjustments.py", line 295, in __init__
setattr(self, k, self._param_map[k](v))
ValueError: invalid literal for int() with base 10: 'tcp://172.20.1.208:8080'
I am trying to figure out, what could this be? Any help would be appreciated.
I don’t have much experience with Kubernetes, but two things come to mind here:
If you use the sqlite backend, make sure you’re placing the database in a persistent volume. My guess is that if you used the default location (in the home directory), that state won’t be persistent, so you’ll lose data. I would probably switch to a different backend — I think sqlite is likely to be a poor fit for Kubernetes.
It looks to me like you’ve got some setting that expects an integer receiving the whole connection string. Perhaps you’re passing the connection string to an argument that expects only the port?
Taking a little step back here, I’m guessing that you want to have a setup where you run some command and Kubernetes launches a new Prodigy task, and you get the URL of the task, right? This workflow requires a couple of steps of indirection.
If you just launch Prodigy tasks one-by-one on your laptop, it listens on localhost and you can point your browser to localhost. But if you’re launching tasks on remote machines, you probably want a reverse proxy, which should map the localhost URLs to something you can access. And then if you’re also starting and stopping tasks under automation, you probably also want something that will keep track of all the Prodigy tasks, allocate them names, get the ports they’re listening on, and organise the mapping for your reverse proxy.
For Prodigy Scale, we’re using Nomad to launch the Prodigy tasks, with consul for service discovery. We then use Traefik as the reverse proxy, which has a neat integration with consul’s service catalog.
Also, for each task allocation, kubernetes handles the port and the host for the prodigy application itself as we are supplying those information in the deployment and service files. May be it has something to do with the reverse proxy.
And, with the integer expectation, how can it be that the same code base and setting works perfectly when I run it manually or the docker image in my local PC, but not in kubernetes. And, also we are using PostgreSQL as a db.
Though, I will look for more here, and update here if I find the solution
How are you informing Prodigy of the port and host? Are you using the prodigy.json, or are you setting environment variables?
I’m guessing environment variables, right? Because otherwise it would seem to me you’d need to rebuild the container image every time just to change that file?
Edit: Ah wait…You possibly don’t need to customize this in Prodigy at all? I guess Prodigy can always listen to the same port within the container…
