Run prodigy in Lambda AWS


Is it possible to run prodigy in Lambda AWS? What's better approach to deploy on Lambda AWS?

I would like to execute some functions of prodigy in my python code.


Hi Tiago,

We haven't run Prodigy on lambda ourselves, so it's not officially "supported" in that sense, and I'm not sure what problems might occur.

In terms of the annotation service, lambda is definitely not a great fit, because Prodigy recipes often assume state within the process (for instance, the queue of data is a Python generator, and if you're using active learning, the weights change). You can implement a recipe that doesn't make these assumptions, for instance you could pull your examples from a Redis queue so that the state within the process is irrelevant.

In terms of the non-annotation tasks, such as training, these are long-running processes so lambda wouldn't be a great fit.

It would be possible to make a custom API that called into Prodigy for some more specific stats or metrics or something, but my suggestion would be to implement the same without Prodigy in a more minimal way. For instance, if you just want some query over the database, you might find it useful that the file is distributed within Prodigy. You could extract it and run what you need without having to depend on the actual library.