Test custom recipes via REST API

, ,
1

Hi, we have a few custom recipes for our annotation tasks and we'd like to add tests for them. Do you have a recommended approach for testing recipes? I would like to follow the FastAPI testing described here, if possible. https://fastapi.tiangolo.com/tutorial/testing/

We also would like to use the DB injection described here to use sqlite in memory DB for tests. https://fastapi.tiangolo.com/advanced/testing-database/

We've had some issues with duplicate data and samples re-appearing after re-starting the service and we'd like to avoid issues like this via tests. The tests would also allow us to verify correct processing inside the recipe.

Cheers, Juha

2

Hi! This is definitely a good plan (yay for testing! :innocent:) If you just want to test the recipe itself and receive/send data, you shouldn't even need to go via FastAPI, unless you want to test the server directly. Instead, you can also let Prodigy create a Controller instance based on your recipe and then interact with it, which is what Prodigy's REST API does under the hood:

from prodigy import get_recipe

# Get any recipe that's available (requires the @recipe decorator to run)
recipe_proxy = get_recipe("ner.manual")
# This takes all recipe arguments as positional arguments
controller = recipe_proxy("test_dataset", "en_core_web_sm", "data.jsonl")
# You can now call controller methods and request/send examples
questions = controller.get_questions()
answers = [{**question, "answer": "accept"} for question in questions]
controller.give_answers(answers)
print(controller.total_annotated)

If you want to use an SQLite database in memory, you can return the following as "db" from your custom recipe:

from prodigy.components.db import connect
db = connect("sqlite", name=":memory:")

Alternatively, you can also register a custom DB like "test_db" – this would let you refer to "db": "test_db" in your prodigy.json. During training, you could point Prodigy to a different PRODIGY_HOME or use a training-specific prodigy.json in the current working directory.

from prodigy.components.db import connect
from prodigy.util import registry

db = connect("sqlite", name=":memory:")
registry.databases("test_db", func=db)

If you want to test Prodigy's FastAPI-powered REST API specifically, you can export it as app from prodigy.app. Note that it depends on global state, though, so you'll have to call prodigy.app.set_controller to define (this is currently internals, but you can see how it works by looking at the app.py in the source).

from prodigy.app import app, set_controller

# An instance of Controller and your Prodigy config (prodigy.json)
set_controller(controller, config)
# Do something with the app here..

Then you should be able to test it like any other FastAPI app. Maybe @tiangolo has something to add about FastAPI specifics, I'm not sure :wink:

1 Like
3

Not really, I don't see anything else to add :nerd_face:

And it's probably not necessary to add tests for Prodigy's internal FastAPI app as we are already testing it, you should only need to add tests for your custom recipe.

...Unless you are extending the internal FastAPI app, or integrating it into another app or something similar, in that case, it would make sense to add a couple of tests just to make sure that the integration is working, but not much more, we are already testing the app so you don't have to :smile: :white_check_mark:

4

Great, thank you very much. I think tests using the controller should be sufficient for us. We are happy FastAPI users in our other services so great to see @tiangolo joining Explosion.

Cheers, Juha

1 Like