simulating prodigy user

[ryanwesslen]

hi @FarisHijazi!

Thanks for your message and welcome to the Prodigy community

Interesting project!

@FarisHijazi

I'm trying to run a grid search on an active learning pipeline, and I want to do this automatically without actually having a human in the loop, I need to somehow be able to simulate prodigy and connect to it's database without launching the server.

Curious - is your goal to run experiments to determine the best strategies on using active learning for improved accuracy? Interesting stuff! I haven't seen Robert's code but I would be interested to learn more. I can get back to you later.

One interesting thing for "simulating" active learning is to also add noise to the annotations (i.e., purposely make x% of annotations incorrect). If you do a simulation where you assume the annotator is correct each time, this isn't reflective on the reality that annotators make mistakes. I've devised AL experiments in the past and found an important "hyperparameter" is the assumed accuracy of the annotators. Just another factor to consider in your experiments.

@FarisHijazi

Here's the list of what I need to do:

• get access current annotations
• get all previous annotations (I can do this by running the db-out and then parsing the output string in python), but I'm sure there's a better way
• write current session annotations to prodigy db (only needed for the simulation to simulate a user labeling)

Have you heard/used Prodigy's entry points?

Entry points let you expose parts of a Python package you write to other Python packages. This lets one application easily customize the behavior of another, by exposing an entry point in its setup.py or setup.cfg. For a quick and fun intro to entry points in Python, check out this excellent blog post. Prodigy can load custom function from several different entry points, for example custom recipe functions. To see this in action, check out the sense2vec package, which provides several custom Prodigy recipes. The recipes are registered automatically if you install the package in the same environment as Prodigy. The following entry point groups are supported:

prodigy_recipes	Entry points for recipe functions.
prodigy_db	Entry points for custom Database classes.
prodigy_loaders	Entry points for custom loader functions.

I haven't used these yet but they may do the trick. Here's where they were used in sense2vec.

Let me think more in general -- I may have some suggestions.

In the meantime, I found a relevant post (which you may have read already):

Is it possible for me to control the entire active learning loop?

This is not necessarily true. Most use cases definitely involve loading data from a file or a single source, because that’s the most common way people go about annotating their data. But in the end, the stream is just a Python generator that Prodigy keeps requesting batches of tasks from. How that batch is composed is up to you – so you could easily implement your own logic that takes previous user decisions into account, randomly adds data from different sources or uses other factors to determ…