MongoDB to Store Annotations

[jonathankingfc (Jonathan King)]

Hi,

I would like to use MongoDB to store my annotations from Prodigy. The product I am developing relies on a Mongo Docker image that I have been using for my other components and I would like to keep everything on the same DB. Has anyone had experience setting up Prodigy with Mongo? I know that one can easily configure their own SQL database with Prodigy but I am having trouble finding instructions on how to set it up with Mongo.

Thanks!!

[ines (Ines Montani)]

Hi! I don't remember anyone sharing an example with MongoDB, but you might find this thread helpful for some more general tips for implementing a custom Database class:

Using Google Firestore as database or not

I have all my documents in Google Firestore so I stream my training examples from Firestore into prodigy app. I’ll annotate the data without any preprocessing but I’ll do some preprocessing when training - this might change over time though so the saved text field by prodigy might need to be updated over time. FYI I train the model in the loop like .teach recipe on the preprocessed data. I figured its easier for me to keep all the annotated data in Firestore as well but it is not exactly clear …

Ultimately, your Database class just needs to provide the same methods as Prodigy's built in Database class (you can find the API specs in your PRODIGY_README.html). How you structure things internally and how you communicate with MongoDB is up to you. For instance, the Database.get_dataset method takes the string name of a dataset and should return a list of dictionaries of the annotations in the given set. That's all Prodigy cares about – so if you can construct a query that gets you that data, everything should work as expected.

The source of the database models is also included with Prodigy, so you can take a look at it for some inspiration.

[jdagdelen (John Dagdelen)]

If anyone in the future also needs this functionality, I have written a custom data loader and database for using MongoDB with Prodigy. They are available at GitHub - jdagdelen/mondigy: A small component for using a Mongodb database as a data loader for Prodigy annotation applications..