Prodigy visualizations inside a web application

Is it possible to use Prodigy visualizations inside a web application, much like how displaCy can be used in a web application?
The difference between using displaCy in a web application and using a Prodigy visualizer in a web app is that a Prodigy visualizer could enable active learning web applications that would add a lot of value for ML practitioners and businesses.

Hi! While I'm sure there would be applications that could use it, we're not planning to unbundle Prodigy's front-end and back-end at this time. We plan to continue developing it as an integrated product. So the best way to build active learning applications for ML with Prodigy's UI components is by... using Prodigy :smiley:

Hi Ines! I'm trying to figure out how to do this in Prodigy, as you suggested, but I'm struggling to figure it out. Prodigy is the best way I have found to annotate data for creating an NLP algorithm. After I develop a model, I typically prototype an application for a group of business users to consume the output of my NLP algorithm in their day-to-day work, trained on my Prodigy annotated data. Unfortunately, I don't see a way to use Prodigy to enable business users to both consume and write back corrections using Prodigy.

The closest thing I have found for this is "JavaScript and CSS" in the "Custom Interfaces" section, but it lacks the ability to create custom buttons, checkboxes, or drop-down lists that are necessary to enable consumption/filtering of the data at the same time as writing back annotations similar to, for example, ner.correct back to a database. I think I could accomplish my goal if custom buttons, custom checkboxes, custom drop-down lists, etc. were available in Prodigy, much like Streamlit's "Display interactive widgets".

The problem I am having is that it is difficult for me to collect data about whether the results of these algorithms match the expectations of large groups of users. The best solution I have found is for me to build in a button into my prototype application for the user to indicate that an algorithm output is incorrect, but it would be so much more valuable for the users to correct the output themselves using the amazing active learning capabilities provided by Prodigy.

Is it possible to use Prodigy to prototype an app for end-user consumption and for NLP model output correction at the same time? If not, would you happen to know of other packages I could use (preferably in Python, but not necessary)?

Ah, the custom JS/CSS/HTML features were specifically designed to allow custom UI components like buttons, checkboxes etc. If you want a button that does something to the current data, you can just add a <button> and call into window.prodigy.update to update the current annotation task. There are a few threads on the forum on implementing custom dropdowns, checkboxes etc. (like this one).

One thing that doesn't really make sense in the context of Prodigy is to have the annotator choose or filter the data in the UI. If your goal is to train a model, you typically want control over what is annotated and what will be used as training data – that can't really be decoupled from the development process, and you can't really expect the annotator to make a meaningful decision for you.

The active learning capabilities exclusively happen on the server, though, and don't really have anything to do with the UI components. The idea is to use the model in the loop to decide what to present for annotation – and for that, you ideally want a focused annotation session that collects enough annotations to allow the model to be updated in the loop and make meaningful suggestions. That wouldn't really work if you only collect very sporadic feedback.

Also, it's probably quite rare that the output the model you're training produces is exactly what you'll be showcasing to the business users – often, you'll do something with the predictions, and what the end user sees is the result of that. So the feedback you may be collecting in your application may not be feedback on the direct model output.

So if you want to implement a tighter feedback loop and allow your users to make corrections to the model's predictions, one solution would be to set up a dedicated annotation session for it and stream in selected examples that need attention from domain experts etc. Those could be predictions with very uncertain scores (e.g. close to 0.5 where the answer could go either way), or predictions that caused mistakes somewhere in your downstream application. This also gives you a more measurable evaluation, lets you calculate agreement (do end users even agree on the problems?) and actually create data you can update the model with to improve it.

1 Like