entity labeling

[ines (Ines Montani)]

Thanks for your questions!

@jeweinb

Is there currently a solution that will allow users to manually highlight entities and tag them with a label? That is our current workflow, and it is important because physicians want to highlight the exact item they are looking for. [...] The users want to be able to read an entire clinical note all at once, highlight/tag the entities they are looking for and then move on to the next note.

For the first release, we've mostly been focusing on the capabilities of Prodigy as a developer tool and reengineering traditional annotation processes to help developers iterate on the data and run experiments faster. This means that the current workflows aim at making the annotator do as little as possible and using the interface to focus on one decision at a time to move as quickly as possible. (You can see our latest NER video tutorial for an example of a development workflow like this. It also shows the use of word vectors and terminology lists to pre-label entities.)

I totally understand your process, though – in some cases, it definitely makes sense to work through an entire document manually and at once, and label everything that needs to be labelled. This is currently not possible in Prodigy. However, we are working on new interfaces for those types of use cases, including text, images (object detection and segmentation), as well as potentially audio files.

@jeweinb

Also, is it possible for Prodigy to take a directory of text files as annotation data and return the annotated text to the database in gold parse json format?

I'm not sure I understand the question correctly – do you mean importing raw text data to the database, but from a directory? Currently, prodigy db-in only works for single files. But you can easily process a whole directory using a simple shell script, or run the function from Python:

from prodigy.__main__ import db_in
for filename in directory:
    db_in('my_dataset', filename)

@jeweinb

Is there also a road map for managing annotations among a group of annotators with an adjudication process? Ideally the annotators would work on their own portion of notes, but we want to randomly share small batches of notes among all annotators so we can check concordance.

Yes, this is exactly what we had in mind for the Prodigy Annotation Manager. We're currently planning this as a Prodigy add-on, i.e. a separate package you can plug into your Prodigy workflow and that extends the app with more functionality and an annotation management console that lets you orchestrate larger annotation projects, handle quality control etc. We don't have a timeline for this yet, but it's definitely something we've been thinking about a lot, and have been experimenting with.