# entity labeling **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner\ **Created:** [December 19, 2017, 9:48pm UTC](https://support.prodi.gy/t/entity-labeling/144 "2017-12-19T21:48:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jeweinb](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@jeweinb](https://support.prodi.gy/u/jeweinb)\ **Post date:** [December 19, 2017, 9:48pm UTC](https://support.prodi.gy/t/entity-labeling/144/1 "2017-12-19T21:48:47Z")
Is there currently a solution that will allow users to manually highlight entities and tag them with a label? That is our current workflow, and it is important because physicians want to highlight the exact item they are looking for. Also, is it possible for Prodigy to take a directory of text files as annotation data and return the annotated text to the database in gold parse json format? The users want to be able to read an entire clinical note all at once, highlight/tag the entities they are looking for and then move on to the next note. They don’t want to read sentence by sentence rejecting or accepting matched phrases. Is there also a road map for managing annotations among a group of annotators with an adjudication process? Ideally the annotators would work on their own portion of notes, but we want to randomly share small batches of notes among all annotators so we can check concordance. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [December 20, 2017, 1:25pm UTC](https://support.prodi.gy/t/entity-labeling/144/2 "2017-12-20T13:25:55Z")
Thanks for your questions! > [@jeweinb](#): > > Is there currently a solution that will allow users to manually highlight entities and tag them with a label? That is our current workflow, and it is important because physicians want to highlight the exact item they are looking for. [...] The users want to be able to read an entire clinical note all at once, highlight/tag the entities they are looking for and then move on to the next note. For the first release, we've mostly been focusing on the capabilities of Prodigy as a developer tool and reengineering traditional annotation processes to help developers iterate on the data and run experiments faster. This means that the current workflows aim at making the annotator do _as little as possible_ and using the interface to focus on one decision at a time to move as quickly as possible. (You can see [our latest NER video tutorial](https://support.prodi.gy/t/video-training-a-new-entity-type-with-prodigy/140) for an example of a development workflow like this. It also shows the use of word vectors and terminology lists to pre-label entities.) I totally understand your process, though – in some cases, it definitely makes sense to work through an entire document manually and at once, and label everything that needs to be labelled. This is currently not possible in Prodigy. However, we are working on new interfaces for those types of use cases, including text, images (object detection and segmentation), as well as potentially audio files. > [@jeweinb](#): > > Also, is it possible for Prodigy to take a directory of text files as annotation data and return the annotated text to the database in gold parse json format? I'm not sure I understand the question correctly – do you mean importing raw text data to the database, but from a directory? Currently, `prodigy db-in` only works for single files. But you can easily process a whole directory using a simple shell script, or run the function from Python: ```python from prodigy. __main__ import db_in for filename in directory: db_in('my_dataset', filename) ``` > [@jeweinb](#): > > Is there also a road map for managing annotations among a group of annotators with an adjudication process? Ideally the annotators would work on their own portion of notes, but we want to randomly share small batches of notes among all annotators so we can check concordance. Yes, this is exactly what we had in mind for the Prodigy Annotation Manager. We're currently planning this as a Prodigy add-on, i.e. a separate package you can plug into your Prodigy workflow and that extends the app with more functionality and an annotation management console that lets you orchestrate larger annotation projects, handle quality control etc. We don't have a timeline for this yet, but it's definitely something we've been thinking about a lot, and have been experimenting with. ---
**Author:** ![jeweinb](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@jeweinb](https://support.prodi.gy/u/jeweinb)\ **Post date:** [January 17, 2018, 6:27pm UTC](https://support.prodi.gy/t/entity-labeling/144/3 "2018-01-17T18:27:13Z")
Thank you Ines! This is helpful. We are in the market for an annotation tool right now. I think prodigy is great for a developer to create a model with limited availability of expert annotators. You may be aware, but physicians are a bit picky about workflows and they wanted it to work similar to past tools. I will share what you mentioned with my manager. I like the progress of the entire explosion AI ecosystem so far. You are creating something really cool here! ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 18, 2018, 8:57am UTC](https://support.prodi.gy/t/entity-labeling/144/4 "2018-01-18T08:57:27Z")
@jeweinb Thanks! And yes, I definitely understand that. Ultimately, the underlying idea behind Prodigy is that you’ll be able to build workflows that require less manual work, and let you use the annotators’ time more efficiently. This is especially relevant in cases like yours, where you’re relying on domain experts to do the labelling. You might still need _some number_ of manual annotations – but by testing out different approaches upfront and annotating with a model in the loop, you’ll eventually be able to focus on having the domain expert annotators (in your case, physicians) work _with_ the model and give it feedback on the edge cases that are most relevant for training, i.e. the predictions the model is most uncertain about. Because the annotators will only have to click accept or reject, they’ll be able to annotate more examples (or you’ll need to ask for less of their time). And because the model in the loop is updated at runtime, it’ll be able to ask better questions as it learns from the feedback. That said, we actually shipped an `ner_manual` interface in the latest update to Prodigy. You can check it out [in the live demo here](https://prodi.gy/demo?view_id=ner_manual). We’re also planning a number of improvements in the upcoming version – for example, you’ll also be able to pre-define spans to be highlighted, that the user can then accept, delete or modify.