I know Prodigy doesn't support entitiy-level sentiment annotation out of the box, so I'm trying to figure out a simple way to implement this. Since the set of entities I want to annotate is a known closed list, I can mark the entity spans with high accuracy and pre-label them. The idea then would be to annotate those entities with sentiment classes (pos, neg, neu) disguised as NER labels using the standard ner.manual recipe. Does this make any sense? Would I get reasonable prediction results? What other alternatives are there?
There are different ways you can set up this type of annotation task in Prodigy. If you can pre-label the entities with high accuracy (both during annotation and at runtime), another solution would be to stream in examples with one entity at a time and add a choice block to annotate sentiment. So for each entity, you're deciding whether it's positive, negative or neutral.
I think this is the more interesting and relevant question: if you're modelling your task as an NER prediction problem with 3 classes per entity type, you might be making your life much much harder than it should be. Your model will have to learn 3 versions of all labels separately and will need a lot of examples, and it'll have to predict a bunch of stuff you already know: the span boundaries and the top-level label. The standard context window of an entity recognition model may also not be enough to capture the context relevant for the sentiment prediction.
Depending on the types of texts and the entities you're expecting, one alternative approach would be to model the sentiment as a text classification layer. This lets you take the whole text (e.g. sentence) into account and it might be enough to solve your problem with sufficient accuracy. Of course, you could also consider implementing a custom model and use your entity list for candidate selection. You'd hen be predicting whether one of the 3 sentiment labels applies to alabelled span given the text.
Thanks for the prompt reply, Ines. I have created a multi-block recipe and feed a pre-labeled example for each entity present in the the text. Good, simple solution.
1 Like