I need to tag the same items to different classes like:
Sentiment: positive, aggresive, neutral, massage, negative
Context: Tech, News, Blog
How can I do it with prodi.gy to not tag the same item 2 times?
Thanks
Hi! Since this is a text classification task, you could just annotate (and also train) this as a flat multi-label scheme and annotate all categories together. So you'd have labels like POSITIVE etc. and CONTEXT_TECH (or however you want to call the labels). You probably also want to have a label like CONTEXT_OTHER for cases that don't fit into any of the categories.
If you want to ensure that the annotator picks at least one sentiment and one context label, you can use the validate_answer callback to check each annotation and raise an error if only one of the class types was selected (or none, or too many).
Alternatively, you can also solve this by making two passes over the data and annotate the class types separately. An advantage of this approach is that it'll let you run separate experiments with the different label schemes and evaluate them independently. For example, it might turn out that your context labels are harder to predict and you want to make adjustments to the categories, while keeping the sentiment labels the same. Or maybe you want to compare two different sentiment scales and see which one works best. You can always train on multiple datasets and different combinations of datasets, but if you do it all jointly, it can often make it more difficult to iterate.
1 Like