Sentiment analysis - annotating different aspects

, , ,
1

Hi,
we are currently looking into the subject sentiment analysis and I already read this post https://support.prodi.gy/t/best-practice-for-aspect-level-sentiment-analysis/856, which was helpful for the beginning. But I still have some questions concerning the workflow with prodigy for sentiment analysis:
How can I possibly handle the following sentence with prodigy’s text classification?

  • The team is friendly, the doctor is competent and I never have to wait longer than 10 minutes.

With my target topics/aspects: friendliness, competence and waiting time.

Using textcat.teach, prodigy shows me a sentence with randomly one of the three mentioned aspect categories, which I would have to reject most of the time because there is actually another category present than the one proposed (or none at all).

So first: Is there a possibility that I get a sentence and then choose one of my categories myself and then accept the sentence?
Second: Can I annotate a single text snippet shown by prodigy with multiple aspects, e.g. here friendliness, competence and waiting time?

Also: Is it possible to annotate a sentence with aspect and sentiment in one go, e.g. friendliness + positive? If so, how exactly?

Thanks in advance!

2

Maybe a custom recipe with the "choice" interface could be a solution here? If you set "choice_style": "multiple" in your config, the options will become multiple choice, so you can annotate them all at the same time if you like. You can find an example of a similar task and script in the custom recipes workflow. The result looks like this:

41

3

Hi Ines, thank you for your reply.

The “choice” option sounds like a reasonable idea. But following this, I would then still be unable to annotate one sentence with multiple aspects, right? However, we want to do aspect-based sentiment analysis and not sentence-based SA. In other words, if there are two sentiments about a different category each present in a single sentence (friendliness AND competence), we want to annotate both. And this seems impossible using textcat.teach.

That’s why we were thinking about using the NER mode instead in order to be able to annotate multiple chunks in a sentence with a certain aspect/sentiment.
Then I would have to annotate the aspect-sentiment combinations
friendliness-pos, competence-pos, waitingtime-pos, friendliness-neg, competence-neg and waitingtime-neg (plus actually the neutral counterparts friendliness-neutral, competence-neutral and waitingtime-neutral).

Problematic might then be that we have so many labels to choose from… But I don’t see a work-around that, because it’s not possible to annotate one entity with two labels (e.g. “friendliness” and “positive”). Or is it with some added recipe?

Have you every heard about or used this approach for aspect-based sentiment analysis, or do you have any other tips and hints?

4

You could use the "choice_style": "multiple" setting to make the options multiple choice and then select both options? Or am I misunderstanding the question? The annotated data you get back from the recipe could then look something like this:

{
    "text": "Some text",
    "options": [...],
    "accept": ["FRIENDLINESS", "COMPETENCE"]
}

You can then convert this data and use it to train spaCy's text classifier. spaCy's built-in text classifier supports mutually exclusive categories, so if you train it from that data, it can predict things like {'FRIENDLINESS': 0.94, 'COMPETENCE': 0.72, 'WAITINGTIME': 0.39}.

For training your classifier, do you actually need the spans of the text the labels are based on? Or are you only looking for labels on the whole text? If you're only trying to predict labels that apply to the whole text, labelling the exact spans that express that sentiment seems like a waste of time.

5

Setting the choice_style to “multiple” in the configuration worked! Thank you for that hint :slight_smile:

1 Like