Annotation for Argument Mining

, ,
1

I’m reading some papers related to Argument Mining, and there is a considerable amount of papers handling AM as a sequence tagging and/or relation extraction problem, such as http://www.aclweb.org/anthology/P/P17/P17-1002.pdf and http://www.aclweb.org/anthology/P16-1105. In their datasets, they annotate long sequences of texts and pass them over to models based on LSTM. These annotations not only contain the information of what type of tag the token is, but also to which other sequence of tokens it would be related.

The annotation below is an example of a persuasive essay annotated with the following types of entities: Major Claim, Claim and Premise. The Claims can either be for or against Major Claims, and Premises can either support or attack claims. “For”, “Against”, “Support” and “Attack” are annotated as relations between premises and claims.

I’m interested in knowing if it’s possible to produce an annotated corpus such as the one below, using prodigy. I was going through other threads mentioning future support for relation extraction, and there were some temporary suggestions that I think are no longer supported, such as using ner.mark (which became only mark, apparently). It would be necessary to annotated longer spans of texts and link them. As an example, B-Premise:111:Support below means that the premise supports the claim span starting at line 111. This would definetely have to be a manual annotation, couldn’t be something suggested by the tool, to be approved/rejected by the annotator.

|86|From|O|
|87|this|O|
|88|point|O|
|89|of|O|
|90|view|O|
|91|,|O|
|92|I|O|
|93|firmly|O|
|94|believe|O|
|95|that|O|
|96|we|B-MajorClaim|
|97|should|I-MajorClaim|
|98|attach|I-MajorClaim|
|99|more|I-MajorClaim|
|100|importance|I-MajorClaim|
|101|to|I-MajorClaim|
|102|cooperation|I-MajorClaim|
|103|during|I-MajorClaim|
|104|primary|I-MajorClaim|
|105|education|I-MajorClaim|
|106|.|O|
|107|First|O|
|108|of|O|
|109|all|O|
|110|,|O|
|111|through|B-Claim:For|
|112|cooperation|I-Claim:For|
|113|,|I-Claim:For|
|114|children|I-Claim:For|
|115|can|I-Claim:For|
|116|learn|I-Claim:For|
|117|about|I-Claim:For|
|118|interpersonal|I-Claim:For|
|119|skills|I-Claim:For|
|120|which|I-Claim:For|
|121|are|I-Claim:For|
|122|significant|I-Claim:For|
|123|in|I-Claim:For|
|124|the|I-Claim:For|
|125|future|I-Claim:For|
|126|life|I-Claim:For|
|127|of|I-Claim:For|
|128|all|I-Claim:For|
|129|students|I-Claim:For|
|130|.|O|
|131|What|B-Premise:111:Support|
|132|we|I-Premise:111:Support|
|133|acquired|I-Premise:111:Support|
|134|from|I-Premise:111:Support|
|135|team|I-Premise:111:Support|
|136|work|I-Premise:111:Support|
|137|is|I-Premise:111:Support|
|138|not|I-Premise:111:Support|
|139|only|I-Premise:111:Support|
|140|how|I-Premise:111:Support|
|141|to|I-Premise:111:Support|
|142|achieve|I-Premise:111:Support|
|143|the|I-Premise:111:Support|
|144|same|I-Premise:111:Support|
|145|goal|I-Premise:111:Support|
|146|with|I-Premise:111:Support|
|147|others|I-Premise:111:Support|
|148|but|I-Premise:111:Support|
|149|more|I-Premise:111:Support|
|150|importantly|I-Premise:111:Support|
|151|,|I-Premise:111:Support|
|152|how|I-Premise:111:Support|
|153|to|I-Premise:111:Support|
|154|get|I-Premise:111:Support|
|155|along|I-Premise:111:Support|
|156|with|I-Premise:111:Support|
|157|others|I-Premise:111:Support|
|158|.|O|

Thanks!

2

The short answer is: yes, you could annotate this sort of thing with Prodigy. I would suggest annotating this task at the sentence level first though, and only making sub-sentence annotations in a later pass. I suspect you’ll have a lot of trouble coming up with an annotation policy for sub-sentence annotations that’s easy to enforce consistently. I would strongly encourage you to align the sub-sentence annotations to syntactic constituents, to help make this more consistent. Otherwise, you’re always going to have boundary errors, which will cause a sequence tagging model a lot of problems.

I must admit I’m also a bit skeptical that the methods in these papers are really ready for production use. The corpus they’re testing on is student essays, which tend to be taught to a fairly rigid formula. This likely makes the task quite easy. In school I always carefully started each paragraph with a Topic Sentence, ended each paragraph with a conclusion, introduced my 3 or 4 claims in the introduction, and restated them in the conclusion. You wouldn’t even need to read my sentences to find the argument structure.

I would focus mostly on finding annotation schemes that can be efficiently and accurately annotated at the sentence and word level. You can then use the dependency parse, entity labels and your auxiliary annotations to create a rule-based extraction system. Once this extraction system is running, you can accept/reject its outputs to quickly create a large set of corrected labels. Finally, once you have this suitably sized corpus, you could experiment with end-to-end systems such as these LSTM approaches, and compare them against your multi-step baseline.

1 Like
3

How exactly? I understood from @ines other posts that this is still a work in progress. If it's possible to do it now, could you indicate which recipes I could use?

I'm actually just studying how they do it with this essay scenario, but I'm actually going to apply it to another scenario, extracting arguments from legal documents containing court decisions, such as is done by https://nightowl.jp/publication/2017/9/8-argmining201703. I'm considering the essay's annotation scheme which tackles the problem as a relation extraction problem.

Definetely, I believe this is why they keep two different scores: one which measures 100% match and one that measures 50% match, with the latter being much more lenient.

Do you think it makes sense to annotate in sentence or sub-sentence levels in this scenario? I believe they annotate in essay and paragraph levels because the arguments and claims linked are hardly in the same sentence.

I'm not sure I understand what you said here. Can you give me an example?

Thanks!

4

The first posts on here discussing the topic were all written before the ner_manual and dep interfaces existed. You'd probably want to use a custom recipe without a model in the loop and start off by highlighting the phrases using the ner_manual interface – for example, MAJOR_CLAIM, PREMISE etc.

When you're done with that, you could export the data and run one experiment where you link up highlighted spans close to each other and collect binary feedback on whether they are connected. You could either create data in the dep format (with a head and child), or use the choice interface with the options “For”, “Against”, “Support” and “Attack”.

Alternatively, you could also use more labels in the ner_manual interface and do two things at once: annotate the claims and the “For”, “Against”, “Support” and “Attack” phrases. To help with visual indicators, you could even create a custom label color scheme (red for negative, green for positive, or something like that).

5

Great, thanks @ines! That sounds reasonable.

6

Hi @ines,

Considering I already have a list of annotated texts and their spans, such as the following:

MajorClaim 503 575	we should attach more importance to cooperation during primary education
MajorClaim 2154 2231	a more cooperative attitudes towards life is more profitable in one's success
Claim 591 714	through cooperation, children can learn about interpersonal skills which are significant in the future life of all students
Premise 716 851	What we acquired from team work is not only how to achieve the same goal with others but more importantly, how to get along with others

With the numbers being the coordinates of the text, how can I convert these annotations into a dataset in prodigy’s format? So I can skip the ner_manual step and proceed to the relation annotations?

Thanks!

7

I think the easiest way would be to create one big annotation task for each text and add all spans to the task’s "spans". If you have the character offsets (i.e. the start and end index) and the label, each span would look something like this:

{"start": 503, "end": 575, "label": "MajorClaim"}

Here’s a pseudocode example of the conversion:

examples = []
for text, span_rows in your_data:
    spans = []
    # let's assume you iterate over the rows of 4 columns each
    for label, start, end, span_text in span_rows:
        spans.append({'start': start, 'end': end, 'label': label, 'text': span_text})
    examples.append({'text': text, 'spans': spans})

If the texts are too long and you want to split them into multiple examples, you can use spaCy to segment the text (into sentences, multiple sentences etc., really depends on the length). If you process the example text with spaCy, you get a Doc object. Each slice of a Doc object (a Span) has a start_char and end_char. So you can use this to calculate the relative offset of the entity spans within the slice. For example, if the sliced example text starts at character 500 in the original text, and the span starts at character 550 and ends at 560, the new "start" would be 50 and the new "end" would be 60.

1 Like
8

Ok, so I don’t need the ‘token_end’ and ‘token_start’ along with the token_ids used in ner.manual annotation, right? I can just produce jsons without these and give them to db.add_examples(), right?

Thanks!

9

Yes, if you don’t plan to actually load these examples into a manual annotation interface and only want to train a model, you won’t need the token mapping on the spans. And even if you do, the add_tokens function can take care of this and resolve existing spans to tokens. (Assuming spaCy’s tokenization matches the spans – which is usually no problem. The texts are all from papers, right? So they likely won’t contain unusual punctuation, missing whitespace and other things that make tokenization difficult.)

10

The samples I pasted here are from an existing dataset for Argument Mining. This dataset has txt files with the raw text and annotated files with the spans as in the examples I pasted here. I was streaming the txt files and annotating them on prodigy according to the annotations, but I realized that it would be far better to do this programatically.

So, after I'm done doing this, I'll use the same annotated dataset from prodigy and use it for training a NER model (to identify the spans using a prodigy NER model) and train a relation-extraction model, as you suggested here:

I think no splitting is required, since the raw texts are quite small, and each txt file is no larger than around 1.5kb.

Sounds good to you?

11

Ah, that's even better. And yes, your strategy sounds good! :+1:

12

Hi @ines

I managed to save the annotations to a prodigy dataset, but now I’m trying to test the exported dataset with a dep.teach recipe, and I’m getting the following:

prodigy dep.teach essays_relations en_core_web_sm essays_test.jsonl --unsegmented

File "cython_src/prodigy/components/preprocess.pyx", line 125, in add_tokens
ValueError: Mismatched tokenization. Can't resolve span to token index 1332. This can happen if your data 
contains pre-set spans. Make sure that the spans match spaCy's tokenization or add a 'tokens' property to 
your task. 

{'start': '1332', 'end': '1376', 'label': 'Claim', 'text': 'competition makes the society more effective'}

I had to use the --unsegmented parameter because I was getting another error.

I read some other topics in the forum related to the Mismatched tokenization exception, and conducted some tests you recommended. If I tokenize the raw text, I get the following:

[(token.text, token.idx, token.idx + len(token.text)) for token in tokens]

 ('competition', 1332, 1343),
 ('makes', 1344, 1349),
 ('the', 1350, 1353),
 ('society', 1354, 1361),
 ('more', 1362, 1366),
 ('effective', 1367, 1376),

Since I’m getting a match for the start and end indexes, I have no Idea what could be going on. Here’s the full span from the jsonl file:

{
"text":"Should students be taught to compete or to cooperate?\n\nIt is always said that competition can effectively promote the development of economy. In order to survive in the competition, companies continue to improve their products and service, and as a result, the whole society prospers. However, when we discuss the issue of competition or cooperation, what we are concerned about is not the whole society, but the development of an individual's whole life. From this point of view, I firmly believe that we should attach more importance to cooperation during primary education.\nFirst of all, through cooperation, children can learn about interpersonal skills which are significant in the future life of all students. What we acquired from team work is not only how to achieve the same goal with others but more importantly, how to get along with others. During the process of cooperation, children can learn about how to listen to opinions of others, how to communicate with others, how to think comprehensively, and even how to compromise with other team members when conflicts occurred. All of these skills help them to get on well with other people and will benefit them for the whole life.\nOn the other hand, the significance of competition is that how to become more excellence to gain the victory. Hence it is always said that competition makes the society more effective. However, when we consider about the question that how to win the game, we always find that we need the cooperation. The greater our goal is, the more competition we need. Take Olympic games which is a form of competition for instance, it is hard to imagine how an athlete could win the game without the training of his or her coach, and the help of other professional staffs such as the people who take care of his diet, and those who are in charge of the medical care. The winner is the athlete but the success belongs to the whole team. Therefore without the cooperation, there would be no victory of competition.\nConsequently, no matter from the view of individual development or the relationship between competition and cooperation we can receive the same conclusion that a more cooperative attitudes towards life is more profitable in one's success.",
"spans":[
    {
        "start":"1332",
        "end":"1376",
        "label":"Claim",
        "text":"competition makes the society more effective"
    },
    {
        "start":"1927",
        "end":"1992",
        "label":"Claim",
        "text":"without the cooperation, there would be no victory of competition"
    },
    {
        "start":"591",
        "end":"714",
        "label":"Claim",
        "text":"through cooperation, children can learn about interpersonal skills which are significant in the future life of all students"
    },
    {
        "start":"2154",
        "end":"2231",
        "label":"MajorClaim",
        "text":"a more cooperative attitudes towards life is more profitable in one's success"
    },
    {
        "start":"503",
        "end":"575",
        "label":"MajorClaim",
        "text":"we should attach more importance to cooperation during primary education"
    },
    {
        "start":"1088",
        "end":"1191",
        "label":"Premise",
        "text":"All of these skills help them to get on well with other people and will benefit them for the whole life"
    },
    {
        "start":"1212",
        "end":"1301",
        "label":"Premise",
        "text":"the significance of competition is that how to become more excellence to gain the victory"
    },
    {
        "start":"1387",
        "end":"1492",
        "label":"Premise",
        "text":"when we consider about the question that how to win the game, we always find that we need the cooperation"
    },
    {
        "start":"1549",
        "end":"1846",
        "label":"Premise",
        "text":"Take Olympic games which is a form of competition for instance, it is hard to imagine how an athlete could win the game without the training of his or her coach, and the help of other professional staffs such as the people who take care of his diet, and those who are in charge of the medical care"
    },
    {
        "start":"716",
        "end":"851",
        "label":"Premise",
        "text":"What we acquired from team work is not only how to achieve the same goal with others but more importantly, how to get along with others"
    },
    {
        "start":"853",
        "end":"1086",
        "label":"Premise",
        "text":"During the process of cooperation, children can learn about how to listen to opinions of others, how to communicate with others, how to think comprehensively, and even how to compromise with other team members when conflicts occurred"
    }
],
"_input_hash":592523368,
"_task_hash":1445145404
}
13

Hmm, that’s interesting. I’ll have a look!

One thing I noticed that could potentially be relevant: In your JSONL file, the "start" and "end" values are strings, not integers. Do you get the same error if you change that? (It probably happened because you converted from a CSV and all values were treated as strings. So the fix should be as simple as calling int() on the values.)

14

Thanks @ines, converting to int fixed the problem!

15

Just following this topic out of interest, thanks for the great info.

To clarify, the NER tagger/model in prodigy doesn’t care if the “entity” is a span of sentence+ length (ARGUMENT or RESPONSE) as opposed to 1~3 words length like PERSON or PLACE?

edit: by “doesn’t care”, I mean it will be performant in extracting longer sentences?

1 Like
16

Hi @sooheon. From what @honnibal told me in two other posts:

It's actually possible to create a NER model to classify spans as entities, but since they lack patterns in both semantics and even writing (regarding embeddings and letter-casing of the words), the model would have a great problem identifying the boundaries of the spans, and even the actual classes.

I'm not quite sure how to follow his suggestion to handle the boundary identification as a dependency parsing problem.

17

Hi @pvcastro,

Is there a sentence labeling workflow, or is one possible to create? (I’ve just downloaded prodigy today) Should we manually tokenize all text into sentences, have each sentence be a separate “text”, and label them as a classification problem? What about also marking important words in these sentences–how should I go about creating an interface to do both in one pass?

18

Prodigy and Spacy already have support to segmentation of texts into sentences, and tokenization of texts too, you should easily find some documentation on these. I had another example in which I was iterating through my training data and making them available to prodigy using stdin, and had no problems with it.

Then you could use the text classification recipe to classify each sentence. In order to get a great perspective on prodigy, I strongly recommend that you watch the two tutorials on new entity type and insults classifier. They were essential to provide the insights on how the tool works.

As for “marking important words”, the ner.manual recipe could be used for this, since it produces annotation data in the format of text and annotated spans, as already exemplified here and in some of my other threads where I discussed this with @ines and @honnibal.

1 Like