Annotation for Argument Mining

[pvcastro (Pedro Castro)]

@honnibal

The short answer is: yes, you could annotate this sort of thing with Prodigy.

How exactly? I understood from @ines other posts that this is still a work in progress. If it's possible to do it now, could you indicate which recipes I could use?

@honnibal

I must admit I’m also a bit skeptical that the methods in these papers are really ready for production use. The corpus they’re testing on is student essays, which tend to be taught to a fairly rigid formula.

I'm actually just studying how they do it with this essay scenario, but I'm actually going to apply it to another scenario, extracting arguments from legal documents containing court decisions, such as is done by https://nightowl.jp/publication/2017/9/8-argmining201703. I'm considering the essay's annotation scheme which tackles the problem as a relation extraction problem.

@honnibal

Otherwise, you’re always going to have boundary errors, which will cause a sequence tagging model a lot of problems.

Definetely, I believe this is why they keep two different scores: one which measures 100% match and one that measures 50% match, with the latter being much more lenient.

@honnibal

I would suggest annotating this task at the sentence level first though, and only making sub-sentence annotations in a later pass.

Do you think it makes sense to annotate in sentence or sub-sentence levels in this scenario? I believe they annotate in essay and paragraph levels because the arguments and claims linked are hardly in the same sentence.

@honnibal

I would strongly encourage you to align the sub-sentence annotations to syntactic constituents, to help make this more consistent.

I'm not sure I understand what you said here. Can you give me an example?

Thanks!