Natural Language Inference (Sentence Pair Evaluation)

, ,
1

Hello,

I am trying to use Prodigy to annotate a sentence-pair dataset for NLI (Entailment).
The data is formatted as a jsonl file with two separate sentences per line.

{"sentence1":"This is the first sentence I want to compare","sentence2":"This is the second sentence I want to compare"}

I then want to label the relation between these two sentences with 1 of 3 labels: A, B, and C.

I am trying to use the choice recipe for this, but I am having trouble getting it set up.
Does anyone have any tips/pointers for this? I am pretty unfamiliar with prodigy so anything will help. Thanks!

2

Hi! In case you haven't seen it yet, the custom recipes docs have some examples with multiple-choice options: https://prodi.gy/docs/custom-recipes#example-choice

You should be able to just adapt the sentiment annotation example and include your labels. An easy way to present your two sentences would be to add a "html" key to each task. For example, here you show them separated by line breaks:

task["html"] = f"{task['sentence1']}<br /><br />{task['sentence2']}"

Of course, you could also add a divider line in between, or show the texts in two columns – this depends on what you prefer and how important nice formatting is to you :slightly_smiling_face: