Reduce list of labels in textcat.manual to pre-annotated labels and none of the above

,
1

Hi,

I am wondering if it is possible with the textcat-manual recipe to reduce the list of labels to pre-selected/pre-annotated labels only, plus "none-of-the-above". I get an error if I don't point to a file of labels or include the labels in the command line after --label.

Command Line:

PRODIGY_ALLOWED_SESSIONS=jane,john prodigy textcat.manual dataset_name /path_to_file/filename.jsonl --label /path_to_file/labels.txt

.jsonl file sample:

{"id":"1234","deployment":"xxxx","date":"2021-11-01","text":"random text goes here","label":"label1","orig_accept":["label1"],"score":"0.5","accept":["label1","label2","label3","none-of-the-above"]}

Additionally, the pre-annotated labels are ranked 1-3. Is there any way to include that ranking in the UI?

For instance, the command line content and .jsonl content below: I was able to get the top three labels in the UI, with the top (pre-annotated) label chosen pre-checked. However, I had to hard code the label names for the individual document's results in the command line arguments. I would like to just have the top three labels in my .jsonl file appear in the UI. This differs for each document.

.jsonl file:

{"id":"1234","deployment":"xxxx","date":"2021-11-02","text":"random text","label":"label1","orig_accept":["label1"],"score":"0.5","accept":["label1"]}

command line arguments:

PRODIGY_ALLOWED_SESSIONS=jane,john prodigy textcat.manual dataset_name /path_to_file/filename.jsonl --label label1,label2,label3,none-of-the-above

sample_top_three_labels
sample_top_three_labels1586×1192 92.1 KB

Thanks!
Cheyanne

2

Hi,

Sorry for taking this long and thanks for waiting!

I am wondering if it is possible with the textcat-manual recipe to reduce the list of labels to pre-selected/pre-annotated labels only, plus "none-of-the-above".

You still need to supply values for the --label so that Prodigy knows what to display in the interface. This is also to ensure that the annotator "sees" the full list of labels when labelling. Another thing that you can do is include an option for none-of-the-above and then have a separate step that re-annotates all examples where the annotator selected none

Additionally, the pre-annotated labels are ranked 1-3. Is there any way to include that ranking in the UI?

There's no built-in feature for ranking labels but you can probably implement a custom HTML component with a dropdown for each label. Here's a good example: custom javascript elements - #3 by ines

1 Like