Label multiple text at the same time

[AakankshaP (Aakanksha)]

Hi I'm working on a text classification problem where I take into account history of the data and label the status. I have 2 labels (Active/ Wait) and 7 sentences that belong to one instance.

In prodigy I want to look at all those 7 sentences at once and label each of those with either (Active/Wait). Is there a custom recipe that I can use to label it and further use it for a text classification problem?

[ryanwesslen]

hi @AakankshaP,

Thanks for your question and welcome to the Prodigy community

@AakankshaP

Hi I'm working on a text classification problem where I take into account history of the data and label the status. I have 2 labels (Active/ Wait) and 7 sentences that belong to one instance.

I'm not sure I understand your setup. I have several questions:

• What do you mean by "instance"? I don't understand what it means for 7 sentences to belong to one instance. Did you mean label?
• How is your input data currently structured? For example, are all of your input data the same, e.g., all on the sentence? Is this how your data comes raw or did you do any pre-processing steps for this?
• Why 7 sentences? 7 seems like an arbitrary number.
• Related, is there anything that links these 7 sentences together? For example, are all of the sentences from the same paragraph? Do you have metadata (e.g., a paragraph ID or page ID) that links the sentences?
• What's the purpose of annotating the 7 sentences together? Are you trying to "save time" by having the labeler annotate multiple sentences at a time? Any context for why you want this setup would be extremely helpful.
• Is it possible to do this step as a pre-processing step, i.e., you combine documents within your input (source) file? This would likely be the easiest way to implement.

Alternatively, if you could give hypothetical examples, that'll likely help us a lot.

@AakankshaP

In prodigy I want to look at all those 7 sentences at once and label each of those with either (Active/Wait).

It's important to mention that Prodigy was optimized for fast, simple decisions one at a time. This is especially true in binary text classification where the annotator can leverage the keyboard shortcuts to speed through examples. By adding multiple sentences, you may find it actually slows down annotators. For binary classification problems, you can even modify Prodigy to have swipes -- kinda like dating apps -- that can be helpful on mobile devices.

[AakankshaP (Aakanksha)]

Hi thank you so much for your reply. Here are the answers for your questions.

When I say instance it means history of a particular event.
For eg:
Patient is under observation [I want to label this as wait]
Patient is being treated [I want to label this as active]

The data is updated in real-time and there are time stamp attached to it. Each patient has a unique id and the details are being updated.

The number 7 is variable, depending upon the number of entries

No sentences are not structured as paragraphs, rather they are just 1-2 line sentences

The purpose of annotating the text together is to keep into account of the history of events as some complicated text cannot be easily labeled without the context

The purpose of this activity is to understand wait/active time, combining the text will not help us identify the difference between wait/active because in meta we include the timestamp to label.

[ryanwesslen]

hi @AakankshaP!

Thanks for the background.

@AakankshaP

The purpose of annotating the text together is to keep into account of the history of events as some complicated text cannot be easily labeled without the context

Ah gotcha. That makes sense.

This reminds me of this custom recipe I created. A user wanted to annotate sentences with a binary classification (like your wait/active) but wanted to provide the full context of the paragraph. In addition, their documents were sequential, so I created this custom recipe to start from the first paragraph, loop through each sentence with a highlight so the annotator knew which sentence was being annotated, then move to the next paragraph.

Classify sentences with paragraph visible

hi @rosamond! Thanks for your question and welcome to the Prodigy community Thanks for the excellent background! That helped me a lot. Would something like this work? [textcat_sent_sequence] It's a custom recipe that loops through highlighting each sentence for binary classification within context (paragraph). It has two user experience designs to enable faster annotations. First, since it's framed as a binary task, you can quickly annotate with key bindings (no mouse or span highl…

@AakankshaP

No sentences are not structured as paragraphs, rather they are just 1-2 line sentences

I know you mentioned your sentences are not structured as paragraphs -- but perhaps you could combine the sentences together you want to be shown together in the UI, but then use the highlighting to represent which sentence the user is actually annotating at a time.

The link includes a GitHub gist I created with the code and example.

Does this help?

[AakankshaP (Aakanksha)]

Hi Thank you so much for your reply I came across this recipe, but we do not want the highlighted interface.

I was thinking some interface like this but without the title and each text has 2 options to select (active/wait) or maybe under each sentence we have the options available.

I came across this issue and picked up the recipe from here

Labelling dataset for extractive text summarization

Okay, so you've definitely verified that the _task_hash that's generated for an incoming example is the same task hash that's already present in your dataset? Setting 'exclude': [dataset] in your recipe shouldn't be necessary, since this is the default behaviour. Also, another thing I noticed in your code: input_keys=("id") should be input_keys=("id",) or input_keys=["id"] – otherwise, the argument will be interpreted as a string instead of a list of keys.

[AakankshaP (Aakanksha)]

Hi is there a way to update the above custom recipe to have 2 options across each sentence? My task is not a binary classification that's why I need two labels as the third one is pre-defined