Dynamically defining subset of labels to use in SpanCat

[ryanwesslen]

hi @darinkishore,

Thanks for the thoughtful post and welcome to the Prodigy community

This is an interesting use case. Let me discuss this with the team next week.

@darinkishore

How can I do this in Prodigy? Dynamically allocating labels does not seem supported, and I have tried working with the recipes, but I'm not sure where I would start. Detailed instructions would be so, so, SO appreciated. This project haunts me.

We just had a related post and this has come up before too. But yes, the challenge is Prodigy was designed as "one-label-set-per-session".

Dynamic label

Hi if we use ner.manual recipe we set up the labels in advance. is it possible to set up the labels while annotating? so the label will be added based on the context taking into account having just one label for each context. [custom_label_example]

@darinkishore

To remedy this, I am trying to add an intermediate detection step, where instead of using all 50 labels for my annotation task, I call a weaker model like 3.5-turbo to get the 12? most likely concerns based on the conversation. Then, I can use these labels dynamically for the annotation task, which allows me to give longer, more precise definitions, and the model a lot more space to reason with, due to not overflowing the context limit.

I like this direction - but now it looks like you're hitting a second problem of nested or hierarchical categorization. This is another UI challenge but also one of designing the categorization scheme (e.g., why 12 of 50? how do you define which subsets).

hierarchical text classification using spancat and potentially expanding/hiding label subclasses as they come in context

Hi, I have nested labels that follow a structure, if the leaf label is true, the parent is deduced to be true. I would love if prodigy can handle this case. Currently the UI only supports simple flat list of labels making it unwieldy, There are two ways suggested, one to use a flat list as if these are all independent classes, The other way to handle this, is by splitting things into multiple recipes which means each annotator has to re-read the text rather than tagging a specific hierarchy in …

@darinkishore

Finally, as a side note—is there a workflow/recipe for making predictions based on input text and validating them? After this task, for each Speaker 1 utterance, I'd like to identify (concern, parties affected, timeframe) based on the span-concern

Have you seen the validate answer callback? Easy way to validate an answer.

@darinkishore

PS: If there is any documentation on using multiple spacy.llm calls in one config file, that would be super appreciated. I'd love an automatic reasoning check to make sure the model is doing alright before it sends me the results for annotation.

I don't think that's possible but perhaps open a discussion post GitHub or I can check with the team next week.

Update: I checked with a spaCy core teammate. So it is possible if you're using different components, the key is naming them something unique but still calling the "llm" factory.

[nlp]
pipeline = ["llm_textcat", "llm_ner"]
...

[components]

[components.llm_textcat]
factory = "llm"

[components.llm_textcat.model]
@llm_models = ...

[components.llm_textcat.task]
@llm_tasks = "spacy.TextCat.v3"

...

[components.llm_ner]
factory = "llm"

[components.llm_ner.model]
@llm_models = ...

[components.llm_ner.task]
@llm_tasks = "spacy.NER.v3"
...

@darinkishore

Edit: apologies for the massive amount of questions in one post. I've been putting my all into learning prodigy, as data quality/consistency issues are my biggest weakness atm. Dynamic-Labels is the most important thing.

No worries at all! We really appreciate your post. As I mentioned, we've been thinking about several of these items for a while. We'll reach back out next week. Thank you!