How I can catch type and subtype in email triaging using prodigy NER model

[ryanwesslen]

hi @Vishal112,

Thanks for your question.

First, please don't ping specific members of the team on posts. We answer questions round-robin and we try our best to reply as quickly as possible. Pinging a specific teammate won't get them to respond faster.

@Vishal112

I want the model to classify my email body text in type and subtype when I pass email body to it ,

It sounds like you really want to classify the entire document (email body). Why aren't you considering text classification?

NER is best suited for proper nouns and self-contained expressions like person names or products because it predicts single-token-based tags and takes advantage of clear span boundaries. If you're trying to classify the entire email, you'll need text classification.

For your question about two "types": a "type" and "subtype". This is typically rephrased as nested or hierarchical label problems.

We discuss our suggested approach for textcat hierarchical labels in our docs:

If you’re working on a task that involves more than 10 or 20 labels, it’s often better to break the annotation task up a bit more, so that annotators don’t have to remember the whole annotation scheme. Remembering and applying a complicated annotation scheme can slow annotation down a lot, and lead to much less reliable annotations. Because Prodigy is programmable, you don’t have to approach the annotations the same way you want your models to work. You can break up the work so that it’s easy to perform reliably, and then merge everything back later when it’s time to train your models.

If your annotation scheme is mutually exclusive (that is, texts receive exactly one label), you’ll often want to organize your labels into a hierarchy, grouping similar labels together. For instance, let’s say you’re working on a chat bot that supports 200 different intents. Choosing between all 200 intents will be very difficult, so you should do a first pass where you annotate much more general categories. You’d then take all the texts annotated for some general type, such as information, and set up a new annotation task to sort them into more specific subtypes. This lets the annotators study up on that part of the annotation scheme, so they can make more reliable decisions.

There have been multiple questions on this in the past (I've included some for ner and textcat):

Two levels of classifications for text classifications

Hi! This is definitely a solution and something we often recommend for hierarchical label schemes. You can read more about the idea and reasoning here: https://prodi.gy/docs/text-classification#large-label-sets It does mean you have to make a second pass over the data, but you'll be able to focus on only the subcategories in the second pass (which can be much more efficient because it's easier to focus) or use automation specific to the top-level categories (to pre-select labels). And it helps …

Hierarchal text classification trouble shooting

Sorry for the late reply here. For hierarchical text classification we typically recommend doing multiple passes over the dataset for each level of the hierarchy. This discussion thread has a lot of good info on this idea: Two levels of classifications for text classifications - #2 by ines If you really want to try this async workflow, it is not very straightforward with Prodigy. Internally, Prodigy uses Python generators for the "stream". Prodigy then pulls a batch of examples at a time to be…

Nested hierarchy for textcat

How many categories in the total hierarchy? If your category scheme has only has a few dozen categories in total, probably the approach that will take the least coding is to keep the classification problem flat, and then use your hierarchy at runtime to find the best scoring leaf class. Let’s say you have a hierarchy of 3 levels, each with three choices within them. So there’s 9 leaf labels: 0.0.0…2.2.2. There are also 12 (?) labels for the non-leaf categories: 0.., 1.., 2.., 0.0., 0.1., 0.2.*…

sublabel for NER

Hi! The NER interface is really optimised for actual named entity recognition tasks where the goal is to predict single token-based tags. One option would be to make two passes over the data and use the first pass to focus on getting the boundaries right (e.g. DiseaseEntity) and then annotate all additional attributes in the next round (e.g. using a multiple choice interface). If you're doing two passes, you can also use the first round to make sure all entities are labelled and that the annotat…

Hope this helps!