# Are there restriction on number of labels? **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, solved\ **Created:** [July 26, 2019, 1:24pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832 "2019-07-26T13:24:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Vonderful](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vonderful/32/893_2.png) [@Vonderful](https://support.prodi.gy/u/Vonderful)\ **Post date:** [July 26, 2019, 1:24pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/1 "2019-07-26T13:24:12Z")
Hi I am in the process of setting up Prodigy to annotate some entities. One case we have 25 entities, but i get an error when running code; prodigy ner.manual symbility\_sample /model-ner-en\_blank /Symbility\_Comm\_cleansed\_sample\_PRODIGY\_VERSION.csv --label "Glass,Undg,InsProp,CompMach,Aircraft,EarthQ,PipeFz,PipeNoFz,OverflowTank,LeakSprinkler,MechFailure,FaultyWork,WearTear,FloodDrainage,FloodRiver,FloodCoast,ImpactNoVehi,ImpactOVeh,Impact3Veh,TreeStorm,HailStorm,SnowStorm,WindStorm,IngressStorm,ImpactOther" Error: OSError: [Errno 36] File name too long: ' Ive reduced the length of the entitiy names to help with other use cases, but cant redude this anymore without it becoming difficult for annotators. Thanks in advance. ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [July 30, 2019, 11:12am UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/2 "2019-07-30T11:12:52Z")
You've sort of hit an edge-case here that we didn't expect. You can provide the labels as either a text file, or on the command line. To figure out which is which, we check whether the file exists. Normally this works fine, but in your case, the is-a-file check fails because the string is too long. We can fix this in Prodigy, but you can work around it in the meantime by putting the labels into a text file (one label per line), and then pointing Prodigy at that. Taking a step back though, I think you might want to consider annotating with far fewer labels initially. Annotating with a lot of labels will give the annotators a much more difficult experience, because they have to select the label from a long list, and remember a lot of detailed definitions. There are a few solutions: 1. You could have one generic category for the span-annotation phase, and divide the extracted texts into types later. 2. You could have one annotation task per entity type, and merge the annotations later. 3. You could make the label scheme hierarchical, and do the top-level categories first. 4. You could make the annotation sentence-based, rather than span-based. I think option 3 is one you should especially consider. You could at least have a sentence level task where you annotate whether the sentence contains any entities. If your data has a low density of entities, this will allow you to move through the data much quicker. You could then annotate only the "has at least one entity" sentences in a subsequent pass, probably working on one entity-type at a time. ---
**Author:** ![Vonderful](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vonderful/32/893_2.png) [@Vonderful](https://support.prodi.gy/u/Vonderful)\ **Post date:** [July 30, 2019, 2:26pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/3 "2019-07-30T14:26:23Z")
Amazing thanks! I wasnt aware we could use a text file for the labels, my bad! I 100% agree re the number of entities, we have way too many and i know that some will not get annotated as no data will exist for those...ill be able to prove this to "management" within a few days thanks to your tool! For the time being we have gone down the route of hierarachical, so what , why , how and when. Then i think we will look at the sentence based approach as that sounds perfect for when we go in to prod with this! THanks SO much! Siobhan ---
**Author:** ![jiebei](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jiebei](https://support.prodi.gy/u/jiebei)\ **Post date:** [September 26, 2022, 2:20pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/4 "2022-09-26T14:20:33Z")
We have the same problem with too many labels. If we are going to follow suggest #3, how can we get started? Thank you! ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [September 26, 2022, 2:54pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/5 "2022-09-26T14:54:32Z")
hi @jiebei, Great question! I don't have a perfect solution but let me provide a few additional resources that help provide more context on Matt's points above. First this chain from Ines and Sofie have some great words of wisdom: > [@NER with dozens of entities](https://support.prodi.gy/t/ner-with-dozens-of-entities/4139/3): > > As a more general comment/question: Is your goal to actually train an NER model? If so, are you sure you want to structure your annotation task and model this way and have that many labels? You might be making your life a lot harder this way, because you'll need significantly more data and make sure you have enough examples per label so the model can distinguish between them. If the labels are hierarchical, you'd also be encoding multiple aspects within each annotation that the model has to lear… Also, you may find some additional inspiration from `textcat` documentation on [Dealing with very large label sets or hierarchical labels](https://prodi.gy/docs/text-classification#large-label-sets). This provides context on the philosophy of try to break up the task to the smallest parts possible: > If you’re working on a task that involves **more than 10 or 20 labels** , it’s often better to break the annotation task up a bit more, so that annotators don’t have to remember the whole annotation scheme. Remembering and applying a complicated annotation scheme can slow annotation down a lot, and lead to much less reliable annotations. Because Prodigy is programmable, you don’t have to approach the annotations the same way you want your models to work. You can break up the work so that it’s easy to perform reliably, and then merge everything back later when it’s time to train your models. It's important to know if this is your first time training any NER model, that very likely you may find your NER entities change as you label, i.e., modify your annotation scheme for the entities. Let's take an example I had the first time I tried to use Prodigy NER for labeling a company's internal policies to identify different internal teams/organizations. When I first started, I thought I would label groups as "ORGANIZATIONS", but as I went through more documentations, I realized that the documents had more groups that I considered "TEAMS" (that is, sub-groups of the Organizations). I realized I needed to modify my mental model as I went through. The key was that my annotation scheme changed as I labeled more and my prior beliefs, while helpful, weren't necessarily aligned with what was actually in the data. That's why I really like Matt/Ines' suggestion is to focus on the top level of the hierarchy first, realizing that you may find after a few hundred annotations your may change. Then after you build your initial model on the high level, only then consider more specific/narrow entities. It's worth watching Matt's 2018 PyData talk where he talks about why many NLP projects fail -- for example, they may have too ambitious plans early on and don't test/iterate enough. [![](https://img.youtube.com/vi/jpWqz85F_4Y/maxresdefault.jpg "Building new NLP solutions with spaCy and Prodigy - Matthew Honnibal") ](https://www.youtube.com/watch?v=jpWqz85F_4Y&t=141) He then goes into a great discussion on the "ML hierarchy of needs" to focus on the problem, then understand the annotation guidelines (aka annotation schemes), which are written instructions that are important for NLP projects to ensure that annotators have clear definitions and examples of what they are annotating. The guidelines could simply be in a Word/Google Doc, but the key is to be explicit with how you're defining your entities and iterate on the document (especially through a group discussion). Matt also goes through a good example of framing a similar entity task and then builds up great recommendations (around 22:40) on why it's important to "worry less about scaling up (e.g., getting so many labels), and more about scaling down". We're actively working on content to provide better case studies of how to evolve annotation guidelines to help reduce the risk of project failure. I would encourage reading my colleague Damian Romero's wonderful [Tweet thread on Annotation Guidelines 101](https://twitter.com/DamianRomero_CL/status/1520930211864489985?s=20&t=lJQXeGIFw93yueh-9D98eA). Also, if you want a great example of annotation guidelines, [check out this GitHub repo from the Guardian newspaper](https://github.com/JournalismAI-2021-Quotes/quote-extraction/blob/main/annotation_rules/Quote%20annotation%20guide.pdf), who had a great post about how they used Prodigy for an NER task to identify quote-related items. They iterated in each round of their annotations to refine their list. You can see their guidelines are detailed for even just three entities because they found quotes can be very complex. This is more reason why trying to do many entities your first time will drastically increase your likelihood of project failure. > [@How The Guardian is using spaCy and Prodigy to identify quotes in text](https://support.prodi.gy/t/how-the-guardian-is-using-spacy-and-prodigy-to-identify-quotes-in-text/5010): > > A great article on how The Guardian is using spaCy and Prodigy to build a model to indentify quotes in new articles :sparkles:Highlights include: why it's so important to iterate on your data and carefully develop custom annotation schemes to deal with ambiguity in language, plus a very pretty custom UI theme for Prodigy! The main challenge in building the training dataset was navigating the ambiguity of different journalistic styles. For several days, we discussed dozens of cases where it … If you make some progress, be sure to document and feel free to share any learnings back here! I know the community would great appreciate it 🙂 ---
**Author:** ![jiebei](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jiebei](https://support.prodi.gy/u/jiebei)\ **Post date:** [September 28, 2022, 4:49am UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/6 "2022-09-28T04:49:07Z")
Thank you so much, Ryan! I guess a related question is shall we use text classification recipe instead of ner if we are going to implement a hierarchical label structure? ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [September 28, 2022, 4:01pm UTC](https://support.prodi.gy/t/are-there-restriction-on-number-of-labels/1832/7 "2022-09-28T16:01:56Z")
> [@jiebei](#): > > I guess a related question is shall we use text classification recipe instead of ner if we are going to implement a hierarchical label structure? You can try to create a hierarchical label structure for either `textcat` or `ner`. For `textcat` your labels would be categories while for `ner` your labels would be entity types. Choosing `textcat` vs. `ner` depends on what problem you're trying to solve. If you're starting out for your first time in Prodigy and want to do a pilot (model you may not use but want to learn Prodigy workflow), I would recommend start with `textcat` and try to label the first level of your `textcat` hierarchy. Try to classify it into a few (say 3-6) useful categories. You can choose whether documents can have only one category (mutually exclusive) or multiple categories (multi-label). Label a few hundred examples, train a model, and learn a lot about your data in a day or so. Along the way, make small observations (e.g., writing down on paper) to think about the entity types you're interested in for `ner`. Hopefully after this first round, you may have a better mental model of appropriate entity types for `ner`. Prodigy's documentation has a [good FAQ on thinking about the pros/cons of each](https://prodi.gy/docs/named-entity-recognition#ner-vs-textcat): ## I'm not sure if NER is a good fit or if I should train a text classifier? > Named entity recognition models work best at detecting **relatively short phrases** that have fairly **distinct start and end points**. A good way to think about how easy the model will find the task is to imagine you had to look at only the first word of the entity, with no context. How accurately would you be able to tell how that word should be labelled? Now imagine you had one word of context on either side, and ask yourself the same question. > With spaCy’s current defaults (as of [v2.2](https://spacy.io/usage/v2-2)), the model gets to see four words on either side of each token (it uses a convolutional neural network with four layers). You don’t have to use spaCy, and even if you do, you can reconfigure the model so that it has a wider contextual window. However, if you find you’re working on a task that requires distant information to make the decisions, you should consider learning at least part of the information you need with a text classification approach. > Entity recognition models will **especially struggle** on problems where the annotators disagree about the **exact end points** of the phrases. This is a common problem if your category is somewhat vague, like “cause of problem” or “complaint”. These meanings can be expressed in a variety of ways, and you can’t always pin down the part of a sentence that expresses a particular opinion. If you find that annotators can’t agree on exactly which words should be tagged as an entity, that’s probably a sign that you’re trying to mark something that’s a bit too semantic, in which case [text classification](https://prodi.gy/docs/text-classification) would be a better approach. There are also several related support issues that ask related questions: > [@first annotation - can I switch mid-way from ner.manual to textcat?](https://support.prodi.gy/t/first-annotation-can-i-switch-mid-way-from-ner-manual-to-textcat/4428/4): > > Hi! To be honest, that doesn't really sound like a textcat challenge to me. Text classification is more about assigning one or multiple labels to the entire text. From what you described, you're still interested in finding the actual token boundaries of your 5 types of entities, correct? That said, you're right that this doesn't sound like a "traditional" NER challenge like recognizing properly capitalized names or cities. But you still need something similar to NER to detect the token boundar… > [@Using a text classifier instead of NER](https://support.prodi.gy/t/using-a-text-classifier-instead-of-ner/2712/2): > > Thanks for the suggestion! I agree that a video on this would be a great idea, I'll start thinking about that thinking . I've been weighing up different ideas for videos and I think that's a great suggestion. There are two ways to do the text-classification-as-NER strategy. One is to structure your downstream application so that you don't require the specific highlighted span. Sometimes this is viable, sometimes it isn't. The other way is to chain together text classification and some sort o… Also, if in doubt, you may be tempted to say: why don't we create a custom recipe to do both `ner` and `textcat` at the same time, because that would save a lot of time. It is possible to do this using `blocks`, we warn against this as in the [Combining Interfaces with Blocks docs](https://prodi.gy/docs/custom-interfaces#blocks): > It’s recommended to only use the [`blocks`](https://prodi.gy/docs/api-interfaces#blocks) interface for annotation tasks that absolutely require the information to be collected at the same time – for instance, comments or answers about the current annotation decision. While it may be tempting to create one big interface that covers all of your labelling needs like text classification and NER at the same time, this can often lead to **worse results** and data quality, since it makes it harder for annotators to focus. It also makes it more difficult to iterate and make changes to the label scheme of one of the components. You can always merge annotations of different types and create a single corpus later on, for instance using the [`data-to-spacy`](https://prodi.gy/docs/recipes#data-to-spacy) recipe.