Good day,
We only have two labels in our dataset for training spancat, but we are getting a very low score (.28) for a training data of 1000 examples. In one of our labels, we have tagged long phrases (between 2 to 10 words). Would that affect the result/score? Is it better to have short phrases for spancat? Or it is because we have too few samples/training data?
Thanks.
What's the task that you're training spancat for? It helps to understand the linguistic complexity of the task before worrying about hyperparameters. Also, is the task in English? Anything you can share about the dataset?
We are classifying transcribed voicemails. Specifically, regarding medications. So, for example we are tagging the medication name, if they are asking for a refill, having an issue/reaction, have a usage question, etc.
So, we used the span to tag "Rx type" (Drug/medication names) and "Rx details" (I need a refill, I have a question, I am having a symptom, how much do I take, etc.)
As a follow up, would it matter if we just tag the most important keywords in a phrase/span, and try to skip as much stop words as possible? Right now, the size of our spans is between 1 to 31 (as reported by spacy debug data).
Is span categorization the best way to handle this? Are there other approaches we can use to find the Rx Type and Rx Details?
It might help to see some examples of the super long spans. I'm a bit suprised that a user might need 31 tokens to declare that they have need of a mere refill or question.
There are a few thoughts in my mind that might inspire progress though.
Here is the partial result of using spacy debug data:
Also does it matter if you use spacy.blank("en") when using add_tokens for our custom recipe?
When I feed your numbers into a small spreadsheet then I get the following:
span_size n percentage
1 1 682 0.248
2 2 252 0.340
3 3 193 0.410
4 4 209 0.487
5 5 240 0.574
6 6 240 0.661
7 7 219 0.741
8 8 201 0.814
9 9 165 0.874
10 10 99 0.910
That means that if you just limit yourself to spans of up to size 10, you'll still be covering ~91% of the cases. That might still be fine as a starting point and seems like a much easier subset to focus on initially.
That said, is there maybe an example of a very long span that you can share? I'm still a bit surpised and I'd like to understand the nature of your problem better. Out of curiosity, when you look at the mistakes that your model makes, are there any patterns?
Related: how was this data annotated? By a team? Are you 100% sure there are no label errors?
Assuming that you're using English, that should be totally fine. All of the pretrained pipelines use the same tokeniser as blank:en.
Do we need to relabel everything so the size of the spans is 10 max? How do we see the errors that the model makes during training? Only one person actually did the labeling, so I really am wondering why we are getting a very low score. 
Here are some examples of the spans (Rx Details) we have tagged:
Do we need to relabel everything so the size of the spans is 10 max?
That shouldn't be needed. What I might do is write a small filtering script that create the subset that you're interested in. You can do this from Python by the way by using the dataset API. It would look something like:
from prodigy.components.db import connect
db = connect()
examples = db.get_dataset_examples("my_dataset_name")
From here you can make whatever subset you like in the examples annotation list.
How do we see the errors that the model makes during training?
Ah, I didn't mean to check during training, but rather afterward.
You can either take the quantitative approach by checking on a validation set which errors are made most frequently. In your case, you could look if this is related to the span length. Another option is to take a more qualitative approach and to just interact with the model manually to see what kinds of texts it gets wrong.
In my experience, it really helps to understand when a model fails if you're looking for inspiration on how to improve it.
Only one person actually did the labeling, so I really am wondering why we are getting a very low score.
It's impossible to say, but it might be good to check if the annotations are consistent. If possible, I might try to get a second person to annotate some of these examples as well just to see if both annotators agree. You can use the review recipe to help with this. If it turns out that there's disagreement between annotators then it might help to take a step back and wonder if the definitions on what needs to be annotated are clear.
Some of the examples that you've given seem like good candidates to kickstart a discussion. Let's consider these few:
When I look at these two examples I immediately wonder ... when do you include the verb "have/had" to the span and when don't you? When do you include "I" to the span? Is there a clear definition for this? If not, it might explain why the model has trouble predicting some of these candidates.
This is all a bit speculative though, merely based on a quick impression. It's certainly possible that there's another reason for your model's performance, but it seems like a valid thing to consider.
I hope this helps. Let me know if you have follow-up questions!