What does it imply if the accuracy keeps on decreasing with every iteration ?
This is hard to answer without more specifics – what label(s) did you annotate, how many examples did you collect, and how does your train-curve command look?
One possible explanation is that you're experiencing the "catastrophic forgetting problem": As your model learns about new entity types, it "forgets" what it had previously learned. For example, if you've only collected annotations for PERSON, and you're training a model with all entity types (and you're not setting --label PERSON during training), you're evaluating the model on all entity types. So if the accuracy keeps going down, this could indicate that your model may be increasing in accuracy on the new label, but decreasing on all the other labels, resulting in a net decrease in accuracy. This thread has some more details on this, and how to prevent it:
1 Like