epochs

[ryanwesslen]

hi @Mohammad!

That's a great question!

Actually, you may want to modify patience because it's likely spaCy's early stopping mechanism is what's holding you back:

python -m prodigy train --ner ner_dataset --training.patience=0

patience controls early-stopping, i.e., the number of steps to continue without improvement before stopping. When you set it as 0, you disable early stopping.

By default, Prodigy enables unlimited epochs, i.e., spaCy's max_epochs are set to 0 by default with prodigy train. As a second lever, you may want to modify spaCy's max_steps which isn't set to unlimited. This can be modified similarly with --training.max_steps=X

This post provides a good background on setting max_steps:

How to under training.max_steps in train

Hello, I am training a ner model using this command. I found the score varies when I change --training.max_steps in my command, e.g., the score of training.max_steps=1000 is 0.29, training.max_steps=2000 is 0.58, and the score goes up to 0.9 when the --training.max_steps =0. I do not quite understnd why. I was wondering if you could help me understand the setting of training.max_steps and its impact on the model performance metrics? Thank you!

And this spaCy GitHub discussions provides a good explanation of the difference between patience, max_steps, and max_epochs:

How to configure when model training ends? · explosion/spaCy · Discussion #7465