My understanding is that when I call data-to-spacy, Prodigy will put some random samples of data into the dev set, so when I call the recipe twice, then I might end up validating with a different dev set.
During development, while I'm still figuring out how to best prepare my data that's fine but I'd like to have a static test set, to validate my "final" model.
Right now, my idea is to just have a seperate dataset and export it with -es 0, to have 1 set of data that I will use for final validation.
Does that make sense, or is there a better way to do this with Prodigy, that I'm missing?
Maybe I'm wrong about data-to-spacy?
Your understanding is correct, and your analysis & solution sounds very reasonable to me as well. The automatic (random) split can be used for quick experimenting, but for larger experiments I'd always recommend having a hold-out dataset as you said.
You can indeed make sure that everything ends up in the training dataset by setting -es 0, or you can specify another dataset as the dev set with eval:other_dataset, so you'd have something like --ner train_dataset,eval:dev_dataset. Or you just export with data_to_spacy and then you can still change the train and dev paths as you like.
One quick additional note: The above applies to the new train and data-to-spacy commands in v1.11 (currently nightly) that allow training multiple components with separate training and evaluation datasets. In the latest stable version, you can provide the training and eval datasets via separate arguments.
