My understanding is that when I call data-to-spacy, Prodigy will put some random samples of data into the dev set, so when I call the recipe twice, then I might end up validating with a different dev set.
During development, while I'm still figuring out how to best prepare my data that's fine but I'd like to have a static test set, to validate my "final" model.
Right now, my idea is to just have a seperate dataset and export it with -es 0, to have 1 set of data that I will use for final validation.
Does that make sense, or is there a better way to do this with Prodigy, that I'm missing?
Maybe I'm wrong about data-to-spacy?
Any pointers would be much appreciated!