I notice that testing is a pretty well-ingrained part of the open-source Explosion projects, so I’m basing this question on the assumption that Prodigy is treated the same. We’d love to test Prodigy on our particular installations - is there any way to access those tests? Are they distributed anywhere? If not, can they be?
We don’t package Prodigy’s tests with the library currently, as they’re integrated with some of our other systems, and we didn’t think it would be helpful to people.
Could you explain a bit more about what bugs you’re hoping to prevent? I don’t think there should be a situation where the wheel installer passes the test cases on our systems but fails on yours.
If you’re looking to test your custom recipes, we can definitely give you some advice on that though. The recipes should be easy to call as Python functions, so that they can be tested with normal pytest workflows.
Mostly seeking it to ensure dependency updates don’t have unintended side-effects. Especially if any security vulnerabilities are found in Prodigy or its dependencies, we’d like to know we can integrate a patch ASAP and be confident in not losing functionality.
That’s fair enough, but I also think you’re unlikely to have problems. Almost all of the dependencies are actually provided by us.
Here’s the current requirements.txt. We’ve really tried to avoid dependencies that could cause problems, preferring to package things ourselves where possible. Of the third-party dependencies we use a lot, I would say maybe an update to the ORM peewee could introduce a problem? Both toolz and plac are very stable.
# Dependencies provided by Explosion
spacy>=2.1.0,<2.2.0
wasabi>=0.1.3,<1.1.0
srsly>=0.0.6,<1.1.0
# Other important dependencies
peewee>=2.10.1,<3.0.0
toolz>=0.8.2,<=1.0.0
plac>=0.9.6,<1.0.0
# Will be replaced by FastAPI and uvicorn in an upcoming release.
# (The author of FastAPI is joining Explosion)
hug>=2.4.1,<2.4.9
waitress>=1.0.0,<1.3.0
# Peripheral functionality
jsonschema>=2.6.0,<3.0.0
cachetools>=2.1.0
pyjwt>=1.6.1,<2.0.0
requests>=2.0.0,<3.0.0
I do appreciate that Prodigy has a low dependency footprint, and that makes me more confident that issues like this will be few and far between, but people make mistakes - that’s why we have tests 
. The more confidence we can have when we’re moving updates, the faster we can move.
Also, the scope of dependencies that can change widens considerably when you look at sub-dependencies - this is a freeze of our environment now (the only change we integrated since 1.8.3 was to update srsly after it was fixed from Cannot load data with db-in on Prodigy 1.8.3 using annotations created with 1.6):
blis==0.2.4
cachetools==3.1.1
certifi==2019.3.9
chardet==3.0.4
cymem==2.0.2
falcon==1.4.1
hug==2.4.8
idna==2.8
jsonschema==2.6.0
murmurhash==1.0.2
numpy==1.16.4
peewee==2.10.2
plac==0.9.6
preshed==2.0.1
prodigy==1.8.3
psycopg2-binary==2.8.2
PyJWT==1.7.1
python-mimeparse==1.6.0
requests==2.22.0
six==1.12.0
spacy==2.1.4
srsly==0.0.7
thinc==7.0.4
toolz==0.9.0
tqdm==4.32.1
urllib3==1.25.3
waitress==1.2.1
wasabi==0.2.2
That said, I understand if you can’t / don’t want to release the testing suites. Just figured it wouldn’t hurt to ask!