Did you point the PRODIGY_HOME variable to the to the directory containing the prodigy.json with your custom database settings? And how does your custom database class work? If it's a fully custom class, you'll need to make sure it's available and Prodigy knows where to find it.
One option for this is via entry points. If you're serving Prodigy from Python, you can also register it like this, and then set "db": "custom_db" in your prodigy.json:
from prodigy.util import registry
db = YourCustomDatabase()
registry.databases("custom_db", func=db)