# How to Connect DB to/from Prodigy

**URL:** <https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850>\
**Category:** Uncategorized\
**Created:** [October 17, 2023, 5:44pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850 "2023-10-17T17:44:26Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![wertzhayden](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wertzhayden/32/4092_2.png) [@wertzhayden](https://support.prodi.gy/u/wertzhayden)\
**Post date:** [October 17, 2023, 5:44pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/1 "2023-10-17T17:44:26Z")

</div>

Hello, I am looking to connect my Postgres DB to and from Prodigy, so that all emails are sent for labeling and all labeling is properly stored for training.

Can you please share how I could do so? Happy to email / connect further to learn. Any documentation is also helpful!

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [October 17, 2023, 6:38pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/2 "2023-10-17T18:38:40Z")

</div>

hi @wertzhayden!

Thanks for your message and welcome to the Prodigy community!

You can find how to connect your database in the Prodigy database [docs](https://prodi.gy/docs/api-database#setup-postgresql). You'll need to modify Prodigy's [configuration](https://prodi.gy/docs/install#config) (aka `prodigy.json`) file like so:

```python
{
  "db": "postgresql",
  "db_settings": {
    "postgresql": {
      "dbname": "prodigy",
      "user": "username",
      "password": "xxx"
    }
  }
}

```

In case it helps, we've even recently created a [deployment docs section](https://prodi.gy/docs/deployment#postgres) that show a bit more detail of how to containerize your Prodigy session with a hosted PostgreSQL database (in our example we use Digital Ocean):

```python
{
  "db": "postgresql",
  "host": "0.0.0.0",
  "port": 8080,
  "db_settings": {
    "postgresql": {
      "host": "db-postgres-prodigy-do-user-243383-0.b.db.ondigitalocean.com",
      "dbname": "defaultdb",
      "user": "doadmin",
      "password": " ******",
      "port": 25060
    }
  }
}

```

If deploying on and putting your credentials on something like git, be sure to read our suggestions in those docs for setting the `.env` file to avoid putting any sensitive secrets (e.g., DB password) hard coded onto a `prodigy.json` that stored on git.

> While it can be a good idea to store the `prodigy.json` file in your git repository, you don’t want to add the database credentials to it. The password is purposefully kept out of the `prodigy.json` file in this example and will instead be filled in by a script that reads from a environment variable.
> 
> The `host` and `user` are also good candidates for environment variables, but we’ve kept these variables in the example to make it easier to explain the required parameters.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![wertzhayden](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wertzhayden/32/4092_2.png) [@wertzhayden](https://support.prodi.gy/u/wertzhayden)\
**Post date:** [October 19, 2023, 3:43pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/3 "2023-10-19T15:43:45Z")

</div>

> [@ryanwesslen](#):
>
> If deploying on and putting your credentials on something like git, be sure to read our suggestions in those docs for setting the `.env` file to avoid putting any sensitive secrets (e.g., DB password) hard coded onto a `prodigy.json` that stored on git.

Very helpful Ryan, thank you!

For next steps, if I create a remote DB and add the values to prodigy.json file, what would I need to do, in order to complete the DB connection to/from Prodigy?

Alternatively, how could I test that the labeled annotations are properly being added to the DB & how can I pull them all for training?

"port": env("port),  
"host": "0.0.0.0",  
"cors": true,  
"db": "postgresql",  
"db\_settings": {  
"postgresql": {  
"host": env("host"),  
"dbname": env("dbname"),  
"user": "admin",  
"password": env("password")  
}  
},

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [October 19, 2023, 3:57pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/4 "2023-10-19T15:57:11Z")

</div>

> [@wertzhayden](#):
>
> For next steps, if I create a remote DB and add the values to prodigy.json file, what would I need to do, in order to complete the DB connection to/from Prodigy?

It should do so automatically so long as your `prodigy.json` with your DB parameters is in the right location. As the docs mention, first Prodigy will look at your Prodigy Home (e.g., run `prodigy stats` to find that location). That's your "global" config file. Next, it'll look at your current working directory, i.e., where you're running the `prodigy` command. That's your "local" config file. The "local" will override anything in the "global". Last, you could add in your DB parameters as global overrides, which would then override your "local" and "global" configs.

My recommendation is that if you expect to always use the same DB, just put those into your global config (aka your Prodigy Home directory) and it'll always be used automatically when you run any commands.

> [@wertzhayden](#):
>
> Alternatively, how could I test that the labeled annotations are properly being added to the DB & how can I pull them all for training?

The easiest first step would be setup your `prodigy.json`, then run `prodigy stats` and you should now see an updated database name and id.

So perhaps just try with a dummy dataset to annotate. Then you could directly see if the annotations are in your Postgres database. Alternatively, you could use the built-in database components to connect directly to the database, see these [docs](https://prodi.gy/docs/api-database#connect) for details.

You can also use Prodigy [logging](https://prodi.gy/docs/install#debugging-logging) which should show some details about your database connection to make sure you're annotating.

Once it's there, you can just pull out the annotations and train as you deem fit -- not sure if you're using Prodigy's built-in spaCy tools for training or some other framework. You could also use `db-out` which is a Prodigy recipe to export out your annotations.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![wertzhayden](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wertzhayden/32/4092_2.png) [@wertzhayden](https://support.prodi.gy/u/wertzhayden)\
**Post date:** [October 19, 2023, 7:56pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/5 "2023-10-19T19:56:05Z")

</div>

Thank you for the info and "prodigy stats" step.

When testing that the labeled annotations are properly being added to the DB & when pulling for training, I see that there is a command "db-in" and "db-out".  
Is there a way to use "db-in" and "db-out" programmatically via Python? Put another way, how would I be able to call "db-in" and "db-out" via Python?

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [October 19, 2023, 7:58pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/6 "2023-10-19T19:58:58Z")

</div>

> [@wertzhayden](#):
>
> Is there a way to use "db-in" and "db-out" programmatically via Python? Put another way, how would I be able to call "db-in" and "db-out" via Python?

Excellent question!

Check out our [database components](https://prodi.gy/docs/api-database#database):

```python
from prodigy.components.db import connect

db = connect()
all_dataset_names = db.datasets
examples = db.get_dataset_examples("my_dataset")

```

Or as I mentioned previously, you can view exactly what `db-in` and `db-out` do by looking within the installed package at their recipes. Look in your `Location:` folder from `prodigy stats` and look for `recipes/commands.py` script. Hope this helps!

---

<div class="post-metadata">

**Author:** ![wertzhayden](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wertzhayden/32/4092_2.png) [@wertzhayden](https://support.prodi.gy/u/wertzhayden)\
**Post date:** [October 19, 2023, 8:20pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/7 "2023-10-19T20:20:09Z")

</div>

Ryan,

It appears that my "Location" folder does not exist within my system? I have attached the terminal as I searched for that folder in my system. Is there another way to find the "recipes/commands.py" script ?  
 ![Screenshot 2023-10-19 at 4.18.48 PM](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/e/ec436766c69e5319470d79fb2efb2863d03ef902.png)

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [October 19, 2023, 8:32pm UTC](https://support.prodi.gy/t/how-to-connect-db-to-from-prodigy/6850/8 "2023-10-19T20:32:05Z")

</div>

Sorry, I may not have been clear. By `Location` folder I meant the folder shown when you run `prodigy stats`. For example:

```plaintext
$ python -m prodigy stats

============================== ✨ Prodigy Stats ==============================

Version 1.14.4                        
Location /opt/homebrew/lib/python3.9/site-packages/prodigy
Prodigy Home /Users/ryan/.prodigy          
Platform macOS-14.0-arm64-arm-64bit    
Python Version 3.9.17                        
Spacy Version 3.6.0                         
Database Name SQLite                        
Database Id sqlite                        
Total Datasets 112                           
Total Sessions 322          

```

Then you can just use `open` in your command line to open up that folder:

```plaintext
$ open /opt/homebrew/lib/python3.9/site-packages/prodigy

```

And it'll open in Finder since you're on a Mac.