# Feature Request: Bulk Dataset Drop

**URL:** <https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301>\
**Category:** Uncategorized\
**Tags:** enhancement, database\
**Created:** [February 9, 2018, 3:55pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301 "2018-02-09T15:55:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wpm/32/103_2.png) [@wpm](https://support.prodi.gy/u/wpm)\
**Post date:** [February 9, 2018, 3:55pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/1 "2018-02-09T15:55:32Z")

</div>

While experimenting with Prodigy I find myself creating and dropping lots of datasets. Every once in a while I go through and clean up every dataset. This is a little tedious because `prodigy drop` only works on one dataset at a time.

It would be nice to have ways to make this faster. `prodigy drop` takes multiple arguments, or maybe patterns to match the names of datasets.

---

<div class="post-metadata">

**Author:** ![andy](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/andy/32/966_2.png) [@andy](https://support.prodi.gy/u/andy)\
**Post date:** [February 9, 2018, 4:12pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/2 "2018-02-09T16:12:48Z")

</div>

You could use the `drop` function in Prodigy’s ` __main__.py` and apply it over a list with something like this (WARNING, untested):

```python
from prodigy.components.db import connect
DB = connect()
# could take as a plac input
to_drop = "db1,temp1,test4"
to_drop = [i.strip() for i in to_drop.split(",")]

# [Copy this function from __main__.py, line 132]
def drop(set_id):
    """
    ...
    """

for db in to_drop:
    drop(db)

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [February 9, 2018, 4:18pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/3 "2018-02-09T16:18:03Z")

</div>

@andy That’s a nice idea actually!

Alternatively, Prodigy’s `Database` also has a `drop_dataset` method – this is a little more direct, but won’t give you any warnings or print any results.

```python
db = connect()
for dataset in ['db1,' 'temp1', 'test4']:
    db.drop_dataset(dataset)

```

I’ve also been thinking about building a little app that lets you view (and potentially manage) datasets in the browser. Like, a “Prodigy Dataset Explorer”. We wouldn’t necessarily ship this with the library, but it could be a nice open-source addon that users could install and contribute to 🙂

---

<div class="post-metadata">

**Author:** ![justindujardin](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/justindujardin/32/169_2.png) [@justindujardin](https://support.prodi.gy/u/justindujardin)\
**Post date:** [February 9, 2018, 6:04pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/4 "2018-02-09T18:04:18Z")

</div>

> [@ines](#):
>
> I’ve also been thinking about building a little app that lets you view (and potentially manage) datasets in the browser. Like, a “Prodigy Dataset Explorer”. We wouldn’t necessarily ship this with the library, but it could be a nice open-source addon that users could install and contribute to 🙂

I've got something like that as an Electron app, if you're using sqlite. I've been using it for reviewing and updating annotations on my project, and as a scratch pad for making [programmatic changes to my db](https://github.com/justindujardin/prodigy-viewer/blob/8331303d9bbd41daf364ff2f7eec5805dbcf0469/src/components/home.component.ts#L58-L101). That's what the beautiful "custom" button does on the top right.

 ![06 AM](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/1X/4bd0725f5ab991bc1f97c255740c7b4b2435348a.png)

I pushed it to Github, in case it's useful for other people: [GitHub - justindujardin/prodigy-viewer: An app for reviewing and changing prodigy annotations after an annotation session is complete.](https://github.com/justindujardin/prodigy-viewer)

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [February 9, 2018, 6:38pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/5 "2018-02-09T18:38:01Z")

</div>

@justindujardin Oh wow, this is amazing!!! It makes me so happy to see all the cool stuff you and others are building with and for Prodigy 💛 (We should probably start compiling a list of all addons, scripts and custom recipes soon. Luckily, the [`prodigy` topic](https://github.com/topics/prodigy) on GitHub is unoccupied, which is pretty nice!)

Btw, in terms of the database connection: Not sure how easy it would be to integrate something like this, but in theory, the app could also communicate with Prodigy’s database via a REST API. For example:

```python
DB = prodigy.components.db.connect()

@hug.get('/dataset/{dataset_id}')
def get_dataset(dataset_id):
    examples = DB.get_dataset(dataset_id)
    return {'examples': examples}

```

To make this work more smoothly, we probably need a few additional database methods, though (like `update_example` etc).

---

<div class="post-metadata">

**Author:** ![justindujardin](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/justindujardin/32/169_2.png) [@justindujardin](https://support.prodi.gy/u/justindujardin)\
**Post date:** [February 9, 2018, 9:15pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/6 "2018-02-09T21:15:54Z")

</div>

> [@ines](#):
>
> Btw, in terms of the database connection: Not sure how easy it would be to integrate something like this, but in theory, the app could also communicate with Prodigy’s database via a REST API

Yeah, that'd be much better! I've been putting all the [sqlite specific stuff in an angular service](https://github.com/justindujardin/prodigy-viewer/blob/master/src/sqlite.service.ts), so it should be pretty easy to swap out.

> [@ines](#):
>
> To make this work more smoothly, we probably need a few additional database methods, though (like update\_example etc).

The public API of the [sqlite service](https://github.com/justindujardin/prodigy-viewer/blob/master/src/sqlite.service.ts) class is probably a good reference for that kind of stuff. You may be right that it could all boil down to an `update_example` endpoint. 😅

---

<div class="post-metadata">

**Author:** ![emiltj](https://avatars.discourse-cdn.com/v4/letter/e/d07c76/32.png) [@emiltj](https://support.prodi.gy/u/emiltj)\
**Post date:** [January 21, 2023, 11:06am UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/7 "2023-01-21T11:06:19Z")

</div>

Just here to agree, that it would be nice to be able to drop multiple in the prodigy drop CLI, without having to use python or add-ons.

And additionally perhaps a `prodigy drop -all` feature

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [January 23, 2023, 10:13am UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/8 "2023-01-23T10:13:05Z")

</div>

I'm a little bit wary of supporting a `drop --all` here, mainly because it can lead to a database loosing all of it's data. Unless everyone has proper backups, supporting this might cause dramatic accidents to take place.

---

<div class="post-metadata">

**Author:** ![emiltj](https://avatars.discourse-cdn.com/v4/letter/e/d07c76/32.png) [@emiltj](https://support.prodi.gy/u/emiltj)\
**Post date:** [January 31, 2023, 2:31pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/9 "2023-01-31T14:31:45Z")

</div>

Could be a good argument against it.

However, dropping a list of datasets would have been hugely beneficial for my latest project.

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [January 31, 2023, 2:34pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/10 "2023-01-31T14:34:17Z")

</div>

I understand. But having a custom Python script seems like a sensible hurdle to make sure folks don't do it by accident.

---

<div class="post-metadata">

**Author:** ![emiltj](https://avatars.discourse-cdn.com/v4/letter/e/d07c76/32.png) [@emiltj](https://support.prodi.gy/u/emiltj)\
**Post date:** [February 1, 2023, 4:52am UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/11 "2023-02-01T04:52:22Z")

</div>

Sorry to keep dragging the thread on, but this doesn't speak against being able to bulk drop by a list, does it?

E.g. have the functionality matching:

prodigy drop dataset1,dataset2,dataset3

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [February 3, 2023, 3:20pm UTC](https://support.prodi.gy/t/feature-request-bulk-dataset-drop/301/12 "2023-02-03T15:20:37Z")

</div>

If you _really_ want to delete all files, you can also choose to manually delete the Sqlite database file locally. It's not something I recommend, because there's a risk of loosing data. But that would really delete it all.

Another option could also be to use bash directly. This would allow you to delete all tables by listing all the names in a file.

Suppose that you have this file called `names.txt`:

```python
name-a
name-b
name-c

```

Then you could run:

```python
cat names.txt | while read line; do prodigy drop "$line"; done

```