Unable to delete dataset

1

Hi, I am trying to delete a dataset (and session associated) to create a new one. The dataset has 36269 annotations and I get this error:

$ prodigy drop -n 500 uc-specific-reviewed
✘ Unable to delete dataset 'uc-specific-reviewed' because a database
limit on the number of query variables was reached. On some systems this limit
is quite low, so using a custom batch size may resolve the issue. Try: prodigy
drop -n 500 uc-specific-reviewed.

As you can see, I have already tried using the corrected command. What should I do?

2

Welcome to the forum @sunnielou :wave:

I understand the same error message appears when you just try prodigy drop uc-specific-reviewed.

Could you try with progressively lower batch sizes starting with 100?

3

Thanks @magdaaniol!

Yes, the same error appears without specifying the batch size. I tried with 100, 50, and 10 and I got the same error message:

prodigy drop -n 10 uc-specific-reviewed
✘ Unable to delete dataset 'uc-specific-reviewed' because a database
limit on the number of query variables was reached. On some systems this limit
is quite low, so using a custom batch size may resolve the issue. Try: prodigy
drop -n 500 uc-specific-reviewed

4

I see. Thanks for trying out different batch sizes.
Thanks for testing with different batch sizes!

Could you also provide the following details:

  1. The output of the prodigy stats command.
  2. The version of peewee you're using (pip freeze | grep peewee).
  3. If you're using SQLite, the version of sqlite3 (sqlite3 --version).
  4. For SQLite users: Could you check what SQLITE_MAX_VARIABLE_NUMBER is set to on your system?

Additionally, are you using the built-in Prodigy DB or a custom DB? Lastly, what kind of annotations are you working with (text, audio, images, video)? If you're working with audio, images, or video, are you storing base64-encoded data in the DB?

5

I am sorry for the delayed response. Here are the outputs:

prodigy stats
Version 1.15.7
Platform macOS-14.6.1-arm64-arm-64bit
Python Version 3.9.6
spaCy Version 3.7.6
Database Name SQLite
Database Id sqlite
Total Datasets 4
Total Sessions 3

The peewee version: peewee==3.16.3
SQLite version: 3.43.2
The SQLITE_MAX_VARIABLE_NUMBER is supposed to be 32766 since I have a version newer than 3.32.
I am also using the built-in Prodigy DB and working with text annotations.