No tasks available in prodigy==1.11.8a4 where there should be during named multi-session

, , ,
1

Related to https://support.prodi.gy/t/no-tasks-available-when-batch-size-1-instant-submit-true/6215/2 but not identical, since I changed the prodigy version and config.

So this is my version prodigy==1.11.8a4, this is my config

{
    "feed_overlap": false,
    "batch_size": 10,
    "experimental_feed": true
}

this is my command

PRODIGY_ALLOWED_SESSIONS=hao,hao2 PRODIGY_LOGGING=basic prodigy ner.manual ner_news_headlines blank:en ./annotated_news_headlines-ORG-PERSON-LOCATION-ner.jsonl --label PERSON,ORG,PRODUCT,LOCATION

I get "No tasks available" when there should be, after labelling a few samples, and after restarting the script, sometimes I get new samples and sometimes I don't, but it always end up leading to "No tasks available" when clearly there are some samples left to annotate:

Screen Shot 2023-01-05 at 12.51.01 PM
Screen Shot 2023-01-05 at 12.51.01 PM2638×956 129 KB

Screen Shot 2023-01-05 at 12.51.17 PM
Screen Shot 2023-01-05 at 12.51.17 PM2872×930 139 KB

Is the aforementioned behavior expected and how to fix it if not?

2

This is the log after I restart the script

root@junwang-ec2:/workspaces/multitask-llm-rnd/labelling/tutorial/ner# PRODIGY_ALLOWED_SESSIONS=hao,hao2 PRODIGY_LOGGING=basic prodigy ner.manual ner_news_headlines blank:en ./annotated_news_headlines-ORG-PERSON-LOCATION-ner.jsonl --label PERSON,ORG,PRODUCT,LOCATION
17:50:29: INIT: Setting all logging levels to 20
17:50:29: CONFIG: Using config from global prodigy.json
17:50:29: CLI: limiting user sessions to list: hao, hao2
17:50:29: RECIPE: Calling recipe 'ner.manual'
Using 4 label(s): PERSON, ORG, PRODUCT, LOCATION
17:50:29: RECIPE: Starting recipe ner.manual
17:50:30: RECIPE: Annotating with 4 labels
17:50:30: LOADER: Using file extension 'jsonl' to find loader
17:50:30: LOADER: Loading stream from jsonl
17:50:30: LOADER: Rehashing stream
17:50:30: CONFIG: Using config from global prodigy.json
17:50:30: VALIDATE: Validating components returned by recipe
17:50:30: CONTROLLER: Initialising from recipe
17:50:30: VALIDATE: Creating validator for view ID 'ner_manual'
17:50:30: VALIDATE: Validating Prodigy and recipe config
17:50:30: CONFIG: Using config from global prodigy.json
17:50:30: CONFIG: Using config from global prodigy.json
17:50:30: DB: Initializing database SQLite
17:50:30: DB: Creating dataset '2023-01-05_17-50-30'
17:50:30: FEED: Initializing from controller
17:50:30: DB: Getting dataset 'ner_news_headlines_feed'
17:50:30: PREPROCESS: Tokenizing examples (running tokenizer only)
17:50:30: FILTER: Filtering duplicates from stream
17:50:30: FILTER: Filtering out empty examples for key 'text'
17:50:30: CORS: initialized with wildcard "*" CORS origins

✨  Starting the web server at http://0.0.0.0:8080 ...
Open the app in your browser and start annotating!

INFO:     Started server process [13832]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     127.0.0.1:51660 - "GET /?session=hao HTTP/1.1" 200 OK
17:50:35: CONFIG: Using config from global prodigy.json
17:50:35: DB: Initializing database SQLite
17:50:35: POST: /close_session
17:50:35: CONTROLLER: Closing session: ner_news_headlines-hao
17:50:35: FEED: Closing session: ner_news_headlines-hao
17:50:35: RESPONSE: /close_session ({'closed': True})
INFO:     127.0.0.1:51660 - "POST /close_session HTTP/1.1" 200 OK
INFO:     127.0.0.1:57728 - "GET /bundle.js HTTP/1.1" 200 OK
17:50:36: CONFIG: Using config from global prodigy.json
17:50:36: DB: Initializing database SQLite
INFO:     127.0.0.1:57728 - "GET /project/hao HTTP/1.1" 200 OK
INFO:     127.0.0.1:51660 - "GET /lato-regular.woff2 HTTP/1.1" 304 Not Modified
17:50:36: CONFIG: Using config from global prodigy.json
17:50:36: DB: Initializing database SQLite
17:50:36: POST: /get_session_questions
17:50:36: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:36: FEED: Finding next batch of questions in stream
17:50:36: DB: Getting dataset 'ner_news_headlines_feed'
17:50:36: DB: Added 10 examples to 1 datasets
17:50:36: FEED: Sending batch of 10 questions for session: ner_news_headlines-hao
17:50:36: RESPONSE: /get_session_questions (10 examples)
INFO:     127.0.0.1:57728 - "POST /get_session_questions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57742 - "GET /lato-bold.woff2 HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:57750 - "GET /robotocondensed-bold.woff2 HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:57750 - "GET /favicon.ico HTTP/1.1" 200 OK
17:50:40: CONFIG: Using config from global prodigy.json
17:50:40: DB: Initializing database SQLite
17:50:40: POST: /get_session_questions
17:50:40: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:40: FEED: Finding next batch of questions in stream
17:50:40: FEED: Stream is empty
17:50:40: DB: Getting dataset 'ner_news_headlines_feed'
17:50:40: DB: Added 3 examples to 1 datasets
17:50:40: FEED: Sending batch of 3 questions for session: ner_news_headlines-hao
17:50:40: RESPONSE: /get_session_questions (3 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:40: CONFIG: Using config from global prodigy.json
17:50:40: DB: Initializing database SQLite
17:50:40: POST: /get_session_questions
17:50:40: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:40: FEED: Finding next batch of questions in stream
17:50:40: DB: Getting dataset 'ner_news_headlines_feed'
17:50:40: DB: Added 0 examples to 1 datasets
17:50:40: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:40: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:41: CONFIG: Using config from global prodigy.json
17:50:41: DB: Initializing database SQLite
17:50:41: POST: /get_session_questions
17:50:41: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:41: FEED: Finding next batch of questions in stream
17:50:41: DB: Getting dataset 'ner_news_headlines_feed'
17:50:41: DB: Added 0 examples to 1 datasets
17:50:41: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:41: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:41: CONFIG: Using config from global prodigy.json
17:50:41: DB: Initializing database SQLite
17:50:41: POST: /get_session_questions
17:50:41: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:41: FEED: Finding next batch of questions in stream
17:50:41: DB: Getting dataset 'ner_news_headlines_feed'
17:50:41: DB: Added 0 examples to 1 datasets
17:50:41: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:41: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:41: CONFIG: Using config from global prodigy.json
17:50:41: DB: Initializing database SQLite
17:50:41: POST: /get_session_questions
17:50:41: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:41: FEED: Finding next batch of questions in stream
17:50:41: DB: Getting dataset 'ner_news_headlines_feed'
17:50:41: DB: Added 0 examples to 1 datasets
17:50:41: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:41: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:42: CONFIG: Using config from global prodigy.json
17:50:42: DB: Initializing database SQLite
17:50:42: POST: /get_session_questions
17:50:42: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:42: FEED: Finding next batch of questions in stream
17:50:42: DB: Getting dataset 'ner_news_headlines_feed'
17:50:42: DB: Added 0 examples to 1 datasets
17:50:42: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:42: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:42: CONFIG: Using config from global prodigy.json
17:50:42: DB: Initializing database SQLite
17:50:42: POST: /get_session_questions
17:50:42: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:42: FEED: Finding next batch of questions in stream
17:50:42: DB: Getting dataset 'ner_news_headlines_feed'
17:50:42: DB: Added 0 examples to 1 datasets
17:50:42: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:42: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:42: CONFIG: Using config from global prodigy.json
17:50:42: DB: Initializing database SQLite
17:50:42: POST: /get_session_questions
17:50:42: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:42: FEED: Finding next batch of questions in stream
17:50:42: DB: Getting dataset 'ner_news_headlines_feed'
17:50:42: DB: Added 0 examples to 1 datasets
17:50:42: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:42: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:42: CONFIG: Using config from global prodigy.json
17:50:42: DB: Initializing database SQLite
17:50:42: POST: /get_session_questions
17:50:42: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:42: FEED: Finding next batch of questions in stream
17:50:42: DB: Getting dataset 'ner_news_headlines_feed'
17:50:42: DB: Added 0 examples to 1 datasets
17:50:42: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:42: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:42: CONFIG: Using config from global prodigy.json
17:50:42: DB: Initializing database SQLite
17:50:42: POST: /get_session_questions
17:50:42: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:42: FEED: Finding next batch of questions in stream
17:50:42: DB: Getting dataset 'ner_news_headlines_feed'
17:50:42: DB: Added 0 examples to 1 datasets
17:50:42: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:42: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:43: CONFIG: Using config from global prodigy.json
17:50:43: DB: Initializing database SQLite
17:50:43: POST: /get_session_questions
17:50:43: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao
17:50:43: FEED: Finding next batch of questions in stream
17:50:43: DB: Getting dataset 'ner_news_headlines_feed'
17:50:43: DB: Added 0 examples to 1 datasets
17:50:43: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao
17:50:43: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:57750 - "POST /get_session_questions HTTP/1.1" 200 OK
17:50:44: CONFIG: Using config from global prodigy.json
17:50:44: DB: Initializing database SQLite
17:50:44: POST: /give_answers (received 13, session ID 'ner_news_headlines-hao')
17:50:44: CONTROLLER: Receiving 13 answers
17:50:44: DB: Getting dataset 'ner_news_headlines-hao'
17:50:44: FEED: Received 13 answers for session: ner_news_headlines-hao
17:50:44: DB: Getting dataset 'ner_news_headlines'
17:50:44: DB: Getting dataset 'ner_news_headlines-hao'
17:50:45: DB: Added 13 examples to 2 datasets
17:50:45: CONTROLLER: Added 13 answers to dataset 'ner_news_headlines' in database SQLite
17:50:45: RESPONSE: /give_answers
INFO:     127.0.0.1:57750 - "POST /give_answers HTTP/1.1" 200 OK
INFO:     127.0.0.1:57750 - "GET /?session=hao2 HTTP/1.1" 200 OK
17:50:47: CONFIG: Using config from global prodigy.json
17:50:47: DB: Initializing database SQLite
17:50:47: POST: /close_session
17:50:47: CONTROLLER: Closing session: ner_news_headlines-hao2
17:50:47: FEED: Closing session: ner_news_headlines-hao2
17:50:47: RESPONSE: /close_session ({'closed': True})
INFO:     127.0.0.1:57750 - "POST /close_session HTTP/1.1" 200 OK
INFO:     127.0.0.1:34832 - "GET /bundle.js HTTP/1.1" 200 OK
17:50:48: CONFIG: Using config from global prodigy.json
17:50:48: DB: Initializing database SQLite
INFO:     127.0.0.1:34832 - "GET /project/hao2 HTTP/1.1" 200 OK
INFO:     127.0.0.1:57750 - "GET /robotocondensed-bold.woff2 HTTP/1.1" 304 Not Modified
17:50:48: CONFIG: Using config from global prodigy.json
17:50:48: DB: Initializing database SQLite
17:50:48: POST: /get_session_questions
17:50:48: CONTROLLER: Getting batch of questions for session: ner_news_headlines-hao2
17:50:48: FEED: Finding next batch of questions in stream
17:50:48: DB: Getting dataset 'ner_news_headlines_feed'
17:50:48: DB: Added 0 examples to 1 datasets
17:50:48: FEED: Sending batch of 0 questions for session: ner_news_headlines-hao2
17:50:48: RESPONSE: /get_session_questions (0 examples)
INFO:     127.0.0.1:34832 - "POST /get_session_questions HTTP/1.1" 200 OK
INFO:     127.0.0.1:34848 - "GET /lato-regular.woff2 HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:34860 - "GET /lato-bold.woff2 HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:34860 - "GET /favicon.ico HTTP/1.1" 200 OK

Note that it says Sending batch of 0 questions for session: ner_news_headlines-hao2 which is weird, there should be plenty of samples left to send, it should not be 0.

3

A post was merged into an existing topic: No tasks available in prodigy==1.11.8 when batch_size=1, instant_submit=True but there should be tasks available