Recipe ner.eval-ab never returns any tasks

, ,
1

I’m trying to get a feel for the difference between two models and decided to try ner.eval-ab. However, no matter what model or source I pass, it returns no tasks.

The browser console shows this error.

Uncaught (in promise) TypeError: Cannot read property ‘getHostNode’ of null
at Object.getHostNode (bundle.js:1)
at c.getHostNode (bundle.js:1)
at Object.getHostNode (bundle.js:1)
at Object.updateChildren (bundle.js:1)
at h._reconcilerUpdateChildren (bundle.js:1)
at h._updateChildren (bundle.js:1)
at h.updateChildren (bundle.js:1)
at h._updateDOMChildren (bundle.js:1)
at h.updateComponent (bundle.js:1)
at h.receiveComponent (bundle.js:1)

2

Thanks for the report – and sorry for the not-so-great error handling on the front-end :sweat:

It’s possible that you came across a bug here. A common cause of front-end issues like this is if two incoming tasks have the same unique identifier (the _task_hash). This usually shouldn’t happen – but because the ner.eval-ab uses its own logic to compile the stream, this (or something similar) could be an explanation.

One thing you could try is print the first batch of the stream (usually 10) within the recipe, and check if the hashes and tasks it produces look reasonable:

stream = list(get_compare_questions(before_stream, after_stream, True))
print(stream[:10])
3

Maintainable javascript is always a struggle.

Here is the first record returned

{'id': 68,
 'input': {'text': '#### Service Provider'},
 'accept': {'text': '#### Service Provider', 'spans': []},
 'reject': {'text': '#### Service Provider', 'spans': [{'start': 5, 'end': 21, 'label': 'JOB'}]},
 'mapping': {'B': 'accept', 'A': 'reject'}}

All the first 10 records have unique ids. None of them have a _task_hash.

I believe that the error must be in get_compare_questions. The get_task method visible in the recipe assigns hashes and they are properly present in before_stream and after_stream. However, they are missing in the final stream list.

4

Thanks for testing this – I think we’re getting closer to the problem here!

The get_compare_questions is get_questions from recipes.compare. It’s true that this function removes the previously set hashes – and if no hashes are set on the stream, the controller will set them using util.set_hashes with the default settings (for comparison tasks like this, this means, the hash is based on the 'input' property).

Is it possible that two tasks in the batch have an identical input? And does setting hashes on the stream manually make a difference, e.g.:

stream = list(get_compare_questions(before_stream, after_stream, True))

def set_hashes_on_stream(stream):
    for i, task in enumerate(stream):
        task['_task_hash'] = i  # not so elegant, but whatever

stream = set_hashes_on_stream(stream)

I’ll also keep investigating this and will try to reproduce the error! While the unique identifiers are a likely explanation, it could also be that something else is wrong, or there’s indeed some bug on the front-end. (I’m looking forward to using a proper JSON schema for the task validation, as described here – this will hopefully make it easier to catch specific task formatting problems and show better warnings and errors in the app.)

5

None of the inputs appear to be duplicated. However, it doesn’t really matter. I tried adding your hash logic (plus a yield task). I also subbed in the hash logic from get_task (for both task and input hashes). These should have produced unique hashes, but the same error persists.

A standardized json schema would be nice. I’m used to service-oriented programming, so interfaces and contracts are my comfort zone.