"Task exception was never retrieved" error"

command run:

python3 -m prodigy pos.teach teachPOS_a es_udv25_spanishancora_trf ./data.json -U 

annotated about 5 sentences, tried to save, received this:
error:

Task exception was never retrieved
future: <Task finished name='Task-49' coro=<RequestResponseCycle.run_asgi() done, defined at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py:394> exception=NotImplementedError("Could not run 'aten::_amp_update_scale_' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_amp_update_scale_' is only available for these backends: [Meta, BackendSelect, Python, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, AutocastCPU, Autocast, Batched, VmapMode, Functionalize].\n\nMeta: registered at /Users/distiller/project/pytorch/build/aten/src/ATen/RegisterMeta.cpp:14951 [kernel]\nBackendSelect: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]\nPython: registered at /Users/distiller/project/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:47 [backend fallback]\nNamed: registered at /Users/distiller/project/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]\nConjugate: registered at /Users/distiller/project/pytorch/aten/src/ATen/ConjugateFallback.cpp:18 [backend fallback]\nNegative: registered at /Users/distiller/project/pytorch/aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]\nZeroTensor: registered at /Users/distiller/project/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]\nADInplaceOrView: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/ADInplaceOrViewType_0.cpp:2566 [kernel]\nAutogradOther: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradCPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradCUDA: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradXLA: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradLazy: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradXPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradMLC: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradHPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradNestedTensor: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradPrivateUse1: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradPrivateUse2: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nAutogradPrivateUse3: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]\nTracer: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/TraceType_2.cpp:12568 [kernel]\nAutocastCPU: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/autocast_mode.cpp:461 [backend fallback]\nAutocast: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/autocast_mode.cpp:305 [backend fallback]\nBatched: registered at /Users/distiller/project/pytorch/aten/src/ATen/BatchingRegistrations.cpp:1059 [backend fallback]\nVmapMode: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]\nFunctionalize: registered at /Users/distiller/project/pytorch/build/aten/src/ATen/RegisterFunctionalization_0.cpp:4018 [kernel]\n")>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 399, in run_asgi
    self.logger.error(msg, exc_info=exc)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/logging/__init__.py", line 1497, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/logging/__init__.py", line 1615, in _log
    self.handle(record)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/logging/__init__.py", line 1624, in handle
    if (not self.disabled) and self.filter(record):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/logging/__init__.py", line 821, in filter
    result = f.filter(record)
  File "cython_src/prodigy/util.pyx", line 145, in prodigy.util.ServerErrorFilter.filter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/prodigy/app.py", line 205, in reset_db_middleware
    response = await call_next(request)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/fastapi/routing.py", line 161, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/prodigy/app.py", line 494, in give_answers
    controller.receive_answers(
  File "cython_src/prodigy/core.pyx", line 234, in prodigy.core.Controller.receive_answers
  File "cython_src/prodigy/core.pyx", line 253, in prodigy.core.Controller.receive_answers
  File "cython_src/prodigy/models/tag.pyx", line 137, in prodigy.models.tag.Tagger.update
  File "cython_src/prodigy/models/tag.pyx", line 155, in prodigy.models.tag.Tagger._update
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/spacy/language.py", line 1164, in update
    proc.finish_update(sgd)
  File "spacy/pipeline/trainable_pipe.pyx", line 252, in spacy.pipeline.trainable_pipe.TrainablePipe.finish_update
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/thinc/model.py", line 323, in finish_update
    shim.finish_update(optimizer)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/thinc/shims/pytorch.py", line 119, in finish_update
    self._grad_scaler.update()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/thinc/shims/pytorch_grad_scaler.py", line 123, in update
    torch._amp_update_scale_(
NotImplementedError: Could not run 'aten::_amp_update_scale_' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_amp_update_scale_' is only available for these backends: [Meta, BackendSelect, Python, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, AutocastCPU, Autocast, Batched, VmapMode, Functionalize].

Meta: registered at /Users/distiller/project/pytorch/build/aten/src/ATen/RegisterMeta.cpp:14951 [kernel]
BackendSelect: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at /Users/distiller/project/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:47 [backend fallback]
Named: registered at /Users/distiller/project/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at /Users/distiller/project/pytorch/aten/src/ATen/ConjugateFallback.cpp:18 [backend fallback]
Negative: registered at /Users/distiller/project/pytorch/aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]
ZeroTensor: registered at /Users/distiller/project/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/ADInplaceOrViewType_0.cpp:2566 [kernel]
AutogradOther: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradCPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradCUDA: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradXLA: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradLazy: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradXPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradMLC: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradHPU: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradNestedTensor: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradPrivateUse1: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradPrivateUse2: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
AutogradPrivateUse3: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/VariableType_3.cpp:11380 [autograd kernel]
Tracer: registered at /Users/distiller/project/pytorch/torch/csrc/autograd/generated/TraceType_2.cpp:12568 [kernel]
AutocastCPU: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/autocast_mode.cpp:461 [backend fallback]
Autocast: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/autocast_mode.cpp:305 [backend fallback]
Batched: registered at /Users/distiller/project/pytorch/aten/src/ATen/BatchingRegistrations.cpp:1059 [backend fallback]
VmapMode: fallthrough registered at /Users/distiller/project/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
Functionalize: registered at /Users/distiller/project/pytorch/build/aten/src/ATen/RegisterFunctionalization_0.cpp:4018 [kernel]

bump

Hi Joe,

sorry for the late reply! It might be that you've found a bug, but just to be sure I'd like to double check a few things.

  1. Are you using the most recent version of Prodigy and spaCy? Could you share the versions? There have been a lot of updates to transformers over the years and it might be that an upgrade fixes everything.
  2. In case this doesn't fix it, could you share some of your examples? Although it's not very likely, it is possible that there's something about your dataset that's causing a mishap. The example you'll share will be used to try to reproduce the error locally.
  3. When I look at your error it seems PyTorch related. That is making me wonder, where did you get the es_udv25_spanishancora_trf model from? Is that something you've trained yourself?