I am trying to run and annotate with fetch command with prodigy llms on Databricks but there is a transformers error. I tried to use poetry because I thought I had issues with dependencies but prodigy doesn't see transformers even though they are installed.
command:
poetry run prodigy ner.llm.fetch config.cfg input_texts.csv output_texts.jsonl
I get the follwing error after running:
2023-10-03 06:44:59.028731: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
/databricks/python/lib/python3.10/site-packages/scipy/__init__.py:155: UserWarning: A NumPy version >=1.18.5 and <1.25.0 is required for this version of SciPy (detected version 1.25.2
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/prodigy/__main__.py", line 63, in <module>
controller = recipe(*args, use_plac=True)
File "cython_src/prodigy/core.pyx", line 883, in prodigy.core.recipe.recipe_decorator.recipe_proxy
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/plac_core.py", line 367, in call
cmd, result = parser.consume(arglist)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/plac_core.py", line 232, in consume
return cmd, self.func(*(args + varargs + extraopts), **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/prodigy/recipes/llm/ner.py", line 134, in llm_fetch_ner
nlp = assemble(config_path, overrides=config_overrides)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/util.py", line 49, in assemble
return assemble_from_config(config)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/util.py", line 29, in assemble_from_config
nlp = load_model_from_config(config, auto_fill=True)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy/util.py", line 587, in load_model_from_config
nlp = lang_cls.from_config(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy/language.py", line 1848, in from_config
nlp.add_pipe(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy/language.py", line 814, in add_pipe
pipe_component = self.create_pipe(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy/language.py", line 702, in create_pipe
resolved = registry.resolve(cfg, validate=validate)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/confection/__init__.py", line 756, in resolve
resolved, _ = cls._make(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/confection/__init__.py", line 805, in _make
filled, _, resolved = cls._fill(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/confection/__init__.py", line 860, in _fill
filled[key], validation[v_key], final[key] = cls._fill(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/confection/__init__.py", line 877, in _fill
getter_result = getter(*args, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/models/hf/falcon.py", line 79, in falcon_hf
return Falcon(name=name, config_init=config_init, config_run=config_run)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/models/hf/falcon.py", line 23, in __init__
super().__init__(name=name, config_init=config_init, config_run=config_run)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/models/hf/base.py", line 37, in __init__
HuggingFace.check_installation()
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-4f9c0c67-1213-4f0f-8220-29e6affd48ae/lib/python3.10/site-packages/spacy_llm/models/hf/base.py", line 78, in check_installation
raise ValueError(
ValueError: The HF model requires `transformers` to be installed, which it is not. See https://huggingface.co/docs/transformers/installation for installation instructions.
In my config.cfg file am using the falcon-40b-instruct model
[components.llm.model]
@llm_models = spacy.Falcon.v1
name = falcon-40b-instruct