Integrating MedSAM with Prodigy for Segmentation in Endoscopic Videos

Dear Support Team,

I'm excited to see that SAM has been successfully integrated with Prodigy—excellent work!

I am currently exploring the possibility of integrating MedSAM, the medical variant of SAM, for enhanced labelling capabilities. MedSAM, detailed here: GitHub - bowang-lab/MedSAM: Segment Anything in Medical Images, offers features specifically tailored to medical applications. I'm particularly interested in its potential for object segmentation in endoscopic videos when combined with Prodigy.

Could you provide any guidance or suggestions on how to achieve this integration? Any assistance would be highly valued.

Thank you very much for your support.

Warm regards,


Hi @nlp-guy ,

Thanks for the nice words :slight_smile:
MedSam (at least medsam_vit_b.pth checkpoint) should pretty much work out of the box?
I have successfully run it by specifying the model's registry id as vit_b and providing the path to the pre-downloaded checkpoint like so:

python -m prodigy segment.image.manual test_segment ./images medsam_vit_b.pth --model-type vit_b --label A,B -R

If you are on CPU, you would just need to modify the loading method here: prodigy-segment/prodigy_segment/segment_anything/ at 05dac5778c6cd157d0f2d1ced9c7ef9d406b7cff · explosion/prodigy-segment · GitHub to instruct it to use the CPU:

torch.load(f,  map_location=torch.device('cpu'))

Thanks @magdaaniol for the guidance. So excited to see MedSAM can be available so easily.

When I run the command, I am getting the following error:

python -m prodigy segment.image.manual test_segment ./frames medsam_vit_b.pth --model-type medsam_vit_b --label A,B -R
2024-05-19 00:10:27.176424: I tensorflow/core/util/] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-19 00:10:27.212225: I tensorflow/core/platform/] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-19 00:10:27.771976: W tensorflow/compiler/tf2tensorrt/utils/] TF-TRT Warning: Could not find TensorRT
2024-05-19 00:10:28.593596: W tensorflow/core/common_runtime/gpu/] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

✘ Can't find recipe or command 'segment.image.manual'.
Run prodigy --help to see available options. If you're using a custom recipe,
provide the path to the Python file using the -F argument.

Can you please guide me how to resolve this?

Many thanks and
Kind regards,

Hi @nlp-guy,

It looks like the Prodigy Segment plugin has not been installed in your current environment?
To install run:

python -m pip install "prodigy-segment @ git+"

That should make the command available.

Thank you @magdaaniol.

I am getting the following error when trying to install the plugin:

python -m pip install "prodigy-segment@git+"
Collecting prodigy-segment@ git+
  Cloning to /tmp/pip-install-i4d5bflx/prodigy-segment_a661c071dd8e4f4ab39828aca6663bba
  Running command git clone --filter=blob:none --quiet /tmp/pip-install-i4d5bflx/prodigy-segment_a661c071dd8e4f4ab39828aca6663bba
  Resolved to commit 05dac5778c6cd157d0f2d1ced9c7ef9d406b7cff
  Preparing metadata ( ... done
Collecting opencv-python== (from prodigy-segment@ git+
  Using cached opencv_python- (19 kB)
Collecting pycocotools==2.0.7 (from prodigy-segment@ git+
  Using cached pycocotools-2.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.1 kB)
Collecting matplotlib==3.8.1 (from prodigy-segment@ git+
  Using cached matplotlib-3.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.8 kB)
Collecting onnxruntime==1.16.1 (from prodigy-segment@ git+
  Using cached onnxruntime-1.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting onnx==1.15.0 (from prodigy-segment@ git+
  Using cached onnx-1.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (15 kB)
Collecting torch==2.1.0 (from prodigy-segment@ git+
  Using cached torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)
Collecting torchvision==0.16.0 (from prodigy-segment@ git+
  Using cached torchvision-0.16.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting diskcache==5.6.3 (from prodigy-segment@ git+
  Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: contourpy>=1.0.1 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (1.0.6)
Requirement already satisfied: cycler>=0.10 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (4.38.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (1.4.4)
Requirement already satisfied: numpy<2,>=1.21 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (1.24.3)
Requirement already satisfied: packaging>=20.0 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (21.3)
Requirement already satisfied: pillow>=8 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (9.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from matplotlib==3.8.1->prodigy-segment@ git+ (2.8.1)
Requirement already satisfied: protobuf>=3.20.2 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from onnx==1.15.0->prodigy-segment@ git+ (3.20.3)
Collecting coloredlogs (from onnxruntime==1.16.1->prodigy-segment@ git+
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: flatbuffers in /home/big-deal/mambaforge/lib/python3.10/site-packages (from onnxruntime==1.16.1->prodigy-segment@ git+ (23.5.26)
Requirement already satisfied: sympy in /home/big-deal/mambaforge/lib/python3.10/site-packages (from onnxruntime==1.16.1->prodigy-segment@ git+ (1.12)
Requirement already satisfied: filelock in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (3.8.0)
Requirement already satisfied: typing-extensions in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (4.5.0)
Requirement already satisfied: networkx in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (2.8.8)
Requirement already satisfied: jinja2 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (3.1.2)
Requirement already satisfied: fsspec in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (2022.10.0)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Requirement already satisfied: nvidia-cudnn-cu12== in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (
Collecting nvidia-cublas-cu12== (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cublas_cu12- (1.5 kB)
Collecting nvidia-cufft-cu12== (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cufft_cu12- (1.5 kB)
Collecting nvidia-curand-cu12== (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_curand_cu12- (1.5 kB)
Collecting nvidia-cusolver-cu12== (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cusolver_cu12- (1.6 kB)
Collecting nvidia-cusparse-cu12== (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_cusparse_cu12- (1.6 kB)
Collecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Requirement already satisfied: triton==2.1.0 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torch==2.1.0->prodigy-segment@ git+ (2.1.0)
Requirement already satisfied: requests in /home/big-deal/mambaforge/lib/python3.10/site-packages (from torchvision==0.16.0->prodigy-segment@ git+ (2.31.0)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==>torch==2.1.0->prodigy-segment@ git+
  Using cached nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Requirement already satisfied: six>=1.5 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib==3.8.1->prodigy-segment@ git+ (1.16.0)
Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime==1.16.1->prodigy-segment@ git+
  Using cached humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from jinja2->torch==2.1.0->prodigy-segment@ git+ (2.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from requests->torchvision==0.16.0->prodigy-segment@ git+ (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from requests->torchvision==0.16.0->prodigy-segment@ git+ (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from requests->torchvision==0.16.0->prodigy-segment@ git+ (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from requests->torchvision==0.16.0->prodigy-segment@ git+ (2023.7.22)
Requirement already satisfied: mpmath>=0.19 in /home/big-deal/mambaforge/lib/python3.10/site-packages (from sympy->onnxruntime==1.16.1->prodigy-segment@ git+ (1.3.0)
Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)
Using cached matplotlib-3.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
Using cached onnx-1.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.7 MB)
Using cached onnxruntime-1.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.2 MB)
Using cached opencv_python- (61.7 MB)
Using cached pycocotools-2.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (426 kB)
Using cached torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
Using cached torchvision-0.16.0-cp310-cp310-manylinux1_x86_64.whl (6.9 MB)
Using cached nvidia_cublas_cu12- (410.6 MB)
Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
Using cached nvidia_cufft_cu12- (121.6 MB)
Using cached nvidia_curand_cu12- (56.5 MB)
Using cached nvidia_cusolver_cu12- (124.2 MB)
Using cached nvidia_cusparse_cu12- (196.0 MB)
Using cached nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Using cached nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl (21.3 MB)
Building wheels for collected packages: prodigy-segment
  Building wheel for prodigy-segment ( ... done
  Created wheel for prodigy-segment: filename=prodigy_segment-0.1.1-py2.py3-none-any.whl size=39468 sha256=d66a058ee679efdbe953fe95b5833fc1d85b8fb0e2f722aa282a567ad055cdc2
  Stored in directory: /tmp/pip-ephem-wheel-cache-ks6_pczh/wheels/a8/b1/a3/eefa2a1def1f024f832d5214b9c3b5a01227521b396c15b680
Successfully built prodigy-segment
Installing collected packages: opencv-python, onnx, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, humanfriendly, diskcache, nvidia-cusparse-cu12, matplotlib, coloredlogs, pycocotools, onnxruntime, nvidia-cusolver-cu12, torch, torchvision, prodigy-segment
  Attempting uninstall: opencv-python
    Found existing installation: opencv-python 4.5.5
ERROR: Cannot uninstall opencv-python 4.5.5, RECORD file not found. Hint: The package was installed by conda.

Is there a way to resolve this error?
Many thanks and
Kind regards,

Oh, it looks like it's impossible to uninstall the older version of one of the dependencies because it's been installed with conda (not pip). Could you try installing Prodigy and the Prodigy-Segment plugin in a fresh virtual environment.
That is:
In your project folder (create a fresh one too if possible):

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install prodigy -f
python -m pip install "prodigy-segment @ git+"

Alternatively, you could also try to uninstall opencv-python using conda so conda uninstall opencv-python but not sure if it's perhaps needed somwhere else in your setup. This is why I would strongly recommend starting with a fresh virtual environment for a project with Prodigy and Prodigy-Segment. That would be the cleanest way for sure.

Dear @magdaaniol
I have tried as you proposed to install it in virtual environment. It installed smoothly however, you I am getting the following error:

python -m prodigy segment.image.manual test_segment ./frames medsam_vit_b.pth --model-type medsam_vit_b --label A,B -R
Traceback (most recent call last):
  File "/home/big-deal/mambaforge/lib/python3.10/", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/big-deal/mambaforge/lib/python3.10/", line 86, in _run_code
    exec(code, run_globals)
  File "/home/big-deal/mlworks/.venv/lib/python3.10/site-packages/prodigy/", line 53, in <module>
  File "/home/big-deal/mlworks/.venv/lib/python3.10/site-packages/catalogue/", line 125, in get_entry_points
    result[] = entry_point.load()
  File "/home/big-deal/mambaforge/lib/python3.10/importlib/metadata/", line 171, in load
    module = import_module('module'))
  File "/home/big-deal/mambaforge/lib/python3.10/importlib/", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/big-deal/mlworks/.venv/lib/python3.10/site-packages/prodigy_segment/", line 12, in <module>
    from prodigy.core import Arg, recipe, Controller
ImportError: cannot import name 'Arg' from 'prodigy.core' (/home/big-deal/mlworks/.venv/lib/python3.10/site-packages/prodigy/

Can you please guide me on how to resolve this?

hmm it looks like Prodigy and the plugin are both in the same .venv now (/home/big-deal/mlworks/.venv/
With this virtual environment activated could you run:

python -m pip list | grep prodigy

and share the output with me? Thanks!

I get the following output:

(.venv) (base) big-deal@~/mlworks$ python -m pip list | grep prodigy
prodigy                  1.12.7
prodigy-segment          0.1.1

Right, I'm afraid you'll need Prodigy >= 1.14.0 to be able to use the plugin. In 1.14.0 we made significant update to the internals which includes moving to radicli for CLI management (hence the import error). This alone would be fairly easy to "undo" and make the plugin code backward compatible but there are also changes to the types which are more involved.
I'll pin Prodigy version on the plugin to avoid this in future.

I removed the .venv and then recreated it to install 1.14.0 version of Prodigy using the following command:

python -m pip install prodigy>=1.14.0 -f

But I still get 1.12.7 version.

Could you plz guide what shall I do to install the latest version of Prodigy.

Hi @nlp-guy,

It might be that your license has expired? If you purchased more than a year ago than it won't let you download the versions that are newer than the ones released within a year from your purchase :frowning:
You can check which Prodigy versions you're license is entitled to by accessing the download URL with your browser:

Just a comment on your command, you probably should put the version in quotes so:

python -m pip install "prodigy>=1.14.0" -f

In any case the following command will get you the highest version your license permits:

python -m pip install prodigy --upgrade -f

If your license is indeed expired, I guess the only option left is to get the 12-month extension, which you can obtain here.