🐛 Describe the bug

Problem seems to arise when loading a model using install_py_dep_per_model=true which will upgrade the setuptools version to latest 70.0 and thus breaking the handler loading.

Solution might be to replace the deprecated import in base handler from pkg_resources import packaging by import packaging

Error logs

2024-06-05T10:24:37,046 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Backend worker process died.
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/usr/lib64/python3.11/importlib/init.py", line 126, in import_module
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1149, in _find_and_load_unlocked
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/handler.py", line 1, in
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from handler_yolo import YoloHandler
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/handler_yolo.py", line 6, in
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from ts.torch_handler.object_detector import ObjectDetector
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/object_detector.py", line 8, in
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from .vision_handler import VisionHandler
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/vision_handler.py", line 16, in
2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from .base_handler import BaseHandler
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/base_handler.py", line 13, in
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from pkg_resources import packaging
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ImportError: cannot import name 'packaging' from 'pkg_resources' (/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/pkg_resources/init.py)
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG -
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - During handling of the above exception, another exception occurred:
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG -
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - worker.run_server()
2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 110, in load
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = self._load_default_handler(handler)
2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 159, in _load_default_handler
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name, "ts.torch_handler")
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/usr/lib64/python3.11/importlib/init.py", line 126, in import_module
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1128, in _find_and_load_unlocked
2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import
2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load
2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1142, in _find_and_load_unlocked
2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'ts.torch_handler.handler'
2024-06-05T10:24:37,061 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2024-06-05T10:24:37,062 [DEBUG] W-9000-crochet_yolov8s_512_240207_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-06-05T10:24:37,062 [DEBUG] W-9000-crochet_yolov8s_512_240207_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec

Installation instructions

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org pip==24.0 setuptools==69.5.1

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir numpy==1.24.3 cython==0.29.34 wheel==0.40.0 pillow==9.3.0 psutil==5.9.5 requests==2.31.0 captum==0.6.0 packaging==23.1 pynvml==11.4.1 pyyaml==6.0

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir torch==2.3.0+cpu torchvision==0.18.0+cpu torchtext==0.18.0 torchaudio==2.3.0+cpu --extra-index-url https://download.pytorch.org/whl/cpu

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir torchserve==0.11.0 torch-model-archiver==0.11.0 torch-workflow-archiver==0.2.13

mkdir /model-server
mkdir /model-server/model-store
mkdir /model-server/wf-store

Model Packaging

torch-model-archiver --model-name my_model_name --handler handler.py --extra-files index_to_name.json --serialized-file best.pt -r requirements.txt -v 1.0 --force

crash:

requirements.txt:
numpy

workaround:

requirements.txt:
numpy
setuptools==69.5.1

loader failing at

serve/ts/torch_handler/base_handler.py
l.13 from pkg_resources import packaging

config.properties

echo "
inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
grpc_inference_port=7070
grpc_management_port=7071
cors_allowed_origin=*
cors_allowed_methods=GET, POST, PUT, DELETE, OPTIONS
default_workers_per_model=2
job_queue_size=1000
model_store=/model-server/model-store
workflow_store=/model-server/wf-store
load_models=all
prefer_direct_buffer=true
install_py_dep_per_model=true
enable_metrics_api=true
prefer_direct_buffer=true
disable_system_metrics=false
metrics_mode=prometheus
cpu_launcher_enable=true
cpu_launcher_args=--use_logical_core
" > /model-server/config.properties

Versions

Environment headers

Torchserve branch:

torchserve==0.11.0
torch-model-archiver==0.11.0

Python version: 3.11 (64-bit runtime)
Python executable: /usr/bin/python3

Versions of relevant python libraries:
captum==0.6.0
numpy==1.24.3
psutil==5.9.5
requests==2.31.0
torch==2.3.0+cpu
torch-model-archiver==0.11.0
torch-workflow-archiver==0.2.13
torchaudio==2.3.0+cpu
torchserve==0.11.0
torchtext==0.18.0+cpu
torchvision==0.18.0+cpu
wheel==0.40.0
torch==2.3.0+cpu
torchtext==0.18.0+cpu
torchvision==0.18.0+cpu
torchaudio==2.3.0+cpu

Java Version:

OS: N/A
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
Clang version: N/A
CMake version: N/A

Environment:
library_path (LD_/DYLD_):

Repro instructions

torchserve --start --ts-config /model-server/config.properties

Possible Solution

Specify setuptool version in model.mar requirement.txt

workaround:

requirements.txt:
numpy
setuptools==69.5.1