Cache directory env vars ignored, causing failures when ~/.cache isn't usable #417
Description
Describe the bug
xet-core does not obey $CACHE_DIRECTORY or $XDG_CACHE_HOME, instead defaulting to $HOME/.cache/ . This goes very poorly when it (via hf_xet, at least) is used by services running under users with a non-writable home directory, like immich on NixOS: NixOS/nixpkgs#418799 . It also likely affects services using systemd.exec hardening tools like DynamicUser=true.
Beyond not working, the resulting error, Operation not permitted (os error 1), gives no information about the problem. I wasted quite a bit of time assuming the download destination wasn't accessible before finding the actual problem in strace.
When used from a normal user account, checking $XDG_CACHE_HOME before using a default ensures you respect the user's preferences. When used from a systemd service, $XDG_CACHE_HOME$CACHE_DIRECTORY is set based on CacheDirectory, so you'll much more reliably have a writable directory that's appropriate for cache instead of a home directory that may be nonexistent, read-only, or otherwise an inappropriate place for cache.
Combined with the existing logic, I think the ideal would be:
$HF_XET_CACHE- else
$HF_HOME/xet - else
$CACHE_DIRECTORY/huggingface/xet(new) - else
$XDG_CACHE_HOME/huggingface/xet(new) - else
$HOME/.cache/huggingface/xet - else
(PWD)/.cache/huggingface/xet
Reproduction
- Choose an application that uses xet-core, e.g. immich (with hf_xet)
- Create a system user with a home directory that isn't writeable or doesn't exist, e.g.
/var/empty - Create a service for the application, with
CacheDirectory=immichandUser=immich - Start the service
- See that downloading models fails with an error like below
Logs
Service log
Traceback (most recent call last):
File "/nix/store/10qwiqz9nlcgxs6zs32zli2kc5fwyg4d-python3.13-uvicorn-0.34.2/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self.scope, self.receive, self.send
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/10qwiqz9nlcgxs6zs32zli2kc5fwyg4d-python3.13-uvicorn-0.34.2/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/8i2ghdw9gfpj0gi28a1hbnpqhg1pba0i-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender) File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir
_download_to_tmp_and_move(
~~~~~~~~~~~~~~~~~~~~~~~~~^
incomplete_path=paths.incomplete_path(etag),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<8 lines>...
xet_file_data=xet_file_data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move
xet_get(
~~~~~~~^
incomplete_path=incomplete_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
displayed_filename=filename,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 627, in xet_get
download_files(
~~~~~~~~~~~~~~^
xet_download_info,
^^^^^^^^^^^^^^^^^^
...<3 lines>...
progress_updater=[progress_updater],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
RuntimeError: Data processing error: I/O error: Operation not permitted (os error 1)
[2025-07-17 00:41:15 +0000] [2550822] [INFO] Shutting down
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/nix/store/79bzydby7azc4ha9l02jfabzrykz3nab-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/nix/store/8i2ghdw9gfpj0gi28a1hbnpqhg1pba0i-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
)
^
File "/nix/store/8i2ghdw9gfpj0gi28a1hbnpqhg1pba0i-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 177, in predict
response = await run_inference(inputs, entries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 200, in run_inference
await asyncio.gather(*[_run_inference(entry) for entry in without_deps])
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 194, in _run_inference
model = await load(model)
^^^^^^^^^^^^^^^^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 238, in load
return await run(_load, model)
^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 213, in run
return await asyncio.get_running_loop().run_in_executor(thread_pool, partial_func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 225, in _load
model.load()
~~~~~~~~~~^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 51, in load
self.download()
~~~~~~~~~~~~~^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 44, in download
self._download()
~~~~~~~~~~~~~~^^
File "/nix/store/5rsj550gnbm52mnllqx11q2vw9qxgxj2-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 76, in _download
snapshot_download(
~~~~~~~~~~~~~~~~~^
f"immich-app/{clean_name(self.model_name)}",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<2 lines>...
ignore_patterns=ignored_patterns.get(self.model_format, []),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 327, in snapshot_download
thread_map(
~~~~~~~~~~^
_inner_hf_hub_download,
^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
tqdm_class=tqdm_class or hf_tqdm,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/355hmwi6b8da8liyq1xg7lcgxjq03nl0-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
File "/nix/store/355hmwi6b8da8liyq1xg7lcgxjq03nl0-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
File "/nix/store/355hmwi6b8da8liyq1xg7lcgxjq03nl0-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/std.py", line 1181, in __iter__
for obj in iterable:
^^^^^^^^
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/nix/store/djck7mx6jad1w0yy6zings96dyxanls6-python3-3.13.5/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 301, in _inner_hf_hub_download
return hf_hub_download(
repo_id,
...<15 lines>...
headers=headers,
)
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 988, in hf_hub_download
return _hf_hub_download_to_local_dir(
# Destination
...<15 lines>...
local_files_only=local_files_only,
)
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir
_download_to_tmp_and_move(
~~~~~~~~~~~~~~~~~~~~~~~~~^
incomplete_path=paths.incomplete_path(etag),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<8 lines>...
xet_file_data=xet_file_data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move
xet_get(
~~~~~~~^
incomplete_path=incomplete_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
displayed_filename=filename,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/rax4b0q521awq5yn5150gzxddgn3f6q0-python3.13-huggingface-hub-0.33.4/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 627, in xet_get
download_files(
~~~~~~~~~~~~~~^
xet_download_info,
^^^^^^^^^^^^^^^^^^
...<3 lines>...
progress_updater=[progress_updater],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
RuntimeError: Data processing error: I/O error: Operation not permitted (os error 1)
Excerpt from strace -vttfe trace=%file -o immich_ml.log -p $pid
mkdir("/var/empty/.cache/huggingface/xet/https___cas_serv-tGqkUaZf_CBPHQ6h", 0777) = -1 ENOENT (No such file or directory)
mkdir("/var/empty/.cache/huggingface/xet", 0777) = -1 ENOENT (No such file or directory)
mkdir("/var/empty/.cache/huggingface", 0777) = -1 ENOENT (No such file or directory)
mkdir("/var/empty/.cache", 0777) = -1 EPERM (Operation not permitted)
System info
- huggingface_hub version: 0.33.4
- Platform: Linux-6.12.34-hardened1-x86_64-with-glibc2.40
- Python version: 3.13.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /var/empty/.cache/huggingface/token
- Has saved token ?: False
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- hf_xet: 1.1.5
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /var/empty/.cache/huggingface/hub
- HF_ASSETS_CACHE: /var/empty/.cache/huggingface/assets
- HF_TOKEN_PATH: /var/empty/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /var/empty/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
no *HF_XET* env vars