Merge remote-tracking branch 'origin/main' into verify-checksum-cli

Author: hanouticelina

.github/workflows/python-quality.yml

@@ -41,6 +41,7 @@ jobs:
       - run: .venv/bin/python utils/check_static_imports.py
       - run: .venv/bin/python utils/check_all_variable.py
       - run: .venv/bin/python utils/generate_async_inference_client.py
+      - run: .venv/bin/python utils/generate_cli_reference.py --verbose
       - run: .venv/bin/python utils/generate_inference_types.py
       - run: .venv/bin/python utils/check_task_parameters.py
       - run: uvx ty check src

Makefile

@@ -11,6 +11,7 @@ quality:
 	python utils/check_static_imports.py
 	python utils/check_all_variable.py
 	python utils/generate_async_inference_client.py
+	python utils/generate_cli_reference.py
 
 	ty check src
 
@@ -20,6 +21,7 @@ style:
 	python utils/check_static_imports.py --update
 	python utils/check_all_variable.py --update
 	python utils/generate_async_inference_client.py --update
+	python utils/generate_cli_reference.py --update
 
 inference_check:
 	python utils/generate_inference_types.py

docs/source/en/_toctree.yml

@@ -58,6 +58,8 @@
       title: Environment variables
     - local: package_reference/hf_api
       title: Hugging Face Hub API
+    - local: package_reference/cli
+      title: CLI
     - local: package_reference/file_download
       title: Downloading files
     - local: package_reference/mixins

docs/source/en/concepts/migration.md

@@ -45,6 +45,10 @@ The migration from `requests` to `httpx` brings several key improvements that en
 
 The transition to `httpx` positions `huggingface_hub` with a modern, efficient, and maintainable HTTP backend. While most users should experience seamless operation, the underlying improvements provide better performance and reliability for all Hub interactions.
 
+## `hf_transfer`
+
+Now that all repositories on the Hub are Xet-enabled and that `hf_xet` is the default way to download/upload files, we've removed support for the `hf_transfer` optional package. The `HF_HUB_ENABLE_HF_TRANSFER` environment variable is therefore ignored. Use [`HF_XET_HIGH_PERFORMANCE`](../package_reference/environment_variables.md) instead.
+
 ## `Repository` class
 
 The `Repository` class has been removed in v1.0. It was a thin wrapper around the `git` CLI for managing repositories. You can still use `git` directly in the terminal, but the recommended approach is to use the HTTP-based API in the `huggingface_hub` library for a smoother experience, especially when dealing with large files.

docs/source/en/guides/cli.md

@@ -6,6 +6,10 @@ rendered properly in your Markdown viewer.
 
 The `huggingface_hub` Python package comes with a built-in CLI called `hf`. This tool allows you to interact with the Hugging Face Hub directly from a terminal. For example, you can login to your account, create a repository, upload and download files, etc. It also comes with handy features to configure your machine or manage your cache. In this guide, we will have a look at the main features of the CLI and how to use them.
 
+> [!TIP]
+> This guide covers the most important features of the `hf` CLI.
+> For a complete reference of all commands and options, see the [CLI reference](../package_reference/cli.md).
+
 ## Getting started
 
 First of all, let's install the CLI:
@@ -15,7 +19,7 @@ First of all, let's install the CLI:
 ```
 
 > [!TIP]
-> The CLI ships with the core `huggingface_hub` package, so there is no longer a `[cli]` extra to install.
+> The CLI ships with the core `huggingface_hub` package.
 
 Alternatively, you can install the `hf` CLI with a single command:
 
@@ -778,39 +782,36 @@ The `hf env` command prints details about your machine setup. This is useful whe
 
 Copy-and-paste the text below in your GitHub issue.
 
-- huggingface_hub version: 0.19.0.dev0
-- Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
-- Python version: 3.10.12
+- huggingface_hub version: 1.0.0.rc6
+- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
+- Python version: 3.11.14
 - Running in iPython ?: No
 - Running in notebook ?: No
 - Running in Google Colab ?: No
+- Running in Google Colab Enterprise ?: No
 - Token path ?: /home/wauplin/.cache/huggingface/token
 - Has saved token ?: True
 - Who am I ?: Wauplin
 - Configured git credential helpers: store
-- FastAI: N/A
-- Torch: 1.12.1
-- Jinja2: 3.1.2
-- Graphviz: 0.20.1
-- Pydot: 1.4.2
-- Pillow: 9.2.0
-- hf_transfer: 0.1.3
-- gradio: 4.0.2
-- tensorboard: 2.6
-- numpy: 1.23.2
-- pydantic: 2.4.2
-- aiohttp: 3.8.4
+- Installation method: unknown
+- Torch: N/A
+- httpx: 0.28.1
+- hf_xet: 1.1.10
+- gradio: 5.41.1
+- tensorboard: N/A
+- pydantic: 2.11.7
 - ENDPOINT: https://huggingface.co
 - HF_HUB_CACHE: /home/wauplin/.cache/huggingface/hub
 - HF_ASSETS_CACHE: /home/wauplin/.cache/huggingface/assets
 - HF_TOKEN_PATH: /home/wauplin/.cache/huggingface/token
+- HF_STORED_TOKENS_PATH: /home/wauplin/.cache/huggingface/stored_tokens
 - HF_HUB_OFFLINE: False
 - HF_HUB_DISABLE_TELEMETRY: False
 - HF_HUB_DISABLE_PROGRESS_BARS: None
 - HF_HUB_DISABLE_SYMLINKS_WARNING: False
 - HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
 - HF_HUB_DISABLE_IMPLICIT_TOKEN: False
-- HF_HUB_ENABLE_HF_TRANSFER: False
+- HF_HUB_DISABLE_XET: False
 - HF_HUB_ETAG_TIMEOUT: 10
 - HF_HUB_DOWNLOAD_TIMEOUT: 10
 ```

docs/source/en/guides/download.md

@@ -243,13 +243,6 @@ Finally, you can also make a dry-run programmatically by passing `dry_run=True`
 
 ## Faster downloads
 
-There are two options to speed up downloads. Both involve installing a Python package written in Rust.
-
-* `hf_xet` is newer and uses the Xet storage backend for upload/download. Xet storage is the [default for all new Hub users and organizations](https://huggingface.co/changelog/xet-default-for-new-users), and is in the process of being rolled out to all users. If you don't have access, join the [waitlist](https://huggingface.co/join/xet) to make Xet the default for all your repositories!
-* `hf_transfer` is a power-tool to download and upload to our LFS storage backend (note: this is less future-proof than Xet). It is thoroughly tested and has been in production for a long time, but it has some limitations. 
-
-### hf_xet
-
 Take advantage of faster downloads through `hf_xet`, the Python binding to the [`xet-core`](https://github.com/huggingface/xet-core) library that enables 
 chunk-based deduplication for faster downloads and uploads. `hf_xet` integrates seamlessly with `huggingface_hub`, but uses the Rust `xet-core` library and Xet storage instead of LFS.
 
@@ -263,23 +256,6 @@ pip install -U "huggingface_hub"
 
 As of `huggingface_hub` 0.32.0, this will also install `hf_xet`.
 
-Note: `hf_xet` will only be utilized when the files being downloaded are being stored with Xet Storage.
-
 All other `huggingface_hub` APIs will continue to work without any modification. To learn more about the benefits of Xet storage and `hf_xet`, refer to this [section](https://huggingface.co/docs/hub/storage-backends).
 
-### hf_transfer
-
-If you are running on a machine with high bandwidth,
-you can increase your download speed with [`hf_transfer`](https://github.com/huggingface/hf_transfer),
-a Rust-based library developed to speed up file transfers with the Hub.
-To enable it:
-
-1. Specify the `hf_transfer` extra when installing `huggingface_hub`
-   (e.g. `pip install huggingface_hub[hf_transfer]`).
-2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
-
-> [!WARNING]
-> `hf_transfer` is a power user tool!
-> It is tested and production-ready,
-> but it lacks user-friendly features like advanced error handling or proxies.
-> For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer).
+Note: `hf_transfer` was formerly used with the LFS storage backend and is now deprecated; use `hf_xet` instead.