Skip to content

Registry.md

Latest commit

 

History

History
358 lines (307 loc) · 10.4 KB

Registry.md

File metadata and controls

358 lines (307 loc) · 10.4 KB

Local Container Registry and mirrors for Jetson (here: on MACOS platform)

Is this plausible on macOS?

Yes — if the macOS machine is only used as a network-accessible registry service (local registry + pull-through cache) for your Jetson devices.

What this does:

  • Caches image layers on your LAN so Jetsons pull faster and you reduce external registry traffic.
  • Lets you docker push locally-built images into a LAN registry (registry.local:5555) and pull them from Jetsons.

What this does not do:

  • It does not make Jetson/L4T GPU images runnable on macOS (even on Apple Silicon). This setup is about pulling and caching, not running.

Remark on naming: registry.local is used as a convenient hostname. Jetsons must resolve it to your Mac's LAN IP (not 127.0.0.1).

Add folders

mkdir ~/docker/registry
mkdir ~/docker/mirror_docker_io
mkdir ~/docker/mirror_nvcr_io
mkdir ~/docker/mirror_ghcr_io
mkdir ~/docker/certs
mkdir ~/docker/config

Add DNS entry to hosts on MACOS

sudo nano /etc/hosts

add 127.0.0.1 registry.local and/or 127.0.0.1 mirror.local

Generate certificate (use newer openssl, to get MACOS compatible certificates)

brew install openssl
cd ~/docker/certs

# this did not work:  
# openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 3650 -out domain.crt

/opt/homebrew/opt/openssl@3.4/bin/openssl req -x509 -nodes -newkey rsa:2048 \
  -keyout domain.key -out domain.crt \
  -subj "/CN=registry.local" \
  -addext "subjectAltName=DNS:registry.local" \
  -days 3650

# remark:
# generate second certificate for 'mirror.local' if needed
# subsequent steps have to be adapted in this case

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/docker/certs/domain.crt

each registry and mirror needs a separate entry (5001: docker.io, 5002: nvcr.io, 5003: ghcr.io, 5555: registry)

mkdir -p ~/.docker/certs.d/registry.local:5001
mkdir -p ~/.docker/certs.d/registry.local:5002
mkdir -p ~/.docker/certs.d/registry.local:5003
mkdir -p ~/.docker/certs.d/registry.local:5555
cp domain.crt ~/.docker/certs.d/registry.local:5001/ca.crt
cp domain.crt ~/.docker/certs.d/registry.local:5002/ca.crt
cp domain.crt ~/.docker/certs.d/registry.local:5003/ca.crt
cp domain.crt ~/.docker/certs.d/registry.local:5555/ca.crt

deprecated (not officially supported)

mkdir -p ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5001
mkdir -p ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5002
mkdir -p ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5003
mkdir -p ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5555
cp domain.crt ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5001/ca.crt
cp domain.crt ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5002/ca.crt
cp domain.crt ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5003/ca.crt
cp domain.crt ~/Library/Group\ Containers/group.com.docker/certs.d/registry.local:5555/ca.crt

Registry container

create config_registry.yml in ~/docker/config

version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5555
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key

start the container

docker run -d \
  --name registry \
  -p 5555:5555 \
  --restart=always \
  -v ~/docker/config/config_registry.yml:/etc/docker/registry/config.yml:ro \
  -v ~/docker/certs/domain.crt:/certs/domain.crt:ro \
  -v ~/docker/certs/domain.key:/certs/domain.key:ro \
  -v ~/docker/registry:/var/lib/registry \
  -e REGISTRY_STORAGE_DELETE_ENABLED=true \
  registry:2

Mirror container (one for each external registry)

create generic config_mirror.yml in ~/docker/config

version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
proxy:
  remoteurl:

start the container:

  • for docker.io:
docker run -d \
  --name mirror_docker_io \
  -p 5001:5000 \
  --restart=always \
  -v ~/docker/config/config_mirror.yml:/etc/docker/registry/config.yml:ro \
  -v ~/docker/certs/domain.crt:/certs/domain.crt:ro \
  -v ~/docker/certs/domain.key:/certs/domain.key:ro \
  -v ~/docker/mirror_docker_io:/var/lib/registry \
  -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
  registry:2
  • for nvcr.io (use real username and password):
docker run -d \
  --name mirror_nvcr_io \
  -p 5002:5000 \
  --restart=always \
  -v ~/docker/config/config_mirror.yml:/etc/docker/registry/config.yml:ro \
  -v ~/docker/certs/domain.crt:/certs/domain.crt:ro \
  -v ~/docker/certs/domain.key:/certs/domain.key:ro \
  -v ~/docker/mirror_nvcr_io:/var/lib/registry \
  -e REGISTRY_PROXY_REMOTEURL="https://nvcr.io" \
  -e REGISTRY_PROXY_USERNAME="$oauthtoken" \
  -e REGISTRY_PROXY_PASSWORD="MTZ..." \
  registry:2
  • for ghcr.io (GitHub Container Registry):
docker run -d \
  --name mirror_ghcr_io \
  -p 5003:5000 \
  --restart=always \
  -v ~/docker/config/config_mirror.yml:/etc/docker/registry/config.yml:ro \
  -v ~/docker/certs/domain.crt:/certs/domain.crt:ro \
  -v ~/docker/certs/domain.key:/certs/domain.key:ro \
  -v ~/docker/mirror_ghcr_io:/var/lib/registry \
  -e REGISTRY_PROXY_REMOTEURL="https://ghcr.io" \
  registry:2

If you need to cache private GHCR images, add credentials (recommended: a GitHub PAT with read:packages):

-e REGISTRY_PROXY_USERNAME="<github-username>" \
-e REGISTRY_PROXY_PASSWORD="<github-pat>" \

Docker on MACOS

Modify daemon.json and restart via UI:

{
  "insecure-registries": [],
  "registry-mirrors": [
    "https://registry.local:5001"
  ]
}

Remark: Docker's registry-mirrors setting applies to docker.io (Docker Hub). For nvcr.io / ghcr.io, pull via the mirror endpoint explicitly (examples below) to populate the cache.

Test

Check api endpoints for valid certificate:

cd ~/docker/certs
curl -v --cacert domain.crt https://registry.local:5001/v2/
curl -v --cacert domain.crt https://registry.local:5002/v2/
curl -v --cacert domain.crt https://registry.local:5003/v2/
curl -v --cacert domain.crt https://registry.local:5555/v2/

Pull/push test images (use library as namespace for docker.io):

docker pull registry.local:5001/library/hello-world:latest
docker pull registry.local:5002/nvidia/l4t-base:r36.2.0
docker pull registry.local:5003/nvidia-ai-iot/vllm:latest-jetson-thor

docker tag hello-world registry.local:5555/hello-world
docker push registry.local:5555/hello-world

docker pull registry.local:5555/hello-world

Jetson

Add DNS entry to hosts:

sudo nano /etc/hosts

Insert one or two entries:

w.x.y.z registry.local

Copy crt file e.g. to git folder by using VSCode Remote and register ca cert for each endpoint:

sudo mkdir -p /etc/docker/certs.d/registry.local:5001
sudo mkdir -p /etc/docker/certs.d/registry.local:5002
sudo mkdir -p /etc/docker/certs.d/registry.local:5003
sudo mkdir -p /etc/docker/certs.d/registry.local:5555
sudo cp domain.crt /etc/docker/certs.d/registry.local:5001/ca.crt
sudo cp domain.crt /etc/docker/certs.d/registry.local:5002/ca.crt
sudo cp domain.crt /etc/docker/certs.d/registry.local:5003/ca.crt
sudo cp domain.crt /etc/docker/certs.d/registry.local:5555/ca.crt
sudo chmod 644 /etc/docker/certs.d/registry.local:5001/ca.crt
sudo chmod 644 /etc/docker/certs.d/registry.local:5002/ca.crt
sudo chmod 644 /etc/docker/certs.d/registry.local:5003/ca.crt
sudo chmod 644 /etc/docker/certs.d/registry.local:5555/ca.crt

Modify daemon.json and restart docker:

sudo nano /etc/docker/daemon.json
sudo systemctl restart docker

daemon.json content:

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia",
    "data-root": "/mnt/nova_ssd/docker",
    "registry-mirrors": [
        "https://registry.local:5001"
    ]
}

Remark:
Only docker.io images are cached by the proxy automatically. nvcr.io images for example have to be loaded using registry.local:5002/nvidia/<image>:<tag> and are then getting cached.
ghcr.io images have to be loaded using registry.local:5003/<org-or-user>/<image>:<tag> and are then getting cached. Example (use mirror instead of GHCR directly):
docker pull registry.local:5003/nvidia-ai-iot/vllm:latest-jetson-thor

If you meant hcr.io/...: that is likely a typo for ghcr.io/... (GitHub Container Registry).

Push locally built images to registry:

docker tag abc registry.local:5555/abc
docker push registry.local:5555/abc

docker tag faiss:r36.4.0-cu126 registry.local:5555/faiss:r36.4.0-cu126
docker push registry.local:5555/faiss:r36.4.0-cu126

Result on MACOS

Filesystem:

docker
├── certs
│   ├── domain.crt
│   └── domain.key
├── config
│   ├── config_mirror.yml
│   └── config_registry.yml
├── mirror_docker_io
├── mirror_ghcr_io
├── mirror_nvcr_io
└── registry

Docker tree:

Docker Registry Tree

Tag, push, copy images

Tagging, pushing and copying to another registry (here: Azure CR) using skopeo.

Install skopeo on Linux:

sudo apt install skopeo -y

Install skopeo on macOS:

brew install skopeo

Create and start the script:

#!/usr/bin/env bash

IMAGES_REGISTRY=(
    "image1"
    "image2"
)
IMAGES_REGISTRY_SKOPEO=(
    "image1"
    "image2"
)

ACR_NAME="<acr-name>.azurecr.io"
ACR_USER="<username>"
ACR_PASS="<password>"
skopeo login "$ACR_NAME" --username "$ACR_USER" --password "$ACR_PASS"

for IMAGE in "${IMAGES_REGISTRY[@]}"; do
    docker tag "$IMAGE" registry.local:5555/"$IMAGE"
    docker push registry.local:5555/"$IMAGE"
done

for IMAGE in "${IMAGES_REGISTRY_SKOPEO[@]}"; do
    skopeo copy docker://registry.local:5555/"$IMAGE":latest docker://"$ACR_NAME"/"$IMAGE":latest
done

echo "Done."

On macOS:

docker login <acr-name>.azurecr.io --username <acr-username> --password <acr-password>

skopeo login <acr-name>.azurecr.io --username <acr-username> --password <acr-password>
skopeo copy docker://registry.local:5555/<image>:latest docker://<acr-name>.azurecr.io/<image>:latest