docker hub build failure due to package version issue

[homework36]

We will not be able to make PR or merge or build new images until this is fixed...
Also this has to be tested on an intel machine.

2025-03-05T00:39:55Z #12 26.43 ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
2025-03-05T00:39:55Z #12 26.43     tensorflow==2.5.1 from https://files.pythonhosted.org/packages/63/b2/2822fad63bdaedf3da841443daf92775d6fff16aba7199415e06f8d8c50f/tensorflow-2.5.1-cp37-cp37m-manylinux2010_x86_64.whl (from calamari_ocr->-r ./text_alignment/requirements.txt (line 2)):
2025-03-05T00:39:55Z #12 26.43         Expected sha256 5cebb9a548adfd5fdc6081df72db93488afbe02a60c4f7c60b94930d6b3fbfa6
2025-03-05T00:39:55Z #12 26.43              Got        a0b27dae3ef99392814cb831e0115ac94947783f3513511201b092f072d79a28
2025-03-05T00:39:55Z #12 26.43 
2025-03-05T00:39:55Z #12 26.44 
2025-03-05T00:39:55Z #12 26.44 [notice] A new release of pip is available: 23.0.1 -> 24.0
2025-03-05T00:39:55Z #12 26.44 [notice] To update, run: pip install --upgrade pip
2025-03-05T00:39:55Z #12 ERROR: executor failed running [/bin/sh -c /opt/install_gpu_rodan_jobs]: exit code: 1
2025-03-05T00:39:55Z ------
2025-03-05T00:39:55Z  > [ 8/13] RUN /opt/install_gpu_rodan_jobs:
2025-03-05T00:39:55Z #12 18.58   Downloading tensorflow-2.5.1-cp37-cp37m-manylinux2010_x86_64.whl (454.4 MB)
2025-03-05T00:39:55Z #12 26.01      ━━━━━━━━━━━━━━━━━━━━━━━━━━╸            319.8/454.4 MB 22.0 MB/s eta 0:00:07
2025-03-05T00:39:55Z #12 26.43 ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
2025-03-05T00:39:55Z #12 26.43     tensorflow==2.5.1 from https://files.pythonhosted.org/packages/63/b2/2822fad63bdaedf3da841443daf92775d6fff16aba7199415e06f8d8c50f/tensorflow-2.5.1-cp37-cp37m-manylinux2010_x86_64.whl (from calamari_ocr->-r ./text_alignment/requirements.txt (line 2)):
2025-03-05T00:39:55Z #12 26.43         Expected sha256 5cebb9a548adfd5fdc6081df72db93488afbe02a60c4f7c60b94930d6b3fbfa6
2025-03-05T00:39:55Z #12 26.43              Got        a0b27dae3ef99392814cb831e0115ac94947783f3513511201b092f072d79a28
2025-03-05T00:39:55Z #12 26.43 
2025-03-05T00:39:55Z #12 26.44 
2025-03-05T00:39:55Z #12 26.44 [notice] A new release of pip is available: 23.0.1 -> 24.0
2025-03-05T00:39:55Z #12 26.44 [notice] To update, run: pip install --upgrade pip
2025-03-05T00:39:55Z ------
2025-03-05T00:39:55Z ERROR: failed to solve: executor failed running [/bin/sh -c /opt/install_gpu_rodan_jobs]: exit code: 1
2025-03-05T00:39:55Z An error occurs when building images.
2025-03-05T00:39:55Z Push to another repo to trigger webhook on failure with tag syntax: {repo name}-{build status}-{source branch}-{docker tag}
2025-03-05T00:39:55Z The push refers to repository [docker.io/ddmal/docker-webhook]
2025-03-05T00:39:55Z 57ea01797809: Preparing
2025-03-05T00:39:55Z 01c9a2a5f237: Preparing
2025-03-05T00:39:55Z 01c9a2a5f237: Layer already exists
2025-03-05T00:39:56Z 57ea01797809: Pushed
2025-03-05T00:39:57Z rodan-fail-develop-nightly: digest: sha256:c02bb0042657ae0069353765ab4b212afda9c627dd29e876bfce41bdf7ffecc9 size: 734
2025-03-05T00:39:57Z build hook failed! (1)

[homework36]

We should fix this ASAP
the work has to be done on an intel machine

[homework36]

@notkaramel Take a look at this and this can be (and should be) fixed with an Intel machine. It is not a hard task at all, but you will get to play around with Docker.
Things to do:

1. Fix the related gpu-celery Dockerfile (https://github.com/DDMAL/Rodan/blob/develop/gpu-celery/Dockerfile) and/or the install GPU jobs script (https://github.com/DDMAL/Rodan/blob/develop/scripts/install_gpu_rodan_jobs) (which is executed at line 169 RUN /opt/install_gpu_rodan_jobs). To fix this, you can

   • manually specify SHA256 hash for tensorflow 2.5.1 cp37, OR
   • remove hash checking by doing things like pip install tensorflow==2.5.1 --no-deps.

**However, there's one thing that I'm not sure. This issue should have been fixed in #990, but we are still getting it.

2. Build and test new docker images according to wiki (https://github.com/DDMAL/Rodan/wiki/Testing-New-Docker-Images). Since you don't manage our Docker Hub, you can let me know Source and Docker Tag or you can go to internal wiki to get credentials.
3. Run Rodan with these new images locally with some easy testing jobs (like classifying).

[notkaramel]

I couldn't rebuild the gpu-celery for various reasons so I decided to rebuild the whole image from scratch. Here's what I have done so far:

---

Attempt to migrate gpu-celery to a newer version of the image.
Reason:

• Ubuntu 18.04 reached its EOL back in 2023
• Ubuntu 20.04 is reaching its EOL in 2025
• Lots of questionable? commands and redundant with unclear documentation
• Old build is 11GB

Attempt 1: nvidia/cuda:12.9.0-cudnn-runtime-ubuntu24.04

(latest runtime cudnn as of writing)

• The image is 2.97 GB
• Python 3 is python 3.12 (needs to downgrade to 3.7 maybe?)
• cudnn? It's CUDA Deep Neural Network Library, built on top of CUDA. It's useful for deep learning operations using TensorFlow and PyTorch, over general purpose of CUDA.
• Total size after built: 13.2GB, oops

Play around:

# Install the image
docker pull nvidia/cuda:12.9.0-cudnn-runtime-ubuntu24.04

# Create a container & get into its shell
docker run -it nvidia/cuda:12.9.0-cudnn-runtime-ubuntu24.04 /bin/bash

Attempt 2: tensorflow/tensorflow:latest-gpu

• "On versions including and after 19.03, you will use the nvidia-container-toolkit package and the --gpus all flag."
• Docs: https://www.tensorflow.org/install/docker
• Base image: 3.63 GB
• Play around:

docker pull tensorflow/tensorflow:latest-gpu
docker run -it tensorflow/tensorflow:latest-gpu bash

• This image does not need installation of python and tensorflow, or another build tools from the original Dockerfile

[homework36]

I think maybe we can solve #1181 too.

[homework36]

Make sure you also take a look at #751 (I will come back to this issue later this week)

[notkaramel]

#751 actually has similar approach that I did with #1294, just 3 years of tensorflow docker image versions apart :D Thanks for letting me know

[notkaramel]

Update on attempts:

Attempt 2: tensorflow/tensorflow:latest-gpu

• Issue with installing dependencies for text-alignment job:
  • First it was scipy has a version not downloadable (can't find on pypi), fix that
  • Then scikit-image has the same issue

Attempt 3: tensorflow/tensorflow:2.5.1-gpu & various other versions

• "On versions including and after 19.03, you will use the nvidia-container-toolkit package and the --gpus all flag."
• Base image: 2.6 GB
• Play around:

docker pull tensorflow/tensorflow:2.5.1-gpu
docker run -it tensorflow/tensorflow:2.5.1-gpu bash

cd Rodan
docker build --file gpu-celery/Dockerfile . -t flamingo:tensorflow

docker run -it flamingo:tensorflow bash
docker image history flamingo:tensorflow

Issues: can't update packages, therefore cannot download git

2.732 Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [64.0 kB]
2.986 Reading package lists...
4.122 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
4.122 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is no longer signed.

Issues with /opt/install_gpu_rodan_jobs

1. /code/jobs/text_alignment scipy==1.7.3 could not find a verion that satisfies

ERROR: Ignored the following yanked versions: 1.11.0, 1.14.0rc1
ERROR: Could not find a version that satisfies the requirement scipy==1.7.3 (from versions: 0.8.0, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.16.0, 0.16.1, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.9.0rc1, 1.9.0rc2, 1.9.0rc3, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0rc1, 1.10.0rc2, 1.10.0, 1.10.1, 1.11.0rc1, 1.11.0rc2, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.12.0rc1, 1.12.0rc2, 1.12.0, 1.13.0rc1, 1.13.0, 1.13.1, 1.14.0rc2, 1.14.0, 1.14.1, 1.15.0rc1, 1.15.0rc2, 1.15.0, 1.15.1, 1.15.2, 1.15.3)

Tensorflow already have 1.11.0

Attempt 4: nvidia/cuda:12.9.0-runtime-ubuntu24.04

• Virtual Environment: https://pythonspeed.com/articles/activate-virtualenv-dockerfile/

 If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

• Same package version issue

Attemp 5: smaller branch

• on install_gpu_rodan_jobs: Keras version issue. But it uninstall a compatible version to download an uncompatible one????

15.66   Attempting uninstall: Keras
15.66     Found existing installation: keras 2.10.0rc0
15.84     Uninstalling keras-2.10.0rc0:
17.54       Successfully uninstalled keras-2.10.0rc0
17.88 Successfully installed Keras-2.3.1 h5py-3.1.0 html5lib-0.9999999 keras-applications-1.0.8 numpy-1.19.2 pyyaml-6.0.2 scipy-1.10.1
17.88 ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
17.88
17.88 We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
17.88
17.88 tensorflow 2.10.0rc1 requires keras<2.11,>=2.10.0rc0, but you'll have keras 2.3.1 which is incompatible.
17.88 tensorflow 2.10.0rc1 requires numpy>=1.20, but you'll have numpy 1.19.2 which is incompatible.
17.88 scipy 1.10.1 requires numpy<1.27.0,>=1.19.5, but you'll have numpy 1.19.2 which is incompatible.

[notkaramel]

I like to make funny graphics to visualise my issues. Here's one for gpu-celery image rebuilding.

I've attempted with:

• tensorflow/tensorflow from tag 2.5.1 to latest-gpu (2.15).

  • git install issue: Older images cannot do apt-get update to get git, although I firmly believe it has to do with NVIDIA archiving CUDA and apt cannot fetch that. apt-get update fails but I could install git just fine while testing the container locally
    • As of writing, I just found a solution: apt-get install || true from https://unix.stackexchange.com/questions/325705/why-is-pattern-command-true-useful and ChatGPT
  • pip install dependencies issue: mainly in text_alignment/requirements.txt. The logs is questionable at times
    • Version mismatch: tensorflow base image from 2.6.0 pre-install a compatible versions of keras, numpy, and scipy (and other ML-related libraries). When pip install, it uninstall the compatible versions, install the tensorflow-incompatible versions, and pray hopelessly that it would work :)

• nvidia/cuda~ with various tags.

  • Built container is giantic (13GB compared to 11GB previously) even before installing rodan jobs
  • Newer versions of this needs a python downgrade because most of the source code could run with 3.7
  • Installing tensorflow takes a lot of time with this approach, although it might work better than the tensorflow base image approach.

I've thought of/tried:

• Isolating images: having images like python3.7-slim as builder base to install all dependecies on virtual environment, then move all of them to the main container
• Downgrade python to version 3.7 on later images of tensorflow: Haven't tried, but since git was having weird issue, I don't know if I could download python3.7 from ubuntu archive repository.
• Download git separately: build/install git from https://git-scm.com

---

Link in case the funny graphic needs to be updated: https://excalidraw.com/#json=FV1xV5zGAMwaolnYApzEP,fVDW87aZ9SHbBYLt1TzbmQ

[homework36]

Can we maybe try to build an image for just one GPU job first?

[homework36]

This is an excellent image - it basically summarizes what we experience on Docker Hub, too. I love it (and not the bugs). These issues seem to take turns (also with chance) to occur on our Docker build CI...