Skip to content

Restructure docker build #321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pehrsoderman merged 2 commits into from
May 23, 2025
Merged

Restructure docker build #321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f5a96c
Large restructure of how our docker images are built.
gkreitz May 23, 2025
d86f384
Remove old Dockerfile in root, unused afaict
gkreitz May 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions Dockerfile
View file
Open in desktop

This file was deleted.

46 changes: 13 additions & 33 deletions admin/docker/Dockerfile.build
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,44 +1,24 @@
# Package for building the problemtools .deb package
# Ends up in the /usr/local/problemtools_build/deb/ directory
# Docker image with all packages needed to build a problemtools .deb
#
# Setting build argument PROBLEMTOOLS_VERSION causes a specific
# version of problemtools to be built (default is latest version of
# develop branch on GitHub)
# Not uploaded anywhere, only used locally during building

FROM ubuntu:22.04

LABEL maintainer="austrin@kattis.com"
ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/runreqs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

# Install packages needed for build
RUN apt update && \
apt install -y \
# Packages required to build and run problemtools
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
rm -f /etc/apt/apt.conf.d/docker-clean \
apt-get update && apt-get install -y \
automake \
build-essential \
debhelper \
dh-python \
dh-virtualenv \
dpkg-dev \
g++ \
git \
make \
libboost-regex-dev \
libgmp-dev \
libgmp10 \
libgmpxx4ldbl \
pandoc \
python3 \
python3-pytest \
python3-setuptools \
python3-yaml \
python3-setuptools

RUN mkdir -p /usr/local/problemtools_build

WORKDIR /usr/local/problemtools_build
RUN git clone --recursive https://github.com/kattis/problemtools

ARG PROBLEMTOOLS_VERSION=develop
RUN cd problemtools && git checkout ${PROBLEMTOOLS_VERSION} && make builddeb

RUN mkdir -p deb
RUN mv kattis-problemtools*.deb deb/
libboost-regex-dev
40 changes: 15 additions & 25 deletions admin/docker/Dockerfile.full
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,23 @@
# Full problemtools docker image, containing problemtools and all
# supported programming languages.
#
#
# Build requirements:
# - The problemtools .deb package must be available from the host file
# system under a file name matching
# artifacts/deb/kattis-problemtools*.deb
# (Version of that .deb file should match the build argument
# PROBLEMTOOLS_VERSION but this is not checked.)

ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/icpc:${PROBLEMTOOLS_VERSION}

LABEL maintainer="austrin@kattis.com"
FROM problemtools/runreqs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
apt-get install -y \
fp-compiler \
gfortran \
gnucobol \
gccgo \
ghc haskell-platform \
gnustep-devel gnustep gnustep-make gnustep-common gobjc \
libgmp3-dev \
libmozjs-78-dev \
lua5.4 \
mono-complete \
nodejs \
ocaml-nox \
pandoc \
php-cli \
pypy \
rustc \
sbcl \
scala \
swi-prolog \
;
RUN mkdir -p /usr/local/artifacts
WORKDIR /usr/local/artifacts
COPY artifacts/deb .
RUN dpkg -i kattis-problemtools*.deb

WORKDIR /
40 changes: 40 additions & 0 deletions admin/docker/Dockerfile.fulllangs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Docker image with all packages needed to run a problemtools .deb, plus
# language support for all supported languages
#
# Not uploaded anywhere, only used locally during building

ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/icpclangs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

# All languages, plus curl which we need to fetch pypy2
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
rm -f /etc/apt/apt.conf.d/docker-clean \
apt-get update && apt-get install -y \
curl \
fp-compiler \
gfortran \
gnucobol \
gccgo \
ghc \
gnustep-devel gnustep gnustep-make gnustep-common gobjc \
lua5.4 \
mono-complete \
nodejs \
ocaml-nox \
php-cli \
rustc \
sbcl \
scala \
swi-prolog

# pypy2 is no longer packaged for Ubuntu, so download tarball (and check a sha256)
RUN curl -LO https://downloads.python.org/pypy/pypy2.7-v7.3.16-linux64.tar.bz2 \
&& echo '04b2fceb712d6f811274825b8a471ee392d3d1b53afc83eb3f42439ce00d8e07 pypy2.7-v7.3.16-linux64.tar.bz2' | sha256sum --check \
&& tar -xf pypy2.7-v7.3.16-linux64.tar.bz2 \
&& mv pypy2.7-v7.3.16-linux64 /opt/pypy \
&& ln -s /opt/pypy/bin/pypy /usr/bin/pypy \
&& rm pypy2.7-v7.3.16-linux64.tar.bz2
23 changes: 23 additions & 0 deletions admin/docker/Dockerfile.githubci
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Docker image with all deb packages needed for our github actions
# - Building a problemtools deb
# - Running verifyproblem on all examples

ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/fulllangs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

# Packages required to build and run problemtools
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
rm -f /etc/apt/apt.conf.d/docker-clean \
apt-get update && apt-get install -y \
automake \
build-essential \
debhelper \
dh-virtualenv \
dpkg-dev \
git \
make \
libboost-regex-dev
50 changes: 12 additions & 38 deletions admin/docker/Dockerfile.icpc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,22 @@
# Basic problemtools docker image, containing problemtools and the
# "ICPC languages" (C, C++, Java, Kotlin, and Python 3)
#
# Build requirements:
# - The problemtools .deb package must be available from the host file
# system under a file name matching
# artifacts/deb/kattis-problemtools*.deb
# (Version of that .deb file should match the build argument
# PROBLEMTOOLS_VERSION but this is not checked.)

ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/minimal:${PROBLEMTOOLS_VERSION}

LABEL maintainer="austrin@kattis.com"
FROM problemtools/icpclangs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

# Install C++, Java, and PyPy 3 via their ppa repository (Kotlin is installed below to get a more up-to-date version)
RUN apt-get update && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:pypy/ppa && \
apt-get update && \
apt-get install -y \
gcc g++ \
openjdk-11-jdk openjdk-11-jre \
pypy3

RUN apt-get install -y curl

ARG KOTLIN_VERSION=1.8.10
RUN curl "https://github.com/JetBrains/kotlin/releases/download/v$KOTLIN_VERSION/kotlin-compiler-$KOTLIN_VERSION.zip" -L -o kotlin.zip
RUN unzip -q kotlin.zip
RUN rm kotlin.zip

RUN mv kotlinc/bin/* /usr/bin
RUN mv kotlinc/lib/* /usr/lib

RUN rm -r kotlinc

# Reconfigure problemtools:
# - Use PyPy for Python 2 (not available in this image but in the full one)
# - Use PyPy for Python 3
RUN mkdir -p /etc/kattis/problemtools
RUN echo " \n\
python2: \n\
name: 'Python 2 w/PyPy'\n\
run: '/usr/bin/pypy \"{mainfile}\"'\n\
\n\
python3: \n\
name: 'Python 3 w/PyPy'\n\
run: '/usr/bin/pypy3 \"{mainfile}\"'\n\
\n" > /etc/kattis/problemtools/languages.yaml
RUN mkdir -p /usr/local/artifacts
WORKDIR /usr/local/artifacts
COPY artifacts/deb .
RUN dpkg -i kattis-problemtools*.deb

WORKDIR /
21 changes: 21 additions & 0 deletions admin/docker/Dockerfile.icpclangs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Docker image with all packages needed to run a problemtools .deb, plus
# language support for the "ICPC languages" (C, C++, Java, Kotlin, and Python 3)
#
# Not uploaded anywhere, only used locally during building

ARG PROBLEMTOOLS_VERSION=develop
FROM problemtools/runreqs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
rm -f /etc/apt/apt.conf.d/docker-clean \
apt-get update && apt-get install -y \
gcc \
g++ \
kotlin \
openjdk-21-jdk \
openjdk-21-jre \
pypy3
20 changes: 2 additions & 18 deletions admin/docker/Dockerfile.minimal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,11 @@
# PROBLEMTOOLS_VERSION but this is not checked.)

ARG PROBLEMTOOLS_VERSION=develop
FROM ubuntu:22.04

LABEL maintainer="austrin@kattis.com"
FROM problemtools/runreqs:${PROBLEMTOOLS_VERSION}

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

RUN apt update && \
apt install -y \
ghostscript \
libgmpxx4ldbl \
pandoc \
python-pkg-resources \
python3-minimal \
python3-yaml \
python3-plastex \
texlive-fonts-recommended \
texlive-lang-cyrillic \
texlive-latex-extra \
texlive-plain-generic \
tidy

RUN mkdir -p /usr/local/artifacts
WORKDIR /usr/local/artifacts
COPY artifacts/deb .
Expand Down
28 changes: 28 additions & 0 deletions admin/docker/Dockerfile.runreqs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Docker image with all packages needed to run a problemtools .deb
#
# Not uploaded anywhere, only used locally during building

ARG PROBLEMTOOLS_VERSION=develop
FROM ubuntu:24.04

LABEL maintainer="contact@kattis.com"
ENV DEBIAN_FRONTEND=noninteractive

# Packages required to build and run problemtools
# For libgmp, we technically just need libgmpxx4ldbl here, but for readability
# (and we need libgmp-dev in other images), we take libgmp-dev here
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
rm -f /etc/apt/apt.conf.d/docker-clean \
apt-get update && apt-get install -y \
dvisvgm \
ghostscript \
libgmp-dev \
pandoc \
python3 \
python3-venv \
texlive-fonts-recommended \
texlive-lang-cyrillic \
texlive-latex-extra \
texlive-plain-generic \
tidy
25 changes: 25 additions & 0 deletions admin/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Our docker images. Note that images depend on each other, please use the script
admin/update_docker.sh to build images in the correct order.

We have 4 images which are only used locally in the build process, and are not
uploaded to a repository.
- `runreqs`: Base image containing just the things needed to run problemtools
- `build`: Base image containing just the things needed to build a deb and run problemtools
- `icpclangs`: Base image containing what is needed to run problemtools, plus the "ICPC languages"
- `fulllangs`: Base image containing what is needed to run problemtools, plus all supported languages

We have 3 images which are meant for end users:
- `minimal`: Image with problemtools installed, but no languages.
- `icpc`: Image with problemtools plus the "ICPC languages" installed.
- `full`: Image with problemtools and all languages

We have 1 image which is used in our CI (to speed up things - it takes a few
minutes to apt-get install all languages and runtime requirements):
- `githubci`: Image with all languages and everything needed to build a deb and run problemtools

Build dependencies:
```
runreqs -> icpclangs -> fullangs -> githubci
/ \ | |
build minimal icpc full
```
Loading