Run gdalraster in a GDAL-provided Docker image

[ctoney]

Edit:
TLDR see https://usdaforestservice.github.io/gdalraster/index.html#docker
the rest may be useful reference for setting up from scratch..

This will install and run gdalraster against latest GDAL in a GDAL-provided Docker image. Used for running the gdalraster test suite against GDAL beta versions / release candidates, and also for gdalraster development and testing under latest GDAL dev.

This is run in Ubuntu and assumes basic familiarity with Docker. It also assumes use of the latest gdalraster development version. Here we install it from R-universe.

(Note: At the time of writing, minor modifications would be needed to make this work with current gdalraster released version on CRAN (1.12.0). Mainly apt install libproj-dev would be needed to provide external PROJ. Latest gdalraster dev does not depend on external PROJ lib, using instead only the GDAL internal PROJ, since 1) gdalraster only uses PROJ via GDAL headers, 2) gdalraster requires GDAL >= 3.1, and 3) GDAL has had PROJ as a build requirement since 3.0.)

Pull GDAL latest Docker image and install R:

sudo docker pull ghcr.io/osgeo/gdal:ubuntu-full-latest

# list docker images on the computer
sudo docker image ls

# create and run a container
sudo docker run -it -d ghcr.io/osgeo/gdal:ubuntu-full-latest bash

# optional
# if we want to mount a directory on the host computer to be available in the container,
# eg., as /mnt/tempdata:
# sudo docker run -v /home/ctoney/data/dockertmp:/mnt/tempdata -it -d ghcr.io/osgeo/gdal:ubuntu-full-latest bash

# see containers:
sudo docker ps -a

# enter the container and work in an interactive bash shell
sudo docker exec -it CONTAINER_ID bash

# install R
# *** check latest instructions from: https://cloud.r-project.org/bin/linux/ubuntu/ ***
# latest CRAN instructions for R 4.5.0 are copied here:
###############################################################################
# update indices
apt update -qq
# install two helper packages we need
apt install --no-install-recommends software-properties-common dirmngr
# add the signing key (by Michael Rutter) for these repos
# To verify key, run gpg --show-keys /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc 
# Fingerprint: E298A3A825C0D65DFD57CBB651716619E084DAB9
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
# add the repo from CRAN -- lsb_release adjusts to 'noble' or 'jammy' or ... as needed
add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
# install R itself
apt install --no-install-recommends r-base
###############################################################################

# add dev
apt install --no-install-recommends r-base-dev

# system requirement for package xml2 which is a gdalraster dependency
apt install libxml2-dev

# libcurl is needed for some gdalraster suggested packages or their dependencies
apt install libcurl4-openssl-dev

# then run R
R

# install packages required by gdalraster
install.packages("Rcpp")
install.packages("RcppInt64")
install.packages("nanoarrow")
install.packages(c("bit64", "wk", "xml2", "yyjsonr"))

# install gdalraster suggested packages
# This takes a while since these packages also need several of their own dependencies.
# The full set is optional. To run the test suite, we really just need testthat. In that case,
# tests that need gt would fail but that could be safely ignored in some testing scenarios
# since it is only for display.
install.packages(c("gt", "knitr", "rmarkdown", "scales", "testthat"))

# install gdalraster

# optionally install from a local tarball that we've placed in /home/ctoney/data/dockertmp,
# mounted in the container as /mnt/tempdata (see optional shell command above):

# cd /mnt/tempdata
# install.packages("gdalraster_1.12.0.9002.tar.gz", repos = NULL)
# or with tests:
# install.packages("gdalraster_1.12.0.9002.tar.gz", repos = NULL, INSTALL_opts = "--install-tests")

# or
# install current dev version from r-universe with tests
install.packages("gdalraster",
    repos = c("https://usdaforestservice.r-universe.dev", "https://cran.r-project.org"),
    INSTALL_opts = "--install-tests")

library(gdalraster)

# run the test suite
testthat::test_package("gdalraster")

# optional for doing full R CMD check
# copy the downloaded tarball to the home directory in the container
# Here we use gdalraster vsi_*() functions, but base R file management functions could
# be used instead. The tarball downloaded from r-universe with install.packages() is by
# default in a sub-directory 'downloaded_packages' of the session temporary directory,
# which will be deleted at the end of the session. Note that alternatively, we could set
# the destdir argument in the call to install.packages() above.

# verify contents of the 'downloaded_packages' sub-directory
file.path(tempdir(), "downloaded_packages") |> vsi_read_dir()

# copy the tarball to /home (or any suitable location)
vsi_copy_file(
    file.path(tempdir(), "downloaded_packages", "gdalraster_1.12.0.9300.tar.gz"),
    file.path("/home", "gdalraster_1.12.0.9300.tar.gz"))

# exists?
file.path("/home", "gdalraster_1.12.0.9300.tar.gz") |> vsi_stat()
# is file?
file.path("/home", "gdalraster_1.12.0.9300.tar.gz") |> vsi_stat("type")

q()

Then in the container bash shell:

# make sure we're in /home and the tarball is there
cd /home
ls

# optional for pdflatex to build the manual during R CMD check
# avoids that error but requires a lot of files
apt install texlive-latex-base texlive-fonts-recommended
apt install texlive-fonts-extra texlive-latex-extra

# locales
# optional to avoid check warning about setting locale
apt install locales
# select `97` for en_US.UTF-8:
dpkg-reconfigure locales

R CMD check gdalraster_1.12.0.9300.tar.gz

# exit the container
exit

Note that we can come back to this container in its current state until it has been removed.
To stop a running container, or re-start a stopped container:

sudo docker stop CONTAINER_ID

sudo docker start CONTAINER_ID
# re-enter
# sudo docker exec -it CONTAINER_ID bash

# or, remove the container with
sudo docker container rm CONTAINER

[ctoney]

@rsbivand, @mdsumner,

This documents the process I've been using to test gdalraster against GDAL beta for the recent minor version releases (3.9, 3.10, 3.11). This is obviously far from the full CRAN reverse dependency testing that Roger has been doing. However, the gdalraster test suite by itself appears to have decent sensitivity. This would be expected since the package provides API-level bindings, so touches the API across a wide range of the library in ways that other R spatial packages may not. The gdalraster test suite detected a regression in each of the 3.9 and 3.10 beta releases, but all tests currently pass with 3.11 beta 1. That is a clearly limited, but usefully informative data point based on previous experience. We've also added substantially to the vector bindings and associated tests since the GDAL 3.10 release.

I started trying to set up sf and terra in the same Docker container but have not succeeded yet. I am less familiar with some of their dependencies, and have had limited time to troubleshoot. I likely will not pursue that further in the near term since completing the gdalraster 2.0 release is currently a high priority, and my capacity is currently limited for other reasons.

I think a community effort involving the maintainers of R spatial packages dependent on GDAL would be the ideal long-term approach to sustain and develop Roger's testing/CI work. I'm not currently in a position to coordinate such an effort. Mike has indicated interest, and hopefully there is broader interest in various levels of participation. I would like to contribute and I'm open to suggestions on what might be helpful from my end.

Toward that goal, the documentation above is also a small start. A repository of some type where we can collect shared code, documentation and discussion would be helpful to coordinate efforts. I don't see the current gdalraster repo as the right place for that. Hopefully others can offer suggestions or alternative viewpoints.

[mdsumner]

thanks! I folded your approach into my gdal-builds as "r-gdal-full", I had to borrow a bit from the first rocker step to have it run unattended, and there's lots to check and improve. All this does is install deps and gdalraster, then checks out the source from github and downloads the latest version from CRAN. Start with

##docker run --rm -ti ghcr.io/mdsumner/gdal-builds:r-gdal-full
ls dev-pkgs
# gdalraster
git  -C "dev-pkgs/gdalraster" log  | head -n 5
#commit 24f864b6f50f6e3d7ff813abb5d223b7ac3d9720
#Author: Chris Toney <jctoney@gmail.com>
#Date:   Tue May 6 16:12:48 2025 -0600
#
#    increment dev version

ls cran
# gdalraster_1.12.0.tar.gz

script : https://github.com/mdsumner/gdal-builds/blob/main/scripts/install-r-gdal-full.sh

dockerfile: https://github.com/mdsumner/gdal-builds/blob/main/dockerfiles/r-gdal-full.Dockerfile

action: https://github.com/mdsumner/gdal-builds/blob/main/.github/workflows/docker-r-gdal-full.yml

I've been using the other "frankendocker" builds in this repo heavily, but they are long overdue a review and cleanup. I think we can fairly easily set up a way to get the revdep checking environment nicely templated with something like this.

[ctoney]

Thanks, this is great. I'll switch to this and try it soon to set up 3.11 released version.

[mdsumner]

cool, the other thing I'm working on is

1. getting all revdeps for a set of packages and installing them (I'm doing this on HPC atm for gdalraster, sf, terra, vapour there are ~ 1160 required packages), this simplifies the check process
2. update that library regularly
3. start docker, link to the library via .libPaths() and use it to run check for a package

For me the key was the separation between the prerequisites on the docker image (GDAL version, R with build and check deps) and the pretty massive installed library of packages. Can't use Posit package manager because it requires apt-installs that match the binaries.

I experimented with parallel installs but that's not so important, because there's an upfront overhead to getting the 1160 packages installed, and then a small daily job to update those. Here's my notes fwiw:

mdsumner/pawsey-aad#9 (using a local mirror rather than cran itself and miniCRAN for now)

More soon.

[ctoney]

Love this. I just pulled ghcr.io/mdsumner/gdal-builds:r-gdal-full.

root@3bdc8369c038:/# gdalinfo --version
GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378, released 2025/05/06

Then in R (it still has gdalraster 1.12.0.9301 which is fine for now, that was the 2.0 prelease version which did not have any more code changes):

packageVersion("gdalraster")
#> [1] '1.12.0.9301'

library(gdalraster)
#> code for methods in class "Rcpp_CmbTable" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_CmbTable" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALRaster" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALRaster" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALVector" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALVector" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_RunningStats" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_RunningStats" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_VSIFile" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_VSIFile" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378 (released 2025-05-06), GEOS 3.12.1, PROJ 9.7.0

After install.packages("codetools"), then restart the R session:

library(gdalraster)
#> GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378 (released 2025-05-06), GEOS 3.12.1, PROJ 9.7.0

Prior to that, I also pulled the tagged release version, ghcr.io/osgeo/gdal:ubuntu-full-3.11.0, and set it up manually with my notes above. When loading gdalraster in that case I did not get the messages "code for methods in class... (recommended package 'codetools' not available?)". I'm not sure where in my process codetools gets installed. I don't install it explicitly. It may be in the dependency chain but your build script installs all the same Imports and Suggests. Maybe a difference in the methods used for the R install itself? I'll try to narrow it down, but we may want to add codetools explicitly since CRAN labels it as "Priority: recommended".

Anyway, your automated setup is great. Thanks again! I'll be using it more and testing some things with that image.

[mdsumner]

I think I've fixed the codetools package, removed no-install-recommends:

apt-get update && apt-get -y install --no-install-recommends r-base-dev=${R_VERSION}*

my setup on HPC is looking good, I have a lustre disk for software installs, and so now I check out the image daily and run install_cran on the set of required packages, it's about 6Gb. After a bit more smoothing out I will start running jobs to loop R CMD check tasks over the revdeps

I originally had a problem because the "software partition" on storage has a 100K file limit, and "scratch" has a much higher allowed number but everything > 30days gets deleted - so I toyed with sync back and forth to object storage but blergh - at some point the software partition might max out if we add more packages (it's 60K now), and I think it's a hard limit - so:

it might keeping the R packages minimal on the build, and rely on a volume that has a constantly updated library of packages, - but it feels messy to constantly have to link that ... we could set this up on generic AWS, with lambda to farm out each check run - but I'll talk to my local cloud expert about that soon

[mdsumner]

I'm toying with r-gdal-full stopping at install R+Recommends, and then we can do this:

export PKG=gdalraster
export REPO=usdaforestservice

NCPUS=$(nproc --all)
export MAKEFLAGS=-j${NCPUS}

echo "options(Ncpus  = ${NCPUS})"  >> .Rprofile

## get the artefacts
mkdir dev-pkgs
cd dev-pkgs
git clone https://github.com/${REPO}/${PKG}.git


Rscript -e "install.packages(\"remotes\")"
Rscript -e "install.packages(remotes::dev_package_deps(\"${PKG}\", dependencies = TRUE)[[\"package\"]], Ncpus = getOption(\"Ncpus\", 1))"
cd ..

Rscript -e "install.packages(\"${PKG}\", repos = c(\"https://${REPO}.r-universe.dev\", \"https://cran.r-project.org\"), INSTALL_opts = \"--install-tests\", Ncpus = getOption(\"Ncpus\", 1))"

mkdir cran
Rscript -e 'download.packages("gdalraster", destdir = "cran")'

but, I think we might as well get the common set of dev deps for a set of packages, and fold that into r-gdal-full, as a common basis for "GDAL packages on CRAN", and then keep separate the full shebang of rev deps as an WIP to try out separately.

[mdsumner]

it's a lot of packages though, because it detects all the deps, 331 currently

NCPUS=$(nproc --all)
export MAKEFLAGS=-j${NCPUS}

echo "options(Ncpus  = ${NCPUS})"  >> .Rprofile

apt-get update

## I'm adding these as I find reqs
apt-get install -y libssl-dev libfontconfig1-dev libudunits2-dev  cmake

Rscript -e "install.packages(\"remotes\")"

mkdir dev-pgs
cd dev-pkgs

## here we should pass in these package names, code clones the repo from "BugReports", and collates dev deps then install them all
Rscript -e 'pdeps <- character(); 
pkgs <- c("gdalraster", "sf", "terra", "vapour"); 
av <- tools::CRAN_package_db() |> subset(Package %in% pkgs); 
for (i in seq_len(nrow(av))) {
  system(sprintf("git clone %s", gsub("/issues", "", av$BugReports[i]))); 
  pdeps <- c(pdeps, remotes::dev_package_deps(av[["Package"]][i], dependencies = TRUE)[["package"]])
}; 
remotes::install_cran(unique(pdeps), dependencies = TRUE, Ncpus = getOption("Ncpus", 1L))'

cd ..

there might be other missing libs, just treating this as experiments for now

[ctoney]

Maybe install packages with pak and let it take care of deps and sysreqs?
https://pak.r-lib.org/reference/sysreqs.html

Either way, its query capabilities could get a full list of sysreqs and generate the install commands, e.g.,

> pak::pkg_sysreqs(c("gdalraster", "sf", "terra", "vapour"), dependencies = TRUE)
── Install scripts ───────────────────────────────────────────── Ubuntu 24.04 ──
apt-get -y update
apt-get -y install libcurl4-openssl-dev libssl-dev make libgdal-dev gdal-bin \
  libgeos-dev pandoc libnode-dev libxml2-dev zlib1g-dev libproj-dev \
  libsqlite3-dev unixodbc-dev libpng-dev libpq-dev libicu-dev \
  libfontconfig1-dev libfreetype6-dev libfribidi-dev libharfbuzz-dev \
  libudunits2-dev libnetcdf-dev

── Packages and their system dependencies ──────────────────────────────────────
curl        – libcurl4-openssl-dev, libssl-dev
fs          – make
gdalraster  – gdal-bin, libgdal-dev, libgeos-dev
httpuv      – make, zlib1g-dev
knitr       – pandoc
lwgeom      – libgeos-dev, libproj-dev, libsqlite3-dev
mapview     – make
ncdf4       – libnetcdf-dev
odbc        – make, unixodbc-dev
openssl     – libssl-dev
png         – libpng-dev
rapidjsonr  – make
rmarkdown   – pandoc
RPostgres   – libpq-dev
s2          – libssl-dev
sass        – make
sf          – gdal-bin, libgdal-dev, libgeos-dev, libproj-dev, libsqlite3-dev
stringi     – libicu-dev
svglite     – libpng-dev
systemfonts – libfontconfig1-dev, libfreetype6-dev
terra       – gdal-bin, libgdal-dev, libgeos-dev, libnetcdf-dev, libproj-dev, libsqlite3-dev
textshaping – libfreetype6-dev, libfribidi-dev, libharfbuzz-dev
units       – libudunits2-dev
V8          – libnode-dev
vapour      – gdal-bin, libgdal-dev, libproj-dev
XML         – libxml2-dev
xml2        – libxml2-dev

The return value is a list which gives the apt-get commands in its $pre_install and $install_scripts elements.

[mdsumner]

Oh yeah good point. I've been reluctant to use pak because it will blat source installs but I'd lost sight of it providing sysreqs, thanks!!

[mdsumner]

I'm going to cut r-gdal-full at the R install. Just let that be the baseline and start with specializing that for various sets of packages.

[mdsumner]

yes, so r-gdal-full is just GDAL + R, and now ghcr.io/mdsumner/gdal-builds:r-gdalraster is r-gdal-full+gdalraster

[mdsumner]

which, feel free to modify as you like - do you want write permissions here?

there's still a lot of testing for me to do, but separating having that GDAL+R baseline is nice clean starting point I think - gdalraster requires libcurl4-openssl-dev but not libssl (iiuc), and I want to be able to separate them for now.

[ctoney]

gdalraster requires libcurl4-openssl-dev but not libssl (iiuc)

Yes I think that is true, only for Suggests though. One of gdalraster Suggests has package curl in its dependencies, and curl has libcurl4-openssl-dev as SysReq. I'll try to confirm where that comes from out of curiosity.

I have family visiting through Monday so I'll be off-and-on for the next few days. Thanks very much for ghcr.io/mdsumner/gdal-builds:r-gdalraster. Will get back to this soon.

[ctoney]

Just pulled a fresh build of ghcr.io/mdsumner/gdal-builds:r-gdalraster. I got

> packageVersion("gdalraster")
[1] '2.0.0.9009'
> library(gdalraster)
GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378 (released 2025-05-06), GEOS 3.12.1, PROJ 9.7.0

The gdalraster build is from <24 hours ago. It then only takes testthat::test_package("gdalraster") to run the latest test suite under the latest GDAL Docker build. Also quick and easy to update gdalraster from R-universe since all sysreq and dependencies are already installed. This is very convenient and awesome.