Run gdalraster in a GDAL-provided Docker image #703
Replies: 14 comments 1 reply
-
|
This documents the process I've been using to test gdalraster against GDAL beta for the recent minor version releases (3.9, 3.10, 3.11). This is obviously far from the full CRAN reverse dependency testing that Roger has been doing. However, the gdalraster test suite by itself appears to have decent sensitivity. This would be expected since the package provides API-level bindings, so touches the API across a wide range of the library in ways that other R spatial packages may not. The gdalraster test suite detected a regression in each of the 3.9 and 3.10 beta releases, but all tests currently pass with 3.11 beta 1. That is a clearly limited, but usefully informative data point based on previous experience. We've also added substantially to the vector bindings and associated tests since the GDAL 3.10 release. I started trying to set up sf and terra in the same Docker container but have not succeeded yet. I am less familiar with some of their dependencies, and have had limited time to troubleshoot. I likely will not pursue that further in the near term since completing the gdalraster 2.0 release is currently a high priority, and my capacity is currently limited for other reasons. I think a community effort involving the maintainers of R spatial packages dependent on GDAL would be the ideal long-term approach to sustain and develop Roger's testing/CI work. I'm not currently in a position to coordinate such an effort. Mike has indicated interest, and hopefully there is broader interest in various levels of participation. I would like to contribute and I'm open to suggestions on what might be helpful from my end. Toward that goal, the documentation above is also a small start. A repository of some type where we can collect shared code, documentation and discussion would be helpful to coordinate efforts. I don't see the current gdalraster repo as the right place for that. Hopefully others can offer suggestions or alternative viewpoints. |
Beta Was this translation helpful? Give feedback.
-
|
thanks! I folded your approach into my gdal-builds as "r-gdal-full", I had to borrow a bit from the first rocker step to have it run unattended, and there's lots to check and improve. All this does is install deps and gdalraster, then checks out the source from github and downloads the latest version from CRAN. Start with script : https://github.com/mdsumner/gdal-builds/blob/main/scripts/install-r-gdal-full.sh dockerfile: https://github.com/mdsumner/gdal-builds/blob/main/dockerfiles/r-gdal-full.Dockerfile action: https://github.com/mdsumner/gdal-builds/blob/main/.github/workflows/docker-r-gdal-full.yml I've been using the other "frankendocker" builds in this repo heavily, but they are long overdue a review and cleanup. I think we can fairly easily set up a way to get the revdep checking environment nicely templated with something like this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, this is great. I'll switch to this and try it soon to set up 3.11 released version. |
Beta Was this translation helpful? Give feedback.
-
|
cool, the other thing I'm working on is
For me the key was the separation between the prerequisites on the docker image (GDAL version, R with build and check deps) and the pretty massive installed library of packages. Can't use Posit package manager because it requires apt-installs that match the binaries. I experimented with parallel installs but that's not so important, because there's an upfront overhead to getting the 1160 packages installed, and then a small daily job to update those. Here's my notes fwiw: mdsumner/pawsey-aad#9 (using a local mirror rather than cran itself and miniCRAN for now) More soon. |
Beta Was this translation helpful? Give feedback.
-
|
Love this. I just pulled Then in R (it still has gdalraster 1.12.0.9301 which is fine for now, that was the 2.0 prelease version which did not have any more code changes): packageVersion("gdalraster")
#> [1] '1.12.0.9301'
library(gdalraster)
#> code for methods in class "Rcpp_CmbTable" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_CmbTable" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALRaster" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALRaster" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALVector" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_GDALVector" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_RunningStats" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_RunningStats" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_VSIFile" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> code for methods in class "Rcpp_VSIFile" was not checked for suspicious field assignments (recommended package 'codetools' not available?)
#> GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378 (released 2025-05-06), GEOS 3.12.1, PROJ 9.7.0After library(gdalraster)
#> GDAL 3.12.0dev-ce5f0f192b46eb67243a037a241e0ad010a2e378 (released 2025-05-06), GEOS 3.12.1, PROJ 9.7.0Prior to that, I also pulled the tagged release version, Anyway, your automated setup is great. Thanks again! I'll be using it more and testing some things with that image. |
Beta Was this translation helpful? Give feedback.
-
|
I think I've fixed the codetools package, removed no-install-recommends: my setup on HPC is looking good, I have a lustre disk for software installs, and so now I check out the image daily and run install_cran on the set of required packages, it's about 6Gb. After a bit more smoothing out I will start running jobs to loop R CMD check tasks over the revdeps I originally had a problem because the "software partition" on storage has a 100K file limit, and "scratch" has a much higher allowed number but everything > 30days gets deleted - so I toyed with sync back and forth to object storage but blergh - at some point the software partition might max out if we add more packages (it's 60K now), and I think it's a hard limit - so: it might keeping the R packages minimal on the build, and rely on a volume that has a constantly updated library of packages, - but it feels messy to constantly have to link that ... we could set this up on generic AWS, with lambda to farm out each check run - but I'll talk to my local cloud expert about that soon |
Beta Was this translation helpful? Give feedback.
-
|
I'm toying with but, I think we might as well get the common set of dev deps for a set of packages, and fold that into |
Beta Was this translation helpful? Give feedback.
-
|
it's a lot of packages though, because it detects all the deps, 331 currently there might be other missing libs, just treating this as experiments for now |
Beta Was this translation helpful? Give feedback.
-
|
Maybe install packages with pak and let it take care of deps and sysreqs? Either way, its query capabilities could get a full list of sysreqs and generate the install commands, e.g., > pak::pkg_sysreqs(c("gdalraster", "sf", "terra", "vapour"), dependencies = TRUE)
── Install scripts ───────────────────────────────────────────── Ubuntu 24.04 ──
apt-get -y update
apt-get -y install libcurl4-openssl-dev libssl-dev make libgdal-dev gdal-bin \
libgeos-dev pandoc libnode-dev libxml2-dev zlib1g-dev libproj-dev \
libsqlite3-dev unixodbc-dev libpng-dev libpq-dev libicu-dev \
libfontconfig1-dev libfreetype6-dev libfribidi-dev libharfbuzz-dev \
libudunits2-dev libnetcdf-dev
── Packages and their system dependencies ──────────────────────────────────────
curl – libcurl4-openssl-dev, libssl-dev
fs – make
gdalraster – gdal-bin, libgdal-dev, libgeos-dev
httpuv – make, zlib1g-dev
knitr – pandoc
lwgeom – libgeos-dev, libproj-dev, libsqlite3-dev
mapview – make
ncdf4 – libnetcdf-dev
odbc – make, unixodbc-dev
openssl – libssl-dev
png – libpng-dev
rapidjsonr – make
rmarkdown – pandoc
RPostgres – libpq-dev
s2 – libssl-dev
sass – make
sf – gdal-bin, libgdal-dev, libgeos-dev, libproj-dev, libsqlite3-dev
stringi – libicu-dev
svglite – libpng-dev
systemfonts – libfontconfig1-dev, libfreetype6-dev
terra – gdal-bin, libgdal-dev, libgeos-dev, libnetcdf-dev, libproj-dev, libsqlite3-dev
textshaping – libfreetype6-dev, libfribidi-dev, libharfbuzz-dev
units – libudunits2-dev
V8 – libnode-dev
vapour – gdal-bin, libgdal-dev, libproj-dev
XML – libxml2-dev
xml2 – libxml2-devThe return value is a list which gives the |
Beta Was this translation helpful? Give feedback.
-
|
Oh yeah good point. I've been reluctant to use pak because it will blat source installs but I'd lost sight of it providing sysreqs, thanks!! |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to cut r-gdal-full at the R install. Just let that be the baseline and start with specializing that for various sets of packages. |
Beta Was this translation helpful? Give feedback.
-
|
yes, so |
Beta Was this translation helpful? Give feedback.
-
Yes I think that is true, only for Suggests though. One of gdalraster Suggests has package curl in its dependencies, and curl has libcurl4-openssl-dev as SysReq. I'll try to confirm where that comes from out of curiosity. I have family visiting through Monday so I'll be off-and-on for the next few days. Thanks very much for |
Beta Was this translation helpful? Give feedback.
-
|
Just pulled a fresh build of The gdalraster build is from <24 hours ago. It then only takes |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Edit:
TLDR see https://usdaforestservice.github.io/gdalraster/index.html#docker
the rest may be useful reference for setting up from scratch..
This will install and run gdalraster against latest GDAL in a GDAL-provided Docker image. Used for running the gdalraster test suite against GDAL beta versions / release candidates, and also for gdalraster development and testing under latest GDAL dev.
This is run in Ubuntu and assumes basic familiarity with Docker. It also assumes use of the latest gdalraster development version. Here we install it from R-universe.
(Note: At the time of writing, minor modifications would be needed to make this work with current gdalraster released version on CRAN (1.12.0). Mainly
apt install libproj-devwould be needed to provide external PROJ. Latest gdalraster dev does not depend on external PROJ lib, using instead only the GDAL internal PROJ, since 1) gdalraster only uses PROJ via GDAL headers, 2) gdalraster requires GDAL >= 3.1, and 3) GDAL has had PROJ as a build requirement since 3.0.)Pull GDAL latest Docker image and install R:
Then in the container bash shell:
Note that we can come back to this container in its current state until it has been removed.
To stop a running container, or re-start a stopped container:
Beta Was this translation helpful? Give feedback.
All reactions