Support maps for CoL Extended taxonomy

[fmendezh]

Background

The Spark module currently generates HBase tables for point and tile maps using taxonomic identifiers from the GBIF Backbone taxonomy. To support the Catalogue of Life (CoL) Extended release, we need to extend the existing infrastructure to process and store maps based on CoL taxonomic identifiers.

Current Architecture

The current implementation:

• Reads GBIF occurrence data from Avro files
• Extracts GBIF Backbone taxonomic keys: kingdomKey, phylumKey, classKey, orderKey, familyKey, genusKey, speciesKey, taxonKey
• Generates point maps (for low-occurrence views) and tile pyramids (for high-occurrence views)
• Stores data in HBase tables with multiple EPSG projections (4326, 3857, 3575, 3031)

Proposed Solution

Extend the existing Spark jobs to support CoL identifiers and create parallel processing workflows for CoL Extended release maps.

Suggested Tasks

1. Spark Job Modifications

Files requiring changes:

• MapBuilder.java: Update readAvroSource() method to select CoL identifier fields

  • Add CoL taxonomic key columns (e.g., colKingdomKey, colPhylumKey, etc.) to the .select() statement
  • Parameterize taxonomy source to support both GBIF and CoL

• MapKeysUDF.java: Extend the UDF to handle CoL taxonomic hierarchies

  • Add support for CoL identifier parameters
  • Ensure map key generation works with CoL taxonomy structure
  • Consider adding a taxonomy source parameter to distinguish between GBIF and CoL keys

• PointMapBuilder.java and TileMapBuilder.java: Update SQL queries to use CoL fields

  • Modify the mapKeys() UDF calls to include CoL identifiers
  • Ensure proper grouping and aggregation with CoL taxonomy

2. Configuration Management

• Create separate configuration files for CoL Extended (e.g., col-extended-dev.yml, col-extended-prod.yml)
  • Define CoL-specific HBase table names (e.g., col_maps_points, col_maps_tiles)
  • Configure separate target directories for CoL map outputs
  • Set appropriate threshold values for tile pyramid generation
  • Define separate Hive database/table names for CoL processing

3. Airflow Workflows

• Create new Airflow DAG for CoL point maps generation
• Create new Airflow DAG for CoL tile maps generation
• Add monitoring and alerting for CoL maps

4. HBase Table Management

• Create separate HBase tables for CoL Extended maps
  • col_maps_points_<timestamp> for point maps
  • col_maps_tiles_<timestamp> for tile pyramids
• Update table creation scripts in PrepareBackfill.java to support taxonomy parameter
• Ensure cleanup and snapshot management works for CoL tables

5. Testing & Validation

• Unit tests for CoL-specific UDF functionality
• Integration tests with sample CoL occurrence data
• Validate map generation for various CoL taxonomic ranks

6. Documentation

• Update README.md with CoL Extended support details
• Document configuration parameters for CoL workflows
• Add examples for running CoL map generation
• Update deployment and operational documentation

[mdoering]

What is the use of the taxonomic keys for the fixed ranks? With COL and other taxonomies like WORMS we do not have a fixed set of ranks. We can expose those classic Linnean ranks, but depending on what they are used for we might maybe miss important data?

[mdoering]

And would it not make sense to do this generically with a customizable prefix for any number of additional taxonomies that might come? Don't we need maps for every indexed taxonomy?

[MattBlissett]

And would it not make sense to do this generically with a customizable prefix for any number of additional taxonomies that might come? Don't we need maps for every indexed taxonomy?

I think the intention is to pre-generate the point/tile maps only for major users, with a lower-use taxonomy (e.g. regional checklist) using ElasticSearch instead.

Or, we might have Clickhouse running by then.

[mdoering]

I was thinking about WoRMS coming up next. And maybe some DNA or phylogenetic taxonomy soon for Novo. Who knows. If it isn't too much extra work I think it makes sense to implement it in a configurable way.

[timrobertson100]

I anticipate we’ll quickly hit issues with the batch job duration for many taxonomies. It’ll be ok (and pretty easy) to adapt the code for COL, but if many taxonomies (3 or more?) come I suspect something like a Clickhouse solution will be needed. There was a fully working Clickhouse branch (major ranks only) developed and running in gbif-dev for a while.

To scale support for many taxonomies, storing the LFT of the nested set for each taxonomy would likely be a good option.

[timrobertson100]

This is a draft example of changes that might build the tile pyramid for all taxa in all classifications (testing underway).

This reports:

Map views that require tile pyramid 10286