MicrobeDB

How to access

MicrobeDB is distributed using the CERN VM File System (CVMFS). Docker and CSI deployment recipes are available in ./destinations. The recipes are executed by Terraform.

Docker may fail to unmount CVMFS during shutdown, run sudo fusermount -u ./microbedb/mount if you encounter transport endpoint is not connected errors.

OSX Peculiarities

OSX does not natively support Docker, it runs Docker within a Linux virtual machine. This workaround means that support is limited to only the most basic use case. While mounting MicrobeDB via CVMFS, it will fail with an error.

To work around this CVMFS must be installed and configured manually. First ensure that FUSE is enabled by running kextstat | grep -i fuse. Download the CVMFS package. Install the pkg and reboot. Copy ../destinations/docker/cvmfs.config to /etc/cvmfs/default.local. Copy ./microbedb.brinkmanlab.ca.pub to /etc/cvmfs/keys/microbedb.brinkmanlab.ca.pub. Ensure everything is configured properly by running sudo cvmfs_config chksetup. You MUST mount the CVMFS repository under a shared folder as configured in your Docker settings for it to be accessible by Docker. By default /tmp should be included as a shared folder and you can mount the repository to /tmp/microbedb. Ensure /tmp/microbedb exists and run sudo mount -t cvmfs microbedb.brinkmanlab.ca /tmp/microbedb.

Schema documentation

Run sqlite3 microbedb.sqlite '.schema' to view documentation of the various tables and columns. The assembly table is largely undocumented because NCBI does not document their data schemas.

Working with taxonomy data

Use SQLite recursive query to determine if tax_id is subclass of ancestor. The following returns 1 if the query_tax_id is a subclass of ancestor_tax_id:

WITH RECURSIVE subClassOf(n) AS (
    VALUES (query_tax_id)
    UNION
    SELECT parent_tax_id
    FROM taxonomy_nodes,
         subClassOf
    WHERE taxonomy_nodes.tax_id = subClassOf.n
      AND taxonomy_nodes.tax_id != ancestor_tax_id
)
SELECT 1
FROM subClassOf
WHERE n = ancestor_tax_id
LIMIT 1;

Build requirements

• bash with filefuncs extension
• yq which also installs the xq executable
• jq compiled with ONIGURUMA regex libary
• Entrez CLI
• SQLite3
• GNU awk
• parallel
• gzip
• biopython.convert
• rsync

Ensure the find command supports -empty by running find --help | grep '-empty'. The most recent CVMFS commit of the repository must be mounted on all compute nodes. cvmfs_config must be accessible on all compute nodes.

Project Layout

• destinations/* - terraform modules to deploy a CVMFS client configured with microbedb to various environments
• update.sh - Script to sync data with NCBI for a CVMFS server
• init_env.sh - Script to install dependencies for update.sh
• fetch.sh - Executed by update.sh per chunk of datasets returned by Entrez
• finalize.sh - Executed by update.sh once all invocations of fetch.sh have completed
• resume.sh - Script to allow resuming execution of fetch.sh invocations in the event that any fail. This script is copied to the job directory by update.sh and is intended to be executed from there.
• schema.sql - Database schema
• temp_tables.sql - Temporary table schema used by fetch.sh
• subclassOf.sh - Example utility to query database taxonomy data