MariaDB Enterprise Cluster / ColumnStore Engine / MaxScale Proxy

Summary

MariaDB ColumnStore is a columnar storage engine that utilizes a massively parallel distributed data architecture. It was built by porting InfiniDB to MariaDB and has been released under the GPL license.

MariaDB ColumnStore is designed for big data scaling to process petabytes of data, linear scalability and exceptional performance with real-time response to analytical queries. It leverages the I/O benefits of columnar storage, compression, just-in-time projection, and horizontal and vertical partitioning to deliver tremendous performance when analyzing large data sets.

This is a Terraform and Ansible project to provision a high availability MariaDB ColumnStore deployment on Amazon Web Services. This automation project will create the following system:

• 3 MariaDB Nodes For Durability & Performance
• 1 S3 Bucket For Object Storage (Data)
• 1 Multi-Attach io2 SSD GFS2 Volume (Shared Metadata)
• 2 MaxScale Nodes For High Availability

Prerequisites:

• Amazon Web Services (AWS) Account
• Install Terraform ^†
• Install Ansible ^‡
• MariaDB Enterprise Token

^† Requires Terraform v0.14.4 or above. ^‡ Requires Full Ansible 2.10.5 or above. (Not Ansible-Core)

Instructions:

Open a terminal window and clone the repository:

1. git clone https://github.com/mariadb-corporation/columnstore-ansible.git
2. cd into the newly cloned folder
3. ./kickstart.sh This script will help you fill in the required Terraform variables (AWS credentials, distro, etc). It will create a terraform.tfvars file with the variables you provided.
4. You can manually edit the terraform.tfvars file to change any of the variables that were not set by kickstart.sh. See variables.tf for variable descriptions.
5. ./deploy.sh This script will call Terraform to create the infrastructure and then Ansible to provision the nodes. Please note that if Terraform already has the infrastructure created, deploy.sh will DESTROY the existing infrastructure and create a new one.

When the playbook completes, it will also create connect.sh in the current directory. This script will allow you to connect to the created nodes by name (e.g. ./connect.sh mcs1).

Further information can be found on our official deployment guide.

Possible Problems

• If you get an error like Failed to update apt cache: E:Failed to fetch https://dlm.mariadb.com/repo/<token>/<rest_of_url> 403 Forbidden when provisioning, you'll have to follow that link and accept the terms and conditions, then re-run deploy.sh.

Current Approved AWS Image(s)

AMI OS	AMI ID	Region	Zone
CentOS 7	ami-0bc06212a56393ee1	us-west-2	us-west-2c

MCS Command-line Instructions

From your primary node:

Set API Code:

mcs cluster set api-key --key <api_key>

Get Status:

mcs cluster status

Start Cluster:

mcs cluster start

Stop Cluster:

mcs cluster stop

Add Node:

mcs cluster node add --node <node>

Remove Node:

mcs cluster node remove --node <node>

Mode Set Read Only:

mcs cluster set mode --mode readonly

Mode Set Read/Write:

mcs cluster set mode --mode readwrite

Other CLI Tools

• core Change directory to /var/log/mariadb/columnstore/corefiles
• dbrm Change directory to /var/lib/columnstore/data1/systemFiles/dbrm
• extentSave Backup extent map
• tcrit Tail crit.log
• tdebug Tail debug.log
• terror Tail error.log
• tinfo Tail info.log
• twarning Tail warning.log

REST-API Instructions

Format of url endpoints for REST API:

https://{server}:{port}/cmapi/{version}/{route}/{command}

Examples urls for available endpoints:

• https://127.0.0.1:8640/cmapi/0.4.0/cluster/status
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/start
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/shutdown
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/node
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/mode-set

Request Headers Needed:

• 'x-api-key': 'somekey123'
• 'Content-Type': 'application/json'

_{Note: x-api-key can be set to any value of your choice during the first call to the server. Subsequent connections will require this same key}

Examples using curl:

Get Status:

$ curl -s https://127.0.0.1:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header 'x-api-key:somekey123' -k | jq .

Start Cluster:

$ curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/start --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20}' -k | jq .

Stop Cluster:

$ curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/shutdown --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20}' -k | jq .

Add Node:

$ curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/node --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "node": "<replace_with_desired_hostname>"}' -k | jq .

Remove Node:

$ curl -s -X DELETE https://127.0.0.1:8640/cmapi/0.4.0/cluster/node --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "node": "<replace_with_desired_hostname>"}' -k | jq .

Mode Set:

$ curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/mode-set --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "mode": "readwrite"}' -k | jq .