MindsDB SQL Parser 🚧

Installation

  pip install mindsdb_sql_parser

How to use

from mindsdb_sql_parser import parse_sql

query = parse_sql('select b from aaa where c=1')

# result is abstract syntax tree (AST) 
query

# string representation of AST
query.to_tree()

# representation of tree as sql string. it can not exactly match with original sql
query.to_string()

Architecture

For parsing is used SLY library.

Parsing consists of 2 stages, (separate module for every dialect):

Defining keywords in lexer.py module. It is made mostly with regexp
Defining syntax rules in parser.py module. It is made by describing rules in BNF grammar
- Syntax is defined in decorator of function. Inside of decorator you can use keyword itself or other function from parser
- Output of function can be used as input in other functions of parser
- Outputs of the parser is listed in "Top-level statements". It has to be Abstract syntax tree (AST) object.

SLY does not support inheritance, therefore every dialect is described completely, without extension one from another.

AST

Structure of AST is defined in separate modules (in parser/ast/).
It can be inherited
Every class have to have these methods:
- to_tree - to return hierarchical representation of object
- get_string - to return object as sql expression (or sub-expression)
- copy - to copy AST-tree to new object

Error handling

For better user experience parsing error contains useful information about problem location and possible solution to solve it.

it shows location of error if

character isn't parsed (by lexer)
token is unexpected (by parser)

it tries to propose correct token instead (or before) error location. Possible options

Keyword will be showed as is.
'[number]' - if float and integer is expected
'[string]' - if string is expected
'[identifier]' - if name of the objects is expected. For example, they are bold words here:
- "select x as name from tbl1 where col=1"

How suggestion works: It uses next possible tokens defined by syntax rules. If this is the end of the query: just shows these tokens. Else:

it tries to replace bad token with other token from list of possible tokens
tries to parse query once again, if there is no error:
- add this token to suggestion list
second iteration: put possible token before bad token (instead of replacement) and repeat the same operation.

How to test

pip install -r requirements_test.txt
env PYTHONPATH=./ pytest

Uh oh!

License

Uh oh!

pankaj-bind/mindsdb_sql_parser

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages