query-interpreter/README.md

query-interpreter

Core program to interpret query language strings into structured data, and back again.

SQL Tokens

We are currently using DataDog's SQL Tokenizer sqllexer to scan through SQL strings. Here are the general token types it defines:

type TokenType int

const (
 ERROR TokenType = iota
 EOF
 SPACE                  // space or newline
 STRING                 // string literal
 INCOMPLETE_STRING      // incomplete string literal so that we can obfuscate it, e.g. 'abc
 NUMBER                 // number literal
 IDENT                  // identifier
 QUOTED_IDENT           // quoted identifier
 OPERATOR               // operator
 WILDCARD               // wildcard *
 COMMENT                // comment
 MULTILINE_COMMENT      // multiline comment
 PUNCTUATION            // punctuation
 DOLLAR_QUOTED_FUNCTION // dollar quoted function
 DOLLAR_QUOTED_STRING   // dollar quoted string
 POSITIONAL_PARAMETER   // numbered parameter
 BIND_PARAMETER         // bind parameter
 FUNCTION               // function
 SYSTEM_VARIABLE        // system variable
 UNKNOWN                // unknown token
 COMMAND                // SQL commands like SELECT, INSERT
 KEYWORD                // Other SQL keywords
 JSON_OP                // JSON operators
 BOOLEAN                // boolean literal
 NULL                   // null literal
 PROC_INDICATOR         // procedure indicator
 CTE_INDICATOR          // CTE indicator
 ALIAS_INDICATOR        // alias indicator
)

Based on these different token types we will be able to parse out the details of said query into structs to represent them. From those structs we will be able to modify their details any way that we choose and reconstruct them into valid SQL statements.