49 lines
1.7 KiB
Markdown
49 lines
1.7 KiB
Markdown
# query-interpreter
|
|
|
|
Core program to interpret query language strings into structured data, and back again.
|
|
|
|
## SQL Tokens
|
|
|
|
We are currently using DataDog's SQL Tokenizer `sqllexer` to scan through SQL strings.
|
|
Here are the general token types it defines:
|
|
|
|
```go
|
|
type TokenType int
|
|
|
|
const (
|
|
ERROR TokenType = iota
|
|
EOF
|
|
SPACE // space or newline
|
|
STRING // string literal
|
|
INCOMPLETE_STRING // incomplete string literal so that we can obfuscate it, e.g. 'abc
|
|
NUMBER // number literal
|
|
IDENT // identifier
|
|
QUOTED_IDENT // quoted identifier
|
|
OPERATOR // operator
|
|
WILDCARD // wildcard *
|
|
COMMENT // comment
|
|
MULTILINE_COMMENT // multiline comment
|
|
PUNCTUATION // punctuation
|
|
DOLLAR_QUOTED_FUNCTION // dollar quoted function
|
|
DOLLAR_QUOTED_STRING // dollar quoted string
|
|
POSITIONAL_PARAMETER // numbered parameter
|
|
BIND_PARAMETER // bind parameter
|
|
FUNCTION // function
|
|
SYSTEM_VARIABLE // system variable
|
|
UNKNOWN // unknown token
|
|
COMMAND // SQL commands like SELECT, INSERT
|
|
KEYWORD // Other SQL keywords
|
|
JSON_OP // JSON operators
|
|
BOOLEAN // boolean literal
|
|
NULL // null literal
|
|
PROC_INDICATOR // procedure indicator
|
|
CTE_INDICATOR // CTE indicator
|
|
ALIAS_INDICATOR // alias indicator
|
|
)
|
|
|
|
```
|
|
|
|
Based on these different token types we will be able to parse out the details of said query into
|
|
structs to represent them. From those structs we will be able to modify their details any way that
|
|
we choose and reconstruct them into valid SQL statements.
|