SPPCompiler.LexicalAnalysis.TokenType module

class SPPCompiler.LexicalAnalysis.TokenType.RawTokenType(value: Any)

Bases: TokenType

The RawTokenType class is a set of tokens that the lexer can recognize. These tokens are single-length, and will be combined in the parser to make more advanced tokens.

TkCharacter = <RawTokenType.TkCharacter: 0>

TkDigit = <RawTokenType.TkDigit: 1>

TkEqualsSign = <RawTokenType.TkEqualsSign: 2>

TkPlusSign = <RawTokenType.TkPlusSign: 3>

TkMinusSign = <RawTokenType.TkMinusSign: 4>

TkAsterisk = <RawTokenType.TkAsterisk: 5>

TkForwardSlash = <RawTokenType.TkForwardSlash: 6>

TkPercentSign = <RawTokenType.TkPercentSign: 7>

TkCaret = <RawTokenType.TkCaret: 8>

TkLessThanSign = <RawTokenType.TkLessThanSign: 9>

TkGreaterThanSign = <RawTokenType.TkGreaterThanSign: 10>

TkLeftParenthesis = <RawTokenType.TkLeftParenthesis: 11>

TkRightParenthesis = <RawTokenType.TkRightParenthesis: 12>

TkLeftSquareBracket = <RawTokenType.TkLeftSquareBracket: 13>

TkRightSquareBracket = <RawTokenType.TkRightSquareBracket: 14>

TkLeftCurlyBrace = <RawTokenType.TkLeftCurlyBrace: 15>

TkRightCurlyBrace = <RawTokenType.TkRightCurlyBrace: 16>

TkQuestionMark = <RawTokenType.TkQuestionMark: 17>

TkColon = <RawTokenType.TkColon: 18>

TkAmpersand = <RawTokenType.TkAmpersand: 19>

TkVerticalBar = <RawTokenType.TkVerticalBar: 20>

TkDot = <RawTokenType.TkDot: 21>

TkComma = <RawTokenType.TkComma: 22>

TkAt = <RawTokenType.TkAt: 23>

TkUnderscore = <RawTokenType.TkUnderscore: 24>

TkSpeechMark = <RawTokenType.TkSpeechMark: 25>

TkExclamationMark = <RawTokenType.TkExclamationMark: 26>

TkSemicolon = <RawTokenType.TkSemicolon: 27>

TkWhitespace = <RawTokenType.TkWhitespace: 28>

TkNewLine = <RawTokenType.TkNewLine: 29>

TkDollar = <RawTokenType.TkDollar: 30>

TkUnknown = <RawTokenType.TkUnknown: 31>

NoToken = <RawTokenType.NoToken: 32>

EndOfFile = <RawTokenType.EndOfFile: 33>

static newline_token()

Gets the token being used by this token set to indicate a “newline”. :rtype: TokenType :return: The newline token.

static whitespace_token()

Gets the token being used by this token set to indicate a “whitespace”. :rtype: TokenType :return: The whitespace token.

class SPPCompiler.LexicalAnalysis.TokenType.RawKeywordType(value: Any)

Bases: TokenType

The RawKeywordType class is a set of keywords that are used for the parser to know, for a specific keyword, which characters to parse. The final TokenAst will not contain a RawKeywordType, but the matching SppTokenType.

Cls = <RawKeywordType.Cls: 'cls'>

Fun = <RawKeywordType.Fun: 'fun'>

Cor = <RawKeywordType.Cor: 'cor'>

Sup = <RawKeywordType.Sup: 'sup'>

Ext = <RawKeywordType.Ext: 'ext'>

Mut = <RawKeywordType.Mut: 'mut'>

Use = <RawKeywordType.Use: 'use'>

Cmp = <RawKeywordType.Cmp: 'cmp'>

Let = <RawKeywordType.Let: 'let'>

Type = <RawKeywordType.Type: 'type'>

Where = <RawKeywordType.Where: 'where'>

SelfVal = <RawKeywordType.SelfVal: 'self'>

SelfType = <RawKeywordType.SelfType: 'Self'>

Case = <RawKeywordType.Case: 'case'>

Iter = <RawKeywordType.Iter: 'iter'>

Of = <RawKeywordType.Of: 'of'>

Loop = <RawKeywordType.Loop: 'loop'>

In = <RawKeywordType.In: 'in'>

Else = <RawKeywordType.Else: 'else'>

Gen = <RawKeywordType.Gen: 'gen'>

With = <RawKeywordType.With: 'with'>

Ret = <RawKeywordType.Ret: 'ret'>

Exit = <RawKeywordType.Exit: 'exit'>

Skip = <RawKeywordType.Skip: 'skip'>

Is = <RawKeywordType.Is: 'is'>

As = <RawKeywordType.As: 'as'>

Or = <RawKeywordType.Or: 'or'>

And = <RawKeywordType.And: 'and'>

Not = <RawKeywordType.Not: 'not'>

Async = <RawKeywordType.Async: 'async'>

True_ = <RawKeywordType.True_: 'true'>

False_ = <RawKeywordType.False_: 'false'>

Res = <RawKeywordType.Res: 'res'>

Caps = <RawKeywordType.Caps: 'caps'>

class SPPCompiler.LexicalAnalysis.TokenType.RawToken(token_type, token_data)

Bases: object

A RawToken allows for pairing metadata will a RawTokenType. For example, when “a” is lexed, the RawToken will contain a RawTokenType.TkCharacter with the data “a”.

token_type: RawTokenType | RawKeywordType

token_data: str

class SPPCompiler.LexicalAnalysis.TokenType.SppTokenType(value: Any)

Bases: TokenType

The SppTokenType class contains all the keyword and tokens that are created by the parser, and stored inside TokenAsts.

LxNumber = <SppTokenType.LxNumber: '<char>'>

LxString = <SppTokenType.LxString: '<num>'>

KwCls = <SppTokenType.KwCls: 'cls'>

KwFun = <SppTokenType.KwFun: 'fun'>

KwCor = <SppTokenType.KwCor: 'cor'>

KwSup = <SppTokenType.KwSup: 'sup'>

KwExt = <SppTokenType.KwExt: 'ext'>

KwMut = <SppTokenType.KwMut: 'mut'>

KwUse = <SppTokenType.KwUse: 'use'>

KwCmp = <SppTokenType.KwCmp: 'cmp'>

KwLet = <SppTokenType.KwLet: 'let'>

KwType = <SppTokenType.KwType: 'type'>

KwCase = <SppTokenType.KwCase: 'case'>

KwElse = <SppTokenType.KwElse: 'else'>

KwLoop = <SppTokenType.KwLoop: 'loop'>

KwIter = <SppTokenType.KwIter: 'iter'>

KwWith = <SppTokenType.KwWith: 'with'>

KwSkip = <SppTokenType.KwSkip: 'skip'>

KwExit = <SppTokenType.KwExit: 'exit'>

KwRet = <SppTokenType.KwRet: 'ret'>

KwGen = <SppTokenType.KwGen: 'gen'>

KwWhere = <SppTokenType.KwWhere: 'where'>

KwAs = <SppTokenType.KwAs: 'as'>

KwIs = <SppTokenType.KwIs: 'is'>

KwTrue = <SppTokenType.KwTrue: 'true'>

KwFalse = <SppTokenType.KwFalse: 'false'>

KwSelfVal = <SppTokenType.KwSelfVal: 'self'>

KwSelfType = <SppTokenType.KwSelfType: 'Self'>

KwAnd = <SppTokenType.KwAnd: 'and'>

KwOr = <SppTokenType.KwOr: 'or'>

KwNot = <SppTokenType.KwNot: 'not'>

KwOf = <SppTokenType.KwOf: 'of'>

KwIn = <SppTokenType.KwIn: 'in'>

KwAsync = <SppTokenType.KwAsync: 'async'>

KwRes = <SppTokenType.KwRes: 'res'>

KwCaps = <SppTokenType.KwCaps: 'caps'>

TkEq = <SppTokenType.TkEq: '=='>

The token for equality. This is a double equals sign, and is used to compare two values.

TkNe = <SppTokenType.TkNe: '!='>

The token for inequality. This is a double exclamation mark, and is used to compare two values.

TkLe = <SppTokenType.TkLe: '<='>

The token for less than or equal to. This is a double less than sign, and is used to compare two values.

TkGe = <SppTokenType.TkGe: '>='>

The token for greater than or equal to. This is a double greater than sign, and is used to compare two values.

TkLt = <SppTokenType.TkLt: '<'>

The token for less than. This is a single less than sign, and is used to compare two values.

TkGt = <SppTokenType.TkGt: '>'>

The token for greater than. This is a single greater than sign, and is used to compare two values.

TkPlus = <SppTokenType.TkPlus: '+'>

The token for addition. This is a single plus sign, and is used to add two values together.

TkMinus = <SppTokenType.TkMinus: '-'>

The token for subtraction. This is a single minus sign, and is used to subtract two values.

TkMultiply = <SppTokenType.TkMultiply: '*'>

The token for multiplication. This is a single asterisk, and is used to multiply two values together.

TkDivide = <SppTokenType.TkDivide: '/'>

The token for division. This is a single forward slash, and is used to divide two values.

TkRemainder = <SppTokenType.TkRemainder: '%'>

The token for remainder. This is a single percent sign, and is used to get the remainder of two values.

TkExponent = <SppTokenType.TkExponent: '**'>

The token for exponentiation. This is a double asterisk, and is used to raise a value to the power of another value.

TkPlusAssign = <SppTokenType.TkPlusAssign: '+='>

The token for addition assignment. This is a plus sign followed by an equals sign, and is used to add a value to a variable.

TkMinusAssign = <SppTokenType.TkMinusAssign: '-='>

The token for subtraction assignment. This is a minus sign followed by an equals sign, and is used to subtract a value from a variable.

TkMultiplyAssign = <SppTokenType.TkMultiplyAssign: '*='>

The token for multiplication assignment. This is an asterisk followed by an equals sign, and is used to multiply a variable by a value.

TkDivideAssign = <SppTokenType.TkDivideAssign: '/='>

The token for division assignment. This is a forward slash followed by an equals sign, and is used to divide a variable by a value.

TkRemainderAssign = <SppTokenType.TkRemainderAssign: '%='>

The token for remainder assignment. This is a percent sign followed by an equals sign, and is used to get the remainder of a variable and a value.

TkExponentAssign = <SppTokenType.TkExponentAssign: '**='>

The token for exponentiation assignment. This is a double asterisk followed by an equals sign, and is used to raise a variable to the power of a value.

TkLeftParenthesis = <SppTokenType.TkLeftParenthesis: '('>

The token for a left parenthesis. This is a single left parenthesis, and is used to group expressions together.

TkRightParenthesis = <SppTokenType.TkRightParenthesis: ')'>

The token for a right parenthesis. This is a single right parenthesis, and is used to group expressions together.

TkLeftSquareBracket = <SppTokenType.TkLeftSquareBracket: '['>

The token for a left square bracket. This is a single left square bracket, and is used to group expressions together.

TkRightSquareBracket = <SppTokenType.TkRightSquareBracket: ']'>

The token for a right square bracket. This is a single right square bracket, and is used to group expressions together.

TkLeftCurlyBrace = <SppTokenType.TkLeftCurlyBrace: '{'>

The token for a left curly brace. This is a single left curly brace, and is used to define a scope boundary.

TkRightCurlyBrace = <SppTokenType.TkRightCurlyBrace: '}'>

The token for a right curly brace. This is a single right curly brace, and is used to define a scope boundary.

TkQuestionMark = <SppTokenType.TkQuestionMark: '?'>

The token for a question mark. This is a single question mark, and is used for optional types or early returning.

TkDoubleDot = <SppTokenType.TkDoubleDot: '..'>

The token for a double dot. This is a double dot, and is used for a range tuple operations or defaulting.

TkColon = <SppTokenType.TkColon: ':'>

The token for a colon. This is a single colon, and is used for type annotations or constraints.

TkAmpersand = <SppTokenType.TkAmpersand: '&'>

The token for an ampersand. This is a single ampersand, and is used to mark borrows.

TkVerticalBar = <SppTokenType.TkVerticalBar: '|'>

The token for a vertical bar. This is a single vertical bar, and is used to mark a lambda expression.

TkLeftShift = <SppTokenType.TkLeftShift: '<<'>

The token for a left shift. This is a double less than sign, and is used to shift bits to the left.

TkRightShift = <SppTokenType.TkRightShift: '>>'>

The token for a right shift. This is a double greater than sign, and is used to shift bits to the right.

TkLeftShiftAssign = <SppTokenType.TkLeftShiftAssign: '<<='>

The token for a left shift assignment. This is a double less than sign followed by an equals sign, and is used to shift bits to the left and assign the result to a variable.

TkRightShiftAssign = <SppTokenType.TkRightShiftAssign: '>>='>

The token for a right shift assignment. This is a double greater than sign followed by an equals sign, and is used to shift bits to the right and assign the result to a variable.

TkBitAnd = <SppTokenType.TkBitAnd: '&'>

The token for a bitwise AND. This is a single ampersand, and is used to perform a bitwise AND operation.

TkBitIor = <SppTokenType.TkBitIor: '|'>

The token for a bitwise OR. This is a single vertical bar, and is used to perform a bitwise OR operation.

TkBitXor = <SppTokenType.TkBitXor: '^'>

The token for a bitwise XOR. This is a single caret, and is used to perform a bitwise XOR operation.

TkBitAndAssign = <SppTokenType.TkBitAndAssign: '&='>

The token for a bitwise AND assignment. This is a single ampersand followed by an equals sign, and is used to perform a bitwise AND operation and assign the result to a variable.

TkBitIorAssign = <SppTokenType.TkBitIorAssign: '|='>

The token for a bitwise OR assignment. This is a single vertical bar followed by an equals sign, and is used to perform a bitwise OR operation and assign the result to a variable.

TkBitXorAssign = <SppTokenType.TkBitXorAssign: '^='>

The token for a bitwise XOR assignment. This is a single caret followed by an equals sign, and is used to perform a bitwise XOR operation and assign the result to a variable.

TkDot = <SppTokenType.TkDot: '.'>

The token for a dot. This is a single dot, and is used to access members of a class or module.

TkDoubleColon = <SppTokenType.TkDoubleColon: '::'>

The token for a double colon. This is a double colon, and is used for static access to members of a class or module.

TkComma = <SppTokenType.TkComma: ','>

The token for a comma. This is a single comma, and is used to separate arguments in a function call or definition.

TkAssign = <SppTokenType.TkAssign: '='>

The token for an assignment. This is a single equals sign, and is used to assign a value to a variable.

TkArrowR = <SppTokenType.TkArrowR: '->'>

The token for a right arrow. This is a dash followed by a greater than sign, and is used to indicate a function return type.

TkAt = <SppTokenType.TkAt: '@'>

The token for an at sign. This is a single at sign, and is used to mark annotations.

TkSpeechMark = <SppTokenType.TkSpeechMark: '"'>

The token for a speech mark. This is a single double-speech mark, and is used to define a string literal.

TkUnderscore = <SppTokenType.TkUnderscore: '_'>

The token for an underscore. This is a single underscore, and is used to indicate a wildcard or ignore a single value.

TkExclamationMark = <SppTokenType.TkExclamationMark: '!'>

The token for an exclamation mark. This is a single exclamation mark, and is used to bind a generator exception.

TkDoubleExclamationMark = <SppTokenType.TkDoubleExclamationMark: '!!'>

The token for a double exclamation mark. This is used to indicate an exhausted generator.

TkSemicolon = <SppTokenType.TkSemicolon: ';'>

The token for a semicolon. Used in repeated array literals.

TkWhitespace = <SppTokenType.TkWhitespace: ' '>

The token for whitespace. This is a single space, and is used to separate tokens.

TkNewLine = <SppTokenType.TkNewLine: '\n'>

The token for a new line. This is a single new line, and is used to separate lines of code.

TkDollar = <SppTokenType.TkDollar: '$'>

The token for a dollar sign. This is a single dollar sign, and is used to generate internal types and identifiers.

NoToken = <SppTokenType.NoToken: ''>

The token for no token. This is an empty string, and is used to indicate that there is no token (skip).

static newline_token()

Gets the token being used by this token set to indicate a “newline”. :rtype: SppTokenType :return: The newline token.

static whitespace_token()

Gets the token being used by this token set to indicate a “whitespace”. :rtype: SppTokenType :return: The whitespace token.