SPPCompiler.LexicalAnalysis.Lexer module

class SPPCompiler.LexicalAnalysis.Lexer.SppLexer(code)

Bases: object

The Lexer translates a string of code into a list of tokens. Only single-length tokens are created in the lexer, and the parser checks for multi-length tokens.

Single line comments and string characters are supported by tracking starting and terminating boundary characters. Unknown tokens aren’t errored, but are stored as an “unknown” token, so the parser can through a syntax error using an error formatter.

lex()

Lex the code with single-character comparisons. Track strings and comments to skip and add characters as necessary.

Return type:

list[RawToken]

Returns:

A list of tokens.