41 std::optional<TToken>
grabId();
ScanExcept(const StringType &why)
Construct a scan exception with a reason string.
StringType why() const
Get the error description.
std::optional< TToken > grabSingle()
Try to grab a single-quoted character literal token.
std::optional< TToken > grabDigits()
Try to grab a numeric literal token.
uint64_t parseLineNumber()
Parse a line directive line number.
size_t size() const
Return the total number of scanned tokens.
std::vector< TToken > & getTokens()
Get the underlying token vector.
TToken & operator[](size_t index)
Access a token by index.
std::optional< TToken > grabId()
Try to grab an identifier or keyword token.
std::string parseFileName()
Parse a line directive filename.
TString string_buffer
source text buffer
std::optional< TToken > grabString()
Try to grab a double-quoted string literal token.
std::vector< TToken > tokens
scanned token list
Scanner(const TString &b)
Construct a scanner from a string buffer.
uint64_t scan()
Tokenize the entire source buffer.
void removeEOL()
Remove end-of-line tokens from the token list.
bool is_c_sym(const StringType &str)
Check if a string is a recognized C-style symbol.
std::optional< TToken > grabSymbols()
Try to grab one or more symbol characters as a token.
TMap token_map
character classification map
Character-by-character string buffer with line/column tracking.
String string_type
string type alias
Maps characters to their lexical category (letter, digit, symbol, etc.).
A scanned token with type, value, and source position.
buffer::StringBuffer< char >::string_type StringType
Underlying string type.
token::TokenMap< char > TMap
Default character classification map.
token::Token< char > TToken
Default token type.
buffer::StringBuffer< char > TString
Default character string buffer.
TokenType
Classification of scanned tokens.
CharType
Classification of individual characters during lexing.
std::ostream & operator<<(std::ostream &out, const types::CharType &c)
Stream output operator for CharType.
Character-by-character string buffer with line/column tracking and Token/TokenMap types.