MXVM 1.8.1
Virtual Machine, Compiler, and Pascal Frontend
Loading...
Searching...
No Matches
scan::Scanner Class Reference

Lexical scanner that tokenizes source text. More...

#include <scanner.hpp>

Public Member Functions

 Scanner (const TString &b)
 Construct a scanner from a string buffer.
uint64_t scan ()
 Tokenize the entire source buffer.
std::optional< TToken > grabId ()
 Try to grab an identifier or keyword token.
std::optional< TToken > grabDigits ()
 Try to grab a numeric literal token.
std::optional< TToken > grabSymbols ()
 Try to grab one or more symbol characters as a token.
std::optional< TToken > grabString ()
 Try to grab a double-quoted string literal token.
std::optional< TToken > grabSingle ()
 Try to grab a single-quoted character literal token.
TToken & operator[] (size_t index)
 Access a token by index.
size_t size () const
 Return the total number of scanned tokens.
void removeEOL ()
 Remove end-of-line tokens from the token list.
std::vector< TToken > & getTokens ()
 Get the underlying token vector.

Private Member Functions

bool is_c_sym (const StringType &str)
 Check if a string is a recognized C-style symbol.
uint64_t parseLineNumber ()
 Parse a #line directive line number.
std::string parseFileName ()
 Parse a #line directive filename.

Private Attributes

TString string_buffer
 source text buffer
TMap token_map
 character classification map
std::vector< TToken > tokens
 scanned token list

Detailed Description

Lexical scanner that tokenizes source text.

Scans a StringBuffer character-by-character and produces a list of typed tokens (identifiers, numbers, strings, symbols, hex literals).

Definition at line 28 of file scanner.hpp.

Constructor & Destructor Documentation

◆ Scanner()

scan::Scanner::Scanner ( const TString & b)

Construct a scanner from a string buffer.

Parameters
bSource text buffer to scan

Definition at line 12 of file scanner.cpp.

References string_buffer.

Member Function Documentation

◆ getTokens()

std::vector< TToken > & scan::Scanner::getTokens ( )
inline

Get the underlying token vector.

Definition at line 62 of file scanner.hpp.

References tokens.

Referenced by mxx::TPValidator::parseUses().

◆ grabDigits()

std::optional< TToken > scan::Scanner::grabDigits ( )

Try to grab a numeric literal token.

Definition at line 211 of file scanner.cpp.

References pos(), string_buffer, types::TT_HEX, and types::TT_NUM.

Referenced by scan().

◆ grabId()

std::optional< TToken > scan::Scanner::grabId ( )

Try to grab an identifier or keyword token.

Definition at line 185 of file scanner.cpp.

References scan::token::Token< Ch, String >::set_filename(), scan::token::Token< Ch, String >::set_pos(), scan::token::Token< Ch, String >::setToken(), string_buffer, and types::TT_ID.

Referenced by scan().

◆ grabSingle()

std::optional< TToken > scan::Scanner::grabSingle ( )

Try to grab a single-quoted character literal token.

Definition at line 420 of file scanner.cpp.

References pos(), string_buffer, and types::TT_STR.

Referenced by scan().

◆ grabString()

std::optional< TToken > scan::Scanner::grabString ( )

Try to grab a double-quoted string literal token.

Definition at line 367 of file scanner.cpp.

References pos(), string_buffer, and types::TT_STR.

Referenced by scan().

◆ grabSymbols()

std::optional< TToken > scan::Scanner::grabSymbols ( )

Try to grab one or more symbol characters as a token.

Definition at line 292 of file scanner.cpp.

References is_c_sym(), pos(), string_buffer, token_map, types::TT_ARG, types::TT_SYM, and types::TT_SYMBOL.

Referenced by scan().

◆ is_c_sym()

bool scan::Scanner::is_c_sym ( const StringType & str)
private

Check if a string is a recognized C-style symbol.

Definition at line 283 of file scanner.cpp.

Referenced by grabSymbols().

◆ operator[]()

TToken & scan::Scanner::operator[] ( size_t index)

Access a token by index.

Parameters
indexZero-based token index
Returns
Reference to the token

Definition at line 480 of file scanner.cpp.

References size(), and tokens.

◆ parseFileName()

std::string scan::Scanner::parseFileName ( )
private

Parse a #line directive filename.

Definition at line 44 of file scanner.cpp.

References string_buffer.

◆ parseLineNumber()

uint64_t scan::Scanner::parseLineNumber ( )
private

Parse a #line directive line number.

Definition at line 22 of file scanner.cpp.

References string_buffer.

◆ removeEOL()

void scan::Scanner::removeEOL ( )

Remove end-of-line tokens from the token list.

Definition at line 14 of file scanner.cpp.

References tokens, and types::TT_SYM.

◆ scan()

uint64_t scan::Scanner::scan ( )

Tokenize the entire source buffer.

Returns
Number of tokens produced

Definition at line 62 of file scanner.cpp.

References grabDigits(), grabId(), grabSingle(), grabString(), grabSymbols(), pos(), string_buffer, token_map, tokens, types::TT_CHAR, types::TT_DIGIT, types::TT_NULL, types::TT_SINGLE, types::TT_SPACE, types::TT_STRING, types::TT_SYM, and types::TT_SYMBOL.

Referenced by mxx::TPValidator::parseUses().

◆ size()

size_t scan::Scanner::size ( ) const

Return the total number of scanned tokens.

Definition at line 487 of file scanner.cpp.

References tokens.

Referenced by operator[]().

Member Data Documentation

◆ string_buffer

TString scan::Scanner::string_buffer
private

source text buffer

Definition at line 65 of file scanner.hpp.

Referenced by Scanner(), grabDigits(), grabId(), grabSingle(), grabString(), grabSymbols(), parseFileName(), parseLineNumber(), and scan().

◆ token_map

TMap scan::Scanner::token_map
private

character classification map

Definition at line 66 of file scanner.hpp.

Referenced by grabSymbols(), and scan().

◆ tokens

std::vector<TToken> scan::Scanner::tokens
private

scanned token list

Definition at line 67 of file scanner.hpp.

Referenced by getTokens(), operator[](), removeEOL(), scan(), and size().

The documentation for this class was generated from the following files: