Lex Class Reference

A simple lexer for tokenizing an input string and reporting errors. More...

#include <lexer.hpp>

Classes
struct	Pattern
	Internal helper: a regex pattern and its associated token type. More...
struct	Token
	Represents a single token produced by the lexer. More...

Public Member Functions
	Lex (std::string input, bool from_string)
void	Tokenize ()
Token	Next ()
	Retrieves the next token from input string in a lazy way.
std::string &	input ()
	Access the raw input buffer.

Static Public Member Functions
static std::string	format_error_window (const std::string &input, size_t pos, size_t &out_err_line, size_t &out_err_col, size_t context_lines=2, size_t max_line_width=40, size_t window_width=20)
	Generate a formatted snippet around an error location.

Private Attributes
std::string	filename_
	Path of the file being lexed.
std::string	input_
	Complete contents of the input file.
Iterator	iter_
Iterator	end_
size_t	current_ {0}
std::vector< Token >	tokens_

Detailed Description

A simple lexer for tokenizing an input string and reporting errors.

Constructor & Destructor Documentation

Lex()

Lex::Lex	(	std::string	input,
		bool	from_string )

Construct a lexer from a raw string instead of a file. The boolean argument is only used to differentiate the constructor signature.

Member Function Documentation

format_error_window()

std::string Lex::format_error_window ( const std::string & input, size_t pos, size_t & out_err_line, size_t & out_err_col, size_t context_lines = 2, size_t max_line_width = 40, size_t window_width = 20 )

static

Generate a formatted snippet around an error location.

Given an input string and a byte-offset pos, extracts a few lines of context and places a caret (^) under the error column. Also computes the zero-based line (out_err_line) and column (out_err_col).

Parameters

	input	The full text being lexed.
	pos	Byte-offset in input where the error occurred.
[out]	out_err_line	Zero-based index of the line containing the error.
[out]	out_err_col	Zero-based index of the column within that line.
	context_lines	Number of lines of context before/after (default 2).
	max_line_width	Maximum characters to show per non-error line (40).
	window_width	Characters shown on either side of the error (20).

Returns

std::string A multi-line string highlighting the error window.

input()

std::string & Lex::input

(

)

Access the raw input buffer.

Returns

std::string& A mutable reference to the entire input string.

Next()

Lex::Token Lex::Next

(

)

Retrieves the next token from input string in a lazy way.

Returns

Token The next token in the sequence; EOF if there is no more symbols.

---

The documentation for this class was generated from the following files:

• include/lexer.hpp
• src/lexer.cpp