class Rley::Lexical::Token

In Rley, a (lexical) token is an object created by a lexer (tokenizer) and passed to the parser. Such token an object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme ‘foo’ is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that ‘foo’ is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given lexeme is an instance of given terminal symbol.