What are tokens and lexemes

Tokens

• Definition: A token is a category or type of lexical unit that represents a specific type of element in the source code. Tokens are the building blocks that a lexical analyzer (or lexer) identifies during the scanning process.
• Examples: Common tokens include keywords (e.g., if, while), identifiers (e.g., variable names like x, total), operators (e.g., +, -), and punctuation (e.g., ;, {).

Lexemes

• Definition: A lexeme is the actual sequence of characters in the source code that matches a token type. It's the specific instance of a token found in the text.
• Examples: In the code snippet int count = 10;, the lexemes are int, count, =, 10, and ;. Each of these corresponds to a specific token type: int is a keyword token, count is an identifier token, = is an operator token, 10 is a literal token, and ; is a punctuation token.