Difference Between Tokens and Terminals

Tokens and terminals are similar words and are often used interchangeably. But there is a conceptual difference between both terms, the differences are discussed in this article. 

Tokens

Tokens are alphanumeric characters. It is the smallest unit of grammar in programming languages. When we give input to the lexical analyzer it reads the characters and converts them to tokens which later proceeds through further phases of compilation. Tokens are categorized into various types: Keywords, Operators, Strings, Constants, Special Characters, and Identifiers. Example: A, @, b, (, ), etc. 

Terminals

The terminal is a symbol that appears on the right side of the production rule and cannot be changed using the grammar rules. Terminal symbols are basically a set of tokens and are basically characters from which strings are produced. They are basically represented by using lower case letters. Example a, b, c, etc.

Difference Between Tokens and Terminals

S.No	Token	Terminal
1.	Tokens are a set of strings used in a programming language.	Terminals are a set of characters used in production rules.
2.	The compiler breaks a program into the smallest units known as tokens which are passed through various stages of the compiler.	Symbols or tokens are called terminals when they are used in formal languages.
3.	Tokens can be written in upper case letters.	Terminals are written only in lower case letters.
4.	It can have letters, numbers, and special characters.	It has only alphabets.
5.	They are generated by a lexical analyzer.	They are defined by formal languages.
6.	Tokens are used to describe the corresponding characters in the source.	Terminals are used to describe the working of the parser that is with which it is working.
7.	Examples are A, @, b, (, ), etc	Examples are a, b, c, d, etc.