Lex code to count total number of tokens

Lex is a computer program that generates lexical analyzers and was written by Mike Lesk and Eric Schmidt. Lex reads an input stream specifying the lexical analyzer and outputs source code implementing the lex in the C programming language.

Tokens: A token is a group of characters forming a basic atomic chunk of syntax i.e. token is a class of lexemes that matches a pattern. Eg – Keywords, identifier, operator, separator.

Example:



Input: int p=0, d=1, c=2;

Output: 
total no. of tokens = 13

Below is the implementation of the above explanation:

filter_none

edit
close

play_arrow

link
brightness_4
code

/*Lex code to count total number of tokens */
  
%{ 
int n = 0 ;  
%} 
  
// rule section
%% 
  
//count number of keywords
"while"|"if"|"else" {n++;printf("\t keywords : %s", yytext);}  
  
// count number of keywords
"int"|"float" {n++;printf("\t keywords : %s", yytext);}   
  
// count number of identifiers
[a-zA-Z_][a-zA-Z0-9_]* {n++;printf("\t identifier : %s", yytext);} 
  
// count number of operators
"<="|"=="|"="|"++"|"-"|"*"|"+" {n++;printf("\t operator : %s", yytext);}
  
// count number of separators
[(){}|, ;]    {n++;printf("\t separator : %s", yytext);} 
  
// count number of floats
[0-9]*"."[0-9]+ {n++;printf("\t float : %s", yytext);}  
  
// count number of integers
[0-9]+ {n++;printf("\t integer : %s", yytext);}                        
  
.    ;
%% 
   
   
int main() 
  
      
    yylex();
      
    printf("\n total no. of token = %d\n", n);   
       

chevron_right


Output:



My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.



Improved By : Ravimaurya2



Article Tags :

2


Please write to us at contribute@geeksforgeeks.org to report any issue with the above content.