PHP token_get_all() Function

The token_get_all() function is an inbuilt function in PHP that is used to tokenize a given PHP source code string into an array of tokens. This function is particularly useful for analyzing, parsing, or manipulating PHP code programmatically.

Syntax:

token_get_all(string $code, int $flags = 0)

Parameters: This function accepts two parameters that are described below.

• $code: This is the string that is going to tokenize.
• $flags: This is an optional parameter that modifies the tokenization behavior of the string.

Return Values: The token_get_all() returns array contains elements where each element represents a token.

Program 1: The following program demonstrates the token_get_all() function.

PHP

<?php

$code = '<?php echo "Hello, world!"; ?>';
$tokens = token_get_all($code, TOKEN_PARSE);

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo "Token: " . token_name($token[0])
              . " - Content: " . $token[1] . "\n";
    } else {
        echo "Token: " . $token . "\n";
    }
}

?>

Output

Token: T_OPEN_TAG - Content: <?php 
Token: T_ECHO - Content: echo
Token: T_WHITESPACE - Content:  
Token: T_CONSTANT_ENCAPSED_STRING - Content: "Hello, world!"
Token: ;
Token: T_WHITESPACE - Content: ...

Program 2: The following program demonstrates the token_get_all() function.

PHP

<?php

$code = '<?php
   $message = "Hello, ";
   $name = "John";
   echo $message . $name;
?>';

$tokens = token_get_all($code);

foreach ($tokens as $token) {
       if (is_array($token)) {
           list($tokenId, $tokenContent) = $token;
           echo "Token: " . token_name($tokenId) .
             " - Content: " . $tokenContent . "\n";
       } else {
           echo "Token: " . $token . "\n";
       }
}

?>

Output

Token: T_OPEN_TAG - Content: <?php

Token: T_WHITESPACE - Content:    
Token: T_VARIABLE - Content: $message
Token: T_WHITESPACE - Content:  
Token: =
Token: T_WHITESPACE - Content:  
Token: T_CONSTAN...

Reference: https://www.php.net/manual/en/function.token-get-all.php