token_get_all

(PHP 4 >= 4.2.0, PHP 5, PHP 7, PHP 8)

token_get_all — Scinde un code source en éléments de base

Description

token_get_all(string $code, int $flags = 0): array

token_get_all() analyse la chaîne donnée source en utilisant l'analyseur lexical du moteur Zend.

Pour une liste des tokens, voir Liste des jetons de l'analyseur, ou utilisez la fonction token_name() pour traduire une valeur token dans une représentation sous forme de chaîne de caractères.

Liste de paramètres

code

Le source PHP à analyser.

flags

Drapeaux valides :

TOKEN_PARSE - Reconnait la possibilité d'utiliser les mots réservés dans des contextes spécifiques.

Valeurs de retour

Un tableau contenant la liste des descriptions des éléments. Chaque élément du tableau peut être un caractère unique (i.e.: ;, ., >, !, etc.), ou un tableau contenant un identifiant de token dans l'élément 0, la représentation textuelle originale de ce token dans l'élément 1, et le numéro de la ligne dans l'élément 2.

Exemples

Exemple #1 Exemple avec token_get_all()

<?php
$tokens = token_get_all('<?php echo; ?>');

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
    }
}
?>

Résultat de l'exemple ci-dessus est similaire à :

Line 1: T_OPEN_TAG ('<?php ')
Line 1: T_ECHO ('echo')
Line 1: T_WHITESPACE (' ')
Line 1: T_CLOSE_TAG ('?>')

Exemple #2 Exemple d'utilisation incorrecte de la fonction token_get_all()

<?php
$tokens = token_get_all('/* commentaire */');

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo "Ligne {$token[2]} : ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
    }
}
?>

Résultat de l'exemple ci-dessus est similaire à :

Ligne 1 : T_INLINE_HTML ('/* comment */')

Notez que dans le précédent exemple la chaîne est analsée comme une T_INLINE_HTML plutôt que T_COMMENT, ce qui aurait pu être attendu. Ceci est dû au fait qu'aucune balise ouvrante n'a été utilisée dans le code fourni. Cela correspond à mettre une commentaire en dehors des balises PHP dans un fichier classique.

Exemple #3 token_get_all() dans une classe en utilisant un mot réservé

<?php

$source = <<<'code'
<?php

class A
{
    const PUBLIC = 1;
}
code;

$tokens = token_get_all($source, TOKEN_PARSE);

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo token_name($token[0]) , PHP_EOL;
    }
}
?>

Résultat de l'exemple ci-dessus est similaire à :

T_OPEN_TAG
T_WHITESPACE
T_CLASS
T_WHITESPACE
T_STRING
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER

Sans le drapeau TOKEN_PARSE, le pénultième jeton (T_STRING) aurait du être T_PUBLIC.

Voir aussi

PhpToken::tokenize() - Splits given source into PHP tokens, represented by PhpToken objects.
token_name() - Retourne le nom d'un élément de code source

add a note

User Contributed Notes 8 notes

down

gomodo at free dot fr ¶

15 years ago


Yes, some problems (On WAMP, PHP 5.3.0 ) with get_token_all() 

1 : bug line numbers
 Since PHP 5.2.2 token_get_all()  should return Line numbers in element 2..
.. but for instance (5.3.0 on WAMP), it work perfectly only with PHP code (not HMTL miwed), but if you have some T_INLINE_HTML detected by token_get_all() ,  sometimes you find wrongs line numbers  (return next line)... :(

2: bug warning message can impact loops
Warning with php code uncompleted (ex : php code line by line) :
for example if a comment tag is not closed  token_get_all()  can block loops on this  warning :
Warning: Unterminated comment starting line

This problem seem not occur in CLI mod (php command line), but only in web mod.

Waiting more stability, used token_get_all()  only on PHP code (not HMTL miwed) :
First extract entirely PHP code (with open et close php tag), 
Second use token_get_all()  on the pure PHP code.

3 : Why there not function to extract PHP code (to extract HTML, we have Tidy..)?

Waiting, I used a function :

The code at end this post :
http://www.developpez.net/forums/d786381/php/langage/
fonctions/analyser-fichier-php-token_get_all/

This function not support :
- Old notation :  "<?  ?>" and "<% %>"
- heredoc syntax 
- nowdoc syntax (since PHP 5.3.0)

down

Dennis Robinson from basnetworks dot net ¶

15 years ago


I wanted to use the tokenizer functions to count source lines of code, including counting comments.  Attempting to do this with regular expressions does not work well because of situations where /* appears in a string, or other situations.  The token_get_all() function makes this task easy by detecting all the comments properly.  However, it does not tokenize newline characters.  I wrote the below set of functions to also tokenize newline characters as T_NEW_LINE.

<?php

define('T_NEW_LINE', -1);

function token_get_all_nl($source)
{
    $new_tokens = array();

    // Get the tokens
    $tokens = token_get_all($source);

    // Split newlines into their own tokens
    foreach ($tokens as $token)
    {
        $token_name = is_array($token) ? $token[0] : null;
        $token_data = is_array($token) ? $token[1] : $token;

        // Do not split encapsed strings or multiline comments
        if ($token_name == T_CONSTANT_ENCAPSED_STRING || substr($token_data, 0, 2) == '/*')
        {
            $new_tokens[] = array($token_name, $token_data);
            continue;
        }

        // Split the data up by newlines
        $split_data = preg_split('#(\r\n|\n)#', $token_data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY);

        foreach ($split_data as $data)
        {
            if ($data == "\r\n" || $data == "\n")
            {
                // This is a new line token
                $new_tokens[] = array(T_NEW_LINE, $data);
            }
            else
            {
                // Add the token under the original token name
                $new_tokens[] = is_array($token) ? array($token_name, $data) : $data;
            }
        }
    }

    return $new_tokens;
}

function token_name_nl($token)
{
    if ($token === T_NEW_LINE)
    {
        return 'T_NEW_LINE';
    }

    return token_name($token);
}

?>

Example usage:

<?php

$tokens = token_get_all_nl(file_get_contents('somecode.php'));

foreach ($tokens as $token)
{
    if (is_array($token))
    {
        echo (token_name_nl($token[0]) . ': "' . $token[1] . '"<br />');
    }
    else
    {
        echo ('"' . $token . '"<br />');
    }
}

?>

I'm sure you can figure out how to count the lines of code, and lines of comments with these functions.  This was a huge improvement on my previous attempt at counting lines of code with regular expressions.  I hope this helps someone, as many of the user contributed examples on this website have helped me in the past.

down

Ivan Ustanin ¶

6 years ago


As a caution: when using TOKEN_PARSE with an invalid php-file, one can get an error like this:
Parse error: syntax error, unexpected '__construct' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST) in  on line 15
Notice the missing filename as this function accepts a string, not a filename and thus has no idea of the latter.
However an exception would be more appreciated.

down

Theriault ¶

8 years ago


The T_OPEN_TAG token will include the first trailing newline (\r, \n, or \r\n), tab (\t), or space. Any additional space after this token will be in a T_WHITESPACE token.

The T_CLOSE_TAG token will include the first trailing newline (\r, \n, or \r\n; as described here http://php.net/manual/en/language.basic-syntax.instruction-separation.php). Any additional space after this token will be in a T_INLINE_HTML token.

down

nicolas dot grekas+php at gmail dot com ¶

16 years ago


Well, there is a way to parse for errors. See
http://www.php.net/manual/function.php-check-syntax.php#77318

down

-3

bart ¶

7 years ago


Not all tokens are returned as an array. The rule appears to be that if a token is not variable, but instead it is one particular constant string, it is returned as a string instead. You don't get a line number. This is the case for braces( "{", "}"), parentheses ("(", ")"), brackets ("[", "]"), comma (","), semi-colon (";"), and a whole slew of operator signs ("!", "=", "+", "*", "/", ".", "+=", ...).

down

-9

kevin at metalaxe dot com ¶

16 years ago


Rogier, thanks for that fix. This bug still exists in php 5.2.5. I did notice though that it is possible for a notice to pop up from your code. Changing this line:

            $temp[] = $tokens[0][2];

To read this:

            $temp[] = isset($tokens[0][2])?$tokens[0][2]:'unknown';

fixes this notice.

down

-11

rogier ¶

16 years ago


Complementary note to code below:
Note that only the FIRST 2 (or 3, if needed) array elements will be updated. 

Since I only encountered incorrect results on the FIRST occurence of T_OPEN_TAG, I wrote this quick fix.
Any other following T_OPEN_TAG are, on my testing system (Apache 2.0.52, PHP 5.0.3), parsed correctly.

So, This function assumes only a possibly incorrect first T_OPEN_TAG.
Also, this function assumes the very first element (and ONLY the first element) of the token array to be the possibly incorrect token.
This effectively translates to the first character of the tokenized source to be the start of a php script opening tag '<', followed by either 'php' OR '%' (ASP_style)

add a note