String Tokenization in C

In C, tokenization is the process of breaking the string into smaller parts using delimiters (characters treated as separators) like space, commas, a specific character, or even a string. Those smaller parts are called tokens where each token is a substring of the original string separated by the delimiter.

Tokenizing can be done manually using user-defined functions but C language also provides a built-in function strtok() defined inside <string.h> standard string library to simplify this process.

strtok() Function

The strtok() function splits the string into substrings by separating the original string using delimiters. It is defined inside the <string.h> header so we need to include it before using the strtok() function.

Syntax

char* strtok(char *str, const char *delimiter)

Parameters

• str: Original String that is to be processed
• delimiter: The separator using which we tokenize the string.

How to use strtok() function?

Initially, when we pass all the parameters of the string, this function will return only the first token of the input string. After that, we need to pass “NULL” instead of the input string to obtain the following tokens. It will return NULL when there is no token left.

For Example,

Assume that the input original string is “This is an example string” and the delimiter is ” ” (space). If we pass these parameters to the strtok() function, the function will return the first token “This”.

To obtain the remaining tokens, we need to pass “NULL” instead of the input string. If we pass “NULL”, the strtok() will continue from where it left. In the last step, strtok() returned the first token “This”. so, if we “NULL” it will continue from the next token “is” and so on until there are no more tokens left.

Examples of String Tokenization in C

Example 1:

C

// C program to illustrate the use of strtok() in C
#include <stdio.h>
#include <string.h>

int main()
{

    // Original string
    char str[] = "GEEKS-FOR-GEEKS";

    // Space is used as the delimiter to split
    char delimiter[] = "-";

    // Declare empty string to store token
    char* token;

    printf("Initial String: %s", str);
    printf("\nAfter Tokenization: \n");
    // Get the first token
    token = strtok(str, delimiter);

    // using loop to get the rest of the token
    while (token) {
        printf("%s\n", token);
        token = strtok(NULL, delimiter);
    }

    return 0;
}

Output

Initial String: GEEKS-FOR-GEEKS
After Tokenization: 
GEEKS
FOR
GEEKS

Example 2:

C

// C program to illustrate the strtok() function
#include <stdio.h>
#include <string.h>

int main()
{

    // Original string
    char str[] = "This is an example string";

    // Space is used as the delimiter to split
    char delimiter[] = " ";

    // Declare empty string to store token
    char* token;

    // Get the first token
    token = strtok(str, delimiter);

    // Continue upto the last token
    while (token != NULL) {
        printf("Token: %s\n", token);
        // Pass NULL to get next token
        token = strtok(NULL, delimiter);
    }

    return 0;
}

Output

Token: This
Token: is
Token: an
Token: example
Token: string