Tokenizing a string in C++

Tokenizing a string denotes splitting a string with respect to a delimiter. There are many ways we can tokenize a string. Let’s see each of them:

stringstream

A stringstream associates a string object with a stream allowing you to read from the string as if it were a stream.

Below is the C++ implementation :

// Tokenizing a string using stringstream
#include <bits/stdc++.h>

using namespace std;

int main()
{
    
    string line = "GeeksForGeeks is a must try";
    
    // Vector of string to save tokens
    vector <string> tokens;
    
    // stringstream class check1
    stringstream check1(line);
    
    string intermediate;
    
    // Tokenizing w.r.t. space ' '
    while(getline(check1, intermediate, ' '))
    {
        tokens.push_back(intermediate);
    }
    
    // Printing the token vector
    for(int i = 0; i < tokens.size(); i++)
        cout << tokens[i] << '\n';
}

Output:



GeeksForGeeks
is
a
must
try


strtok()

// Splits str[] according to given delimiters.
// and returns next token. It needs to be called
// in a loop to get all tokens. It returns NULL
// when there are no more tokens.
char * strtok(char str[], const char *delims);

Below is the C++ implementation :


// C/C++ program for splitting a string
// using strtok()
#include <stdio.h>
#include <string.h>

int main()
{
    char str[] = "Geeks-for-Geeks";

    // Returns first token 
    char *token = strtok(str, "-");

    // Keep printing tokens while one of the
    // delimiters present in str[].
    while (token != NULL)
    {
        printf("%s\n", token);
        token = strtok(NULL, "-");
    }

    return 0;
}


Output:

Geeks
for
Geeks


Another Example of strtok() :


// C code to demonstrate working of
// strtok
#include <string.h>
#include <stdio.h>

// Driver function
int main()
{
 // Declaration of string
    char gfg[100] = " Geeks - for - geeks - Contribute";

    // Declaration of delimiter
    const char s[4] = "-";
    char* tok;

    // Use of strtok
    // get first token
    tok = strtok(gfg, s);

    // Checks for delimeter
    while (tok != 0) {
        printf(" %s\n", tok);

        // Use of strtok
        // go through other tokens
        tok = strtok(0, s);
    }

    return (0);
}


Output:

Geeks
 for
 geeks
 Contribute


strtok_r()

Just like strtok() function in C, strtok_r() does the same task of parsing a string into a sequence of tokens. strtok_r() is a reentrant version of strtok().

There are two ways we can call strtok_r()

// The third argument saveptr is a pointer to a char * 
// variable that is used internally by strtok_r() in 
// order to maintain context between successive calls
// that parse the same string.
char *strtok_r(char *str, const char *delim, char **saveptr);

Below is a simple C++ program to show the use of strtok_r() :


// C/C++ program to demonstrate working of strtok_r()
// by splitting string based on space character.
#include<stdio.h>
#include<string.h>

int main()
{
    char str[] = "Geeks for Geeks";
    char *token;
    char *rest = str;

    while ((token = strtok_r(rest, " ", &rest)))
        printf("%s\n", token);

    return(0);
}


Output:

Geeks
for
Geeks




Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

Recommended Posts:



2.5 Average Difficulty : 2.5/5.0
Based on 2 vote(s)