How to build a Math Expression Tokenizer using JavaScript ?

A math expression tokenizer is a fundamental component in parsing mathematical expressions. It breaks down a mathematical expression into smaller units called tokens, which are easier to process and evaluate. In JavaScript, building a math expression tokenizer can be achieved through various approaches, each with its advantages and considerations.

These are the following approaches:

Regular Expression Approach

This approach utilizes JavaScript’s regular expression capabilities to match and extract tokens from a given math expression. Regular expressions are crafted to identify numbers, operators, parentheses, and other relevant components of mathematical expressions.

Syntax:

const regex = /pattern/g;
const tokens = expression.match(regex);

Example: To demonstrate defining a regular expression pattern to match numbers, operators, and parentheses. We then use the match method to extract tokens from the expression.

JavaScript

function tokenizeRegex(expression) {
    const regex = /\d+\.?\d*|\+|-|\*|\/|\(|\)/g;
    return expression.match(regex);
}

const expression = "2 * (3 + 5) / 4";
console.log(tokenizeRegex(expression));

Output

[
  '2', '*', '(',
  '3', '+', '5',
  ')', '/', '4'
]

Iterative Approach with Custom Logic

In this approach, we iterate through each character of the input expression, categorizing them into tokens based on predefined rules. Custom logic is employed to identify numbers, operators, and other elements of the expression.

Syntax:

const tokens = [];
for (let i = 0; i < expression.length; i++) {
    // Custom logic to categorize characters into tokens
}

Example: In this example, we iterate through each character of the expression. We identify numbers by checking for digits and decimals. Operators and parentheses are identified based on their characters.

JavaScript

function tokenizeCustom(expression) {
    const tokens = [];
    let currentToken = '';

    for (let i = 0; i < expression.length; i++) {
        const char = expression[i];

        if (/\d/.test(char) || char === '.') {
            currentToken += char;
        } else {
            if (currentToken !== '') {
                tokens.push(currentToken);
                currentToken = '';
            }
            if (char.trim() !== '') {
                tokens.push(char);
            }
        }
    }

    if (currentToken !== '') {
        tokens.push(currentToken);
    }

    return tokens;
}

const expression = "2 * (3 + 5) / 4";
console.log(tokenizeCustom(expression));

Output

[
  '2', '*', '(',
  '3', '+', '5',
  ')', '/', '4'
]