Python Tokens and Character Sets

Python is a general-purpose, high-level programming language. It was designed with an emphasis on code readability, and its syntax allows programmers to express their concepts in fewer lines of code, and these codes are known as scripts. These scripts contain character sets, tokens, and identifiers. In this article, we will learn about these character sets, tokens, and identifiers.

Character set

A character set is a set of valid characters acceptable by a programming language in scripting. In this case, we are talking about the Python programming language. So, the Python character set is a valid set of characters recognized by the Python language. These are the characters we can use during writing a script in Python. Python supports all ASCII / Unicode characters that include:

• Alphabets: All capital (A-Z) and small (a-z) alphabets.
• Digits: All digits 0-9.
• Special Symbols: Python supports all kind of special symbols like, ” ‘ l ; : ! ~ @ # $ % ^ ` & * ( ) _ + – = { } [ ] \ .
• White Spaces: White spaces like tab space, blank space, newline, and carriage return.
• Other: All ASCII and UNICODE characters are supported by Python that constitutes the Python character set.

Tokens 

A token is the smallest individual unit in a python program. All statements and instructions in a program are built with tokens. The various tokens in python are :

1. Keywords: Keywords are words that have some special meaning or significance in a programming language. They can’t be used as variable names, function names, or any other random purpose. They are used for their special features. In Python we have 33 keywords some of them are: try, False, True, class, break, continue, and, as, assert, while, for, in, raise, except, or, not, if, elif, print, import, etc.

Python3

# for loop
for x in range(1, 9):

    # Print the value of x
    print(x)

    # Check if the value of x is less than 6
    # Here if the value of x is less than 6
    # then the loop will continue
    # Here, if, continue, else, break,
    # for loop are keywords
    if x < 6:
        continue

    # If i greater than 6 then break loop
    else:
        break

Output:

1
2
3
4
5
6

2. Identifiers: Identifiers are the names given to any variable, function, class, list, methods, etc. for their identification. Python is a case-sensitive language and it has some rules and regulations to name an identifier. Here are some rules to name an identifier:-

• As stated above, Python is case-sensitive. So case matters in naming identifiers. And hence geeks and Geeks are two different identifiers.
• Identifier starts with a capital letter (A-Z) , a small letter (a-z) or an underscore( _ ). It can’t start with any other character.
• Except for letters and underscore, digits can also be a part of identifier but can’t be the first character of it.
• Any other special characters or whitespaces are strictly prohibited in an identifier.
• An identifier can’t be a keyword.

For Example: Some valid identifiers are gfg, GeeksforGeeks, _geek, mega12, etc. While 91road, #tweet, i am, etc. are not valid identifiers.

Python3

# Here GFG and b are the identifier
GFG = 'Hello'
b = "Geeks"

# Driver code
print(GFG)
print(b)

Output:

Hello
Geeks

3. Literals or Values: Literals are the fixed values or data items used in a source code. Python supports different types of literals such as:

(i) String Literals: The text written in single, double, or triple quotes represents the string literals in Python. For example: “Computer Science”, ‘sam’, etc. We can also use triple quotes to write multi-line strings.

Python3

# String Literals
a = 'Hello'
b = "Geeks"
c = '''Geeks for Geeks is a
        learning platform'''

# Driver code
print(a)
print(b)
print(c)

Output

Hello
Geeks
Geeks for Geeks is a 
        learning platform

(ii) Character Literals: Character literal is also a string literal type in which the character is enclosed in single or double-quotes.

Python3

# Character Literals
a = 'G'
b = "W"

# Driver code
print(a)
print(b)

Output:

G
W

(iii) Numeric Literals: These are the literals written in form of numbers. Python supports the following numerical literals:

• Integer Literal: It includes both positive and negative numbers along with 0. It doesn’t include fractional parts. It can also include binary, decimal, octal, hexadecimal literal.
• Float Literal: It includes both positive and negative real numbers. It also includes fractional parts.
• Complex Literal: It includes a+bi numeral, here a represents the real part and b represents the complex part.

Python3

# Numeric Literals
a = 5
b = 10.3
c = -17

# Driver code
print(a)
print(b)
print(c)

Output

5
10.3
-17

(iv) Boolean Literals: Boolean literals have only two values in Python. These are True and False.

Python3

# Boolean Literals
a = 3
b = (a == 3)
c = True + 10

# Driver code
print(a, b, c)

Output

3 True 11

(v) Special Literals: Python has a special literal ‘None’. It is used to denote nothing, no values, or the absence of value.

Python3

# Special Literals
var = None
print(var)

Output

None

(vi) Literals Collections: Literals collections in python includes list, tuple, dictionary, and sets.

1. List: It is a list of elements represented in square brackets with commas in between. These variables can be of any data type and can be changed as well.
2. Tuple: It is also a list of comma-separated elements or values in round brackets. The values can be of any data type but can’t be changed.
3. Dictionary: It is the unordered set of key-value pairs.
4. Set: It is the unordered collection of elements in curly braces ‘{}’.

Python3

# Literals collections
# List
my_list = [23, "geek", 1.2, 'data']

# Tuple
my_tuple = (1, 2, 3, 'hello')

# Dictionary
my_dict = {1:'one', 2:'two', 3:'three'}

# Set
my_set = {1, 2, 3, 4}

# Driver code
print(my_list)
print(my_tuple)
print(my_dict)
print(my_set)

Output

[23, 'geek', 1.2, 'data']
(1, 2, 3, 'hello')
{1: 'one', 2: 'two', 3: 'three'}
{1, 2, 3, 4}

4. Operators: These are the tokens responsible to perform an operation in an expression. The variables on which operation is applied are called operands. Operators can be unary or binary. Unary operators are the ones acting on a single operand like complement operator, etc. While binary operators need two operands to operate.

Python3

# Operators
a = 12

 # Unary operator
b = ~ a

# Binary operator
c = a+b

# Driver code
print(b)
print(c)

Output

-13
-1

5. Punctuators: These are the symbols that used in Python to organize the structures, statements, and expressions. Some of the Punctuators are: [ ] { } ( ) @  -=  +=  *=  //=  **==  = , etc.