Python Token
A token is the smallest individual unit of a Python program
that has meaning to the Python interpreter.
For example:
This statement contains the following tokens:
- a → Identifier
- = → Operator
- 10 → Literal/Constant
Therefore:
A statement inside a Python program is a collection of tokens.
Types of Tokens in Python
The important types of tokens used in Python are:
- Keywords
- Identifiers
- Literals / Constants
- Operators
- Separators / Delimiters
1) Keywords
Keywords are special words that have a predefined meaning in the Python
language.
Keywords are also called reserved words because they have a
specific meaning defined by Python and cannot normally be used as identifiers.
Each keyword is used for a specific purpose.
Example
import math
# Calculate square root
print("Square root of 9:",
math.sqrt(9))
# Assign values to variables
a = 10
b = 5
# Check which number is greater
if a > b:
print(a)
else:
print(b)
In the above program:
- import is a keyword.
- if is a keyword.
- else is a keyword.
However:
- math, a, and b are identifiers.
- print and sqrt are names referring to
built-in/function objects.
How to Find Keywords in Python?
Python provides the built-in keyword module to check the keywords.
>>> import keyword
>>> keyword.kwlist
output:
['False', 'None', 'True', 'and', 'as',
'assert', 'async', 'await',
'break', 'case', 'class', 'continue', 'def',
'del', 'elif', 'else',
'except', 'finally', 'for', 'from', 'global',
'if', 'import', 'in',
'is', 'lambda', 'match', 'nonlocal', 'not',
'or', 'pass', 'raise',
'return', 'try', 'while', 'with', 'yield']
The exact list can vary between Python versions.
In current Python 3 versions, Python has 35 hard keywords.
Soft Keywords
Python also has soft keywords. These words have a special meaning
only in particular language contexts.
>>> keyword.softkwlist
output:
['_', 'case', 'match', 'type']
There are 4 soft keywords in current Python versions.
Therefore:
Python has 35 hard keywords and 4 soft keywords.
2) Identifiers
An identifier is a name created by the programmer to identify a
programming element.
Identifiers are used for naming different programming elements, such as:
- Variable names
- Function names
- Class names
- Module names
- Package names
- Other user-defined objects
Example
rollno = 12
Here, rollno is an identifier
because it is the name given to the variable.
Another example:
def calculate():
print("Hello")
Here, calculate is an identifier used as a function name.
Characters Allowed in an Identifier
An identifier can be created using:
- English alphabets (A-Z, a-z)
- Digits (0-9)
- Underscore (_)
However, there are certain rules that must be followed while creating
identifiers.
An identifier can contain letters, digits, and underscores, but it must
start with a letter or an underscore.
For example:
student
student1
student_name
_student
are valid identifiers.
The underscore (_) is the only
special character allowed in a normal Python identifier.
Rules for Creating Identifiers in
Python
Python follows certain rules while creating identifiers.
1. Identifier should not be a keyword
Keywords cannot be used as identifiers.
Python has reserved words such as if, else, for, while, break, pass, class, etc.
We cannot use these keywords as variable names or other identifiers.
Example
rollno = 12
rollno
Output:
12
But:
pass = 100
Output:
SyntaxError: invalid syntax
Similarly:
break = 1
Output:
SyntaxError: invalid syntax
However, Python is case-sensitive, so PASS is different from the keyword pass.
PASS = 5
PASS
Output:
5
Note: pass is a keyword, but PASS is a valid identifier.
2. Identifier should not start with a
digit
An identifier must start with a letter or an underscore (_).
An identifier can contain digits, but a digit cannot be the first
character.
Valid identifiers
n1 = 10
n2 = 20
n1
Output:
10
n2
Output:
20
Invalid identifier
3n = 30
Output:
SyntaxError: invalid decimal literal
An identifier also cannot start with a special symbol such as $.
$amt = 5
Output:
SyntaxError: invalid syntax
Identifier can start with an
underscore
_n = 100
_n
Output:
100
A single underscore can also be used:
_ = 200
_
Output:
200
Similarly:
__ = 300
__
Output:
300
3. There should not be any space in an
identifier
Spaces are not allowed inside an identifier.
For example:
player score = 100
This is invalid because there is a space between player and score.
Output:
SyntaxError: invalid syntax
Similarly:
employee no = 1
student rollno = 102
room number = 1
All of these are invalid because they contain spaces.
Use underscore instead of space
We can use an underscore (_) to separate words.
player_score = 100
player_score
Output:
100
Other examples:
employee_no = 1
student_rollno = 102
room_number = 1
These are valid identifiers.
4. Python identifiers are
case-sensitive
Python is a case-sensitive language.
This means uppercase and lowercase letters are treated as different.
For example:
rollno = 1
ROLLNO = 2
rollno
Output:
1
ROLLNO
Output:
2
Here, rollno and ROLLNO are two different identifiers.
Similarly:
name
Name
NAME
are treated as different identifiers.
5. There is no fixed small maximum
length for an identifier
Python does not specify a small fixed maximum length for identifiers.
For example:
aaaaaaaaaaaaaaaaaaaaaaa = 100
aaaaaaaaaaaaaaaaaaaaaaa
Output:
100
Another long identifier can also be used:
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
= 2000
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
Output:
2000
Therefore, Python allows identifiers to be very long.
However, it is better to use short, meaningful, and readable
identifiers rather than unnecessarily long names.
Simple Rule to Remember
An identifier can start with a letter or underscore, followed by letters,
digits, or underscores. It cannot contain spaces or special symbols, and it
cannot be a Python keyword.
L iteral
A literal is a value that we directly write in a Python program.
For example,10, 25.5, "Hello", and True are literals.Python uses these values directly in the program.
0 Comments