The User-Agent (UA) string is one of the most important factors in website identification in Web Scraping. It is like a fingerprint that a client or targeted website can identify. With the help of this, you can retrieve the data by shuffling the user agent into Python requests. It typically includes information about the software, operating system, and sometimes additional details such as the device type or version numbers. In this article, we will explain User Agent in Python Request.
What is a User-Agent Header in Python Request?
The user agent is a key component of the HTTP header sent with every HTTP request. The User-Agent header is one of many standard request headers. These headers contain information the server uses to generate its response, such as the preferred content format and language. It identifies the requester's device including information about the device type, operating system, browser name, and version. For example, The User Agent String Looks like this
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
or
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'"
- Application (Mozilla/5.0): This identifies the application making the request, which in this case is a generic application using the Mozilla compatibility standard.
- Operating System (Macintosh; Intel Mac OS X 10.15.7): This specifies the operating system the user is running, which is macOS Catalina (version 10.15.7).
- Layout Engine (AppleWebKit/537.36): This indicates the layout engine used to render web pages, which here is AppleWebKit.
- Browser Engine (KHTML, like Gecko): This specifies the underlying browser engine that powers the layout engine. Here, it indicates KHTML (Kranberry HTML) which is the original rendering engine used in Safari. It also mentions "like Gecko" for compatibility purposes, signifying it adheres to the Gecko engine's standards used by Firefox.
- Version (Chrome/123.0.0.0): This reveals the version of the browser being used, which is Chrome 123.0.0.0.
- Additional Info (Safari/537.36-Let's): This part contains additional information specific to the browser. Here, "Safari/537.36" indicates the version of Safari the browser is compatible with, and "Let's" seems to be a custom addition that might not have a standard meaning.
User Agent in Python Request
Below, are the explanation of how to set user agent using Python requests. To set the Python request user agent, pass a dictionary containing the new string to your request configuration.
Set User Agent in Python Using Request
In this example, below code imports the 'requests' library, sets custom headers for the user-agent, then sends a GET request to 'http://httpbin.org/headers' with those headers. Finally, it prints out the status code of the response and the response text.
import requests as r
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'}
response = r.get('http://httpbin.org/headers', headers=headers)
print("The Status Code", response.status_code)
print("The Response Text",response.text)
Output:
The Status Code 200
The Response Text {
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-661e27a2-5742f30f09e3ffcc5b31eb99"
}
}
Set Random User Agent Using Python Requests
In this example, below code randomly selects a user-agent from a list and sends a GET request to 'http://httpbin.org/headers' with that user-agent. It repeats this process five times, printing the response text each time. This is typically used for testing purposes to check how a server responds to different user-agents.
import random
import requests
user_agents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
]
for i in range(5):
headers = {'User-Agent': random.choice(user_agents)}
response = requests.get('http://httpbin.org/headers', headers=headers)
print(response.text)
Output
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-661e26e8-3e64ab07632b4a0344303771"
}
}
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15",
"X-Amzn-Trace-Id": "Root=1-661e26e8-16b7c776465b78e91ec10843"
}
}
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-661e26f2-670057b1104122f901f73f4f"
}
}
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-661e26f2-2504d8432784a1907f91ae89"
}
}
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15",
"X-Amzn-Trace-Id": "Root=1-661e26f2-0fd2e3fd45ece1b4374b2e07"
}
}