Open In App
Related Articles

Spoofing IP address when web scraping using Python

Like Article
Save Article
Report issue

In this article, we are going to scrap a website using Requests by rotating proxies in Python.

Modules Required

  • Requests module allows you to send HTTP requests and returns a response with all the data such as status, page content, etc.


requests.get(url, parameter) 

  • JSON JavaScript Object Notation is a format for structuring data. It is mainly used for storing and transferring data between the browser and the server. Python too supports JSON with a built-in package called json. This package provides all the necessary tools for working with JSON Objects including parsing, serializing, de-serializing, and many more.


  • Manually create a set of http proxies if you don’t have use rapidapi.(Here create_proxy() function is used to generate a set of http proxies using rapidapi)
  • Iterate the set of proxies and send a GET request using requests.get(url, proxies=proxies) to the website along with the proxies as parameters.


requests.get(url, proxies=proxies)

  • If the proxy is working perfectly then it should return an object of the URL.

Apart from working with the code, there are few more set-ups that need to be done, and given below are the details of these setups.

Using Rapidapi to get a set of proxies: 

  • Firstly, you need to buy a subscription of this API from rapidapi and then go to dashboard and select Python and copy the api_key.
  • Initialize the headers with the API key and the rapidapi host.


headers = {

       ‘x-rapidapi-key’: “paste_api_key_here”,

       ‘x-rapidapi-host’: “”


  • Send a GET request to the API along with headers ,


response = requests.request(“GET”, url, headers=headers)

  • This will return a JSON, parsing the text using json.loads(), we can find the proxy server address in the “curl” key.


response = json.loads(response.text)

proxy = response[‘curl’]

Sending Proxy in requests.get() as parameter:

Sending a GET request using requests.get() along with a proxy to this url which will return the proxy server address of current session.


 # Note : Opening in browser will show the current ip address of the session.

 proxies = ‘’

 page = requests.get(‘’, proxies={“http”: proxy, “https”: proxy})




import requests
import json
# Gets proxies from rapidapi to create
# a set of proxies.
# Use this function only if you have rapidapi key.
def create_proxy():
    # Initialise the headers and paste the API key
    # of proxy-orbit1 from rapidapi.
    headers = {
        'x-rapidapi-key': "paste_api_key_here",
        'x-rapidapi-host': ""
    # Sends a GET request to the above url along with api
    # keys which returns an object containing data in json
    # format which is then parsed using json.loads.
    response = requests.request("GET", url, headers=headers)
    response = json.loads(response.text)
    # The proxy server ip address is present in 'curl' key.
    proxy = response['curl']
    return proxy
# Main Function
if __name__ == "__main__":
    # Create an empty set and call the create_proxy()
    # function to generate a set of proxies from rapidapi.
    # Orbit proxy Rapid api key is required.
    proxies = set()
    print("Creating Proxy List")
    for __ in range(10):
    # If you do not have rapidapi then create a set of
    # proxies manually.
    # proxies = {'',
    # Iterate the proxies and check if it is working.
    for proxy in proxies:
        print("\nChecking proxy:", proxy)
            # returns the ip address
            # of the current session if a GET request is sent.
            page = requests.get('',
                                proxies={"http": proxy, "https": proxy})
            print("Status OK, Output:", page.text)
        except OSError as e:
            # Proxy returns Connection error



Last Updated : 13 Jul, 2021
Like Article
Save Article
Share your thoughts in the comments
Similar Reads