Web Scraping CryptoCurrency price and storing it in MongoDB using Python

Let us see how to fetch history price in USD or BTC, traded volume and market cap for a given date range using Santiment API and storing the data into MongoDB collection.

Python is a mature language and getting much used in the Cryptocurrency domain. MongoDB is a NoSQL database getting paired with Python in many projects which helps to hold details that got retrieved from Python Programs. PyMongo is a Python distribution containing tools for working with MongoDB and it is a very convenient way  to work with MongoDB from Python to do Create/Update/Delete/Read operations easily.

Let us see the code in using Santiment’s API in getting cryptocurrency prices for a given date range

Few Examples of cryptocurrency are :

bitcoin
ethereum
ripple
bitcoin-cash
litecoin
eos
cardano
stellar
neo
iota

1. In the place of id, we can pass bitcoin/ethereum or any cryptocurrency name which the below code can parse and get the data
2. For valid cryptocurrency name, data will be retrieved properly
3. In the place of from_date and to_date valid dates in valid date formats in yyyy-mm-dd pattern need to be given. For understanding purpose, it is given to take 7 days of data. We can able to get 1 month old data too. API call can able to get data for the given cryptocurrencies in the given date range or else if not available (due to invalid cryptocurrency name/invalid date range specification), we can catch the errors
4. In the below program, for easier understanding, taking bitcoin and ethereum in “IndexCoins.idx” file and hence in the place of id, they are passed

Example :

For the following input

For bitcoin above API call, returns data as below

Similarly for ethereum also, we will get value

Retrieved value has to be stored in MongoDB

Database : geeksforgeeks

Collection : Coll_santiment_Price

As we are going to take for 7 days of data, it has to be given in a loop

In MongoDB, _id is the default column which can be got by ObjectId(). Our process has multiple rows where each and every row is identified by means of “cryptocurrencyname” and “time”. In our code, we are having it as “id” and “time” respectively.

Let “Coll_santiment_Price” be created with 3 columns namely _id, id and time first. Then upon successful execution of API call, for the id and time, let API call output namely (verify column name from bitcoinprice.png) ‘priceBtc’, ‘priceUsd’,’volume’ and ‘marketcap’ are updated in a loop.

Code Implementation in Python

Python3

# importing the modules
from datetime import date, timedelta, datetime
from pymongo import MongoClient
import san
from bson.objectid import ObjectId

client = MongoClient('mongodb://localhost:27017/')
db1 = client.geeksforgeeks
data = {}

with open("IndexCoins.idx") as openfileobject:
    for line in openfileobject:

        # One by one File's cryptocurrency names are read
        id = line.strip()
        try:
            print(id)

            # Collecting the data for 7 days and hence range(7) is taken
            for idx in range(7):
                daystr = str(date.today() - timedelta(days = idx))

                # Coll_santiment_Price collection documents need to be created
                # It will have columns namely "_id", "time", "id",
                # "priceBtc", "priceUsd", "volume", "marketcap"
                data['id'] = id
                data['time'] = daystr

                # _id column for unique key and can be generated by using ObjectId()
                data['_id'] = ObjectId()

                # Initially create 3 columns in collection as rest of the
                # columns are updated after running santiment api call
                db1.Coll_santiment_Price.insert(data)
            try:
                # Santiment API call to get Cryptocurrency prices
                daa = san.get("prices/" + id,
                              from_date = "2020-08-20",
                              to_date = "2020-08-27",
                              interval = "1d")

                # API call output and for bitcoin it is given in
                # https://media.geeksforgeeks.org/wp-content/uploads/20200827191739/bitcoinprice.png
                print(daa)

            except:
                print("URL error")
                continue;

            # 7 days output
            for idx in range(7):

                daystr = str(date.today() - timedelta(days=idx))

                # API call output for the above chosen date
                row = daa.loc[daystr]

                # priceBtc, priceUsd, volume and marketcap are
                # collected and stored in separate variables
                priceBtc = row['priceBtc']
                priceUsd = row['priceUsd']
                volume = row['volume']
                marketcap = row['marketcap']
                print(id, daystr, priceBtc, priceUsd, volume, marketcap)
                try:
                    # Update the collection with above details against
                    # cryptocurrency id and time  i.e. bitcoin and 2020-08-27
                    db1.Coll_santiment_Price.update(
                        {'time': daystr, 'id': id},
                        {"$set": {"priceBtc": priceBtc,
                                  "priceUsd": priceUsd,
                                  "volume": volume,
                                  "marketcap": marketcap,
                                  }
                         },
                        upsert = True
                        )
                except Exception as e:
                    print(e)
        except:
            print("Error")

Sample Output : (For bitcoin)

Sample Output : (For Ethereum)

Santiment API calls for getting prices are very much used across cryptocurrency projects. As we are getting historical data, for data analytics, this is very much useful. Moreover, it is freely accessible and hence any beginner-level projects can use it without any issues. Price variations are not much varying and hence well set for demo and small scale projects.