Web Scraping CryptoCurrency price and storing it in MongoDB using Python
Let us see how to fetch history price in USD or BTC, traded volume and market cap for a given date range using Santiment API and storing the data into MongoDB collection.
Python is a mature language and getting much used in the Cryptocurrency domain. MongoDB is a NoSQL database getting paired with Python in many projects which helps to hold details that got retrieved from Python Programs. PyMongo is a Python distribution containing tools for working with MongoDB and it is a very convenient way to work with MongoDB from Python to do Create/Update/Delete/Read operations easily.
Let us see the code in using Santiment’s API in getting cryptocurrency prices for a given date range
Few Examples of cryptocurrency are :
bitcoin ethereum ripple bitcoin-cash litecoin eos cardano stellar neo iota
- In the place of id, we can pass bitcoin/ethereum or any cryptocurrency name which the below code can parse and get the data
- For valid cryptocurrency name, data will be retrieved properly
- In the place of from_date and to_date valid dates in valid date formats in yyyy-mm-dd pattern need to be given. For understanding purpose, it is given to take 7 days of data. We can able to get 1 month old data too. API call can able to get data for the given cryptocurrencies in the given date range or else if not available (due to invalid cryptocurrency name/invalid date range specification), we can catch the errors
- In the below program, for easier understanding, taking bitcoin and ethereum in “IndexCoins.idx” file and hence in the place of id, they are passed
For the following input
For bitcoin above API call, returns data as below
Similarly for ethereum also, we will get value
Retrieved value has to be stored in MongoDB
Database : geeksforgeeks
Collection : Coll_santiment_Price
As we are going to take for 7 days of data, it has to be given in a loop
In MongoDB, _id is the default column which can be got by ObjectId(). Our process has multiple rows where each and every row is identified by means of “cryptocurrencyname” and “time”. In our code, we are having it as “id” and “time” respectively.
Let “Coll_santiment_Price” be created with 3 columns namely _id, id and time first. Then upon successful execution of API call, for the id and time, let API call output namely (verify column name from bitcoinprice.png) ‘priceBtc’, ‘priceUsd’,’volume’ and ‘marketcap’ are updated in a loop.
Code Implementation in Python
Sample Output : (For bitcoin)
Sample Output : (For Ethereum)
Santiment API calls for getting prices are very much used across cryptocurrency projects. As we are getting historical data, for data analytics, this is very much useful. Moreover, it is freely accessible and hence any beginner-level projects can use it without any issues. Price variations are not much varying and hence well set for demo and small scale projects.