Project Title : Distributed Downloading System
Introduction: Working in an organization where downloading data is limited to few hundred Mbs or maybe a few Gbs? And guess what you need to download a software which has a size twice, thrice or maybe 4 times your data limit for a day. Do you think that you can wait for 2, 3 or 4 days to get the ware downloaded? LOL.
Want to bluff your organization with a faster download of the file where speed is constant (say FUP speed -> 2mbps ) to all the users? Yep! you heard it right. Get 2 friends and get the job done 2 times faster. Get n friends together and the job is done n times
Conceptual framework
Above mentioned are very few basic use cases of our project. Now let us talk about our project.
The idea of our project is to provide a platform for the users to download large files in less time. This is done by promoting download of the file in the form of a specific number of partitions. Once all the partitions have been downloaded, they can be merged using the app to obtain the original file. The project comprises of two parts:
1. Server: A server implemented on Node where a request is received from the clients. This request includes a URL to the file which user has to download and the number of partitions in which the file has to be downloaded (Let us assume the number of partitions is n). The server in response sends n unique keys. Now the file is downloaded to the server and divided into n parts (each chunk of size = OrignalFileSize/n). Once the files are divided, each data chunk is hashed with a unique key. {This key is generated by combining the download u_id appended with the serial number of the chunk}. Now when the server receives a download request of a chunk. The server sends the raw data chunk.
The JSON object which needs to be sent to the server must look like this:
{ parts : '20'
} If no error in url link or no internal error then a response as such is sent back { "__v": 0,
"name": "somename",
"ext" : ".ext"
"partCount": 3,
"_id": "58c522831a3a1919a103c863",
"reason": "none",
"parts": [],
"createdOn": "2017-03-12T10:27:15.827Z",
"status": "downloading"
} If some error is occurred then err is return with a status code of 500.
To initiate downloading parts send a POST request to localhost:3000/retrieve/status.
The response looks like this.
{ "_id": "58c52387f4e3e81a0ccaa1cd",
"name": "somename",
"ext": ".ext",
"partCount": 3,
"__v": 1,
"reason": "none",
"createdOn": "2017-03-12T10:31:35.037Z",
"status": "done"
} |
2. Android client: An android app which allows a user to make initial download request {The user making this request will be flagged as admin for this particular file}. As said earlier, this request has the URL and a number in which the file will be downloaded. Client receives n service keys. Each keys will be stored with the admin. Now the admin sends these n keys to her n friends. Each of them downloads the file and sends it back to the admin.
Tools Used: First things first, the server is Node based. The app has been created using Android Studio. The coding has been done using Java while the UI has been designed using XML. The local database of the app (for managing downloads) has been created using SQLite. The interactions with the API have been done using Volley library of Android which enables fast networking for apps and the requests to the server are being made in the form of JSON objects.
Application: The project has many real life applications. Two of them which demonstrate its utility are:
- Faster Downloads: This can be achieved by downloading partitions of the file onto different devices in parallel and then merging them. For example, if a file of size 5 GB is to be downloaded, it can be downloaded by five different people on their devices and then send their parts to one device and merge them. This reduces the download time to about 1/5th of the time required to download the entire file.
- Batching Downloads: If a person is getting a fixed amount of data from his ISP per day, he can download different parts of the file every day until he has all the parts. For example, in our hostel, we get 1GB of data per day which gets renewed at midnight. If the file isn’t required very urgently a student can wait for a minimum of 5 days to download one partition of 1 GB each day and finally merge them.
Status:
The project is under development. Given below are the GitHub links to the server repository and the android client repository. Current status of the project is that server gets the request, downloads the file and divides it into n parts (sent by the user) {This request is obviously sent from the android client}. The android client yet does not has the ability to merge all the files once they are received. Also, the current UX is a bit laggy. But nevertheless, the developers are finding ways to complete the project and launch it as a proper product in the market.
Future Plans:
1. Get an iOS developer for an iOS client.
2. Develop a desktop client to remove the space dependence. {mobile phones have less space than PC/laptop}
3. Develop a terminal based system for those who love to give commands.
Team Members:
1. Avinash Sharma
2. Dravit Lochan Gupta
3. Priyansh Gupta
4. Prabhat Shukla