Data Scraping for Android Apps using google-play-scraper in Node.js

The most commonly existing method for scraping the web is the one in which we use selenium & beautifulsoup in Python. Even though it helps us in a variety of tasks, if we are specifically looking to extract information about an already existing android app on the google play store for research purposes or even for self-use, there’s another way of doing things.

The google-play-scraper is a Node.js module that helps in scraping App data from the Google Play store by providing methods that make our job easy.

Installation:

npm install google-play-scraper

Important methods:
1. app: It returns the entire data associated with that application.

Parameters:

• appId: The id associated with the app on Google Play. It can be seen in the “?id=” of the URL.
• lang: It is an optional parameter. It has a default value of “en” for English. If we want out app data to be fetched in a different language. We can provide the two-letter code of that language to this parameter.
• country: It is also an optional parameter. It has a default value of “us” for the United States of America. This parameter is helpful when the app we’re looking at is available only in specific countries.

Example:

var gPlayScraper = require('google-play-scraper');

gPlayScraper.app({appId: 'free.programming.programming'})
    .then(console.log, console.log);

Output:

[ { title: 'Learn DS & Algo, Programming Interview Preparation',
  description:
   'GeeksforGeeks is a one-stop destination for programmers.
   The app features 20000+ Programming Questions, 40, 000+....',
  descriptionHTML:
   'GeeksforGeeks is a one-stop destination for programmers.....',
  summary:
   'Learn Data Structures Algorithms, C Programming, C++,
    Java, Python, JS, Aptitude',
  installs: '500, 000+',
  minInstalls: 500000,
  score: 4.6594124,
  ...
  developer: 'GeeksforGeeks',
  developerId: 'GeeksforGeeks',
  developerEmail: 'support@geeksforgeeks.org',
  developerWebsite: 'https://www.geeksforgeeks.org/',
  developerAddress: 'Noida, UP, India',
  privacyPolicy: 'https://www.geeksforgeeks.org/privacy-policy/',
  developerInternalID: '5323597028845965498',
  genre: 'Education',
  genreId: 'EDUCATION',
  .....' ],
  .....
  url:
   'https://play.google.com/store/apps/details?id=free.programming.programming&hl=en&gl=us' } ]

2. search: It retrieves a list of apps that results in searching by the given term.

Parameters:

• term: It contains the search term.
• num: The number of apps that we wish to retrieve. It is an optional parameter with the default value of 20. Note that the maximum value of this parameter can be 250.
• lang: It is an optional parameter. It has a default value of “en” for English. If we want out app data to be fetched in a different language, we can provide the two-letter code of that language to this parameter.
• country: It is also an optional parameter. It has a default value of “us” for the United States of America. This parameter is helpful when the app we’re looking at is available only in specific countries.
• price: We can pass “all” to return both free and paid apps, “free” for free apps and “paid” for paid apps.

Example:

var gPlayScraper = require('google-play-scraper');

gPlayScraper.search({
    term: "tech",
    num: 2
  }).then(console.log, console.log);

Output:

[ { title: 'CNET: Best Tech News, Reviews, Videos & Deals',
    appId: 'com.cbsinteractive.cnet',
    url:
     'https://play.google.com/store/apps/details?id=com.cbsinteractive.cnet',
    icon:
     'https://lh3.googleusercontent.com/DeIoPrQ4jp2STHmWzbWI8Ss8JRnPgFrmDoOLje2PXcpA7CQN8hFxOvxXCSOOEGLUUQ',
    developer: 'CBS Interactive, Inc.',
    developerId: 'CBS+Interactive, +Inc.',
    priceText: 'FREE',
    currency: undefined,
    price: 0,
    free: true,
    summary:
     'Breaking technology news and expert product how-tos,
      comparisons, advice & tips',
    scoreText: '4.0',
    score: 4 },
  { title: 'Tech Coach',
    appId: 'com.asurion.solutohome.verizon',
    ..
    score: 4.4551244 } ]

3. suggest: It takes a string input and returns a list containing five suggestions to complete our search query.

Parameters:

• term: The term we want to get suggestions for.
• lang: It is an optional parameter. It has a default value of “en” for English. If we want out app data to be fetched in a different language, we can provide the two-letter code of that language to this parameter.
• country: It is also an optional parameter. It has a default value of “us” for the United States of America. This parameter is helpful when the app we’re looking at is available only in specific countries.

Example:

var gPlayScraper = require('google-play-scraper');

gPlayScraper.suggest({term: 'algorithms'}).then(console.log);

Output:

[ 'algorithms',
  'algorithms to live by',
  'algorithms and data structures',
  'algorithms explained and animated',
  "algorithms for rubik's cube" ]

4. permissions: It returns the list of permissions an app has access to.

Parameters:

• appId: The id associated with the app on Google Play. It can be seen in the “?id=” of the URL.
• lang: It is an optional parameter. It has a default value of “en” for English. If we want out app data to be fetched in a different language, we can provide the two-letter code of that language to this parameter.

Example:

var gPlayScraper = require('google-play-scraper');

 // Let's analyse the permissions requested by SHAREit
gPlayScraper.permissions({appId: 'com.lenovo.anyshare.gps'}).
     .then(console.log, console.log);

Output:

[ { permission: 'take pictures and videos', type: 'Camera' },
  { permission:
     "add or modify calendar events and send email to
      guests without owners' knowledge",
    type: 'Calendar' },
  { permission: 'record audio', type: 'Microphone' },
  { permission: 'read sensitive log data',
    type: 'Device & app history' },
  { permission: 'retrieve running apps',
    type: 'Device & app history' },
  ....
  { permission: 'send sticky broadcast', type: 'Other' },
  { permission: 'expand/collapse status bar', type: 'Other' },
  { permission: 'control vibration', type: 'Other' } ]

The other methods which you can take a look at are:

• list: It returns the list of applications from one of the collections at Google Play.
• developer: It returns the list of applications by the given developer name.
• reviews: It returns a page full of the reviews of the currently specified app.
• similar: It returns a list of apps that are similar to the apps that are specified.
• categories: It returns a list of categories available on the Google Play Store.