How to scrap iOS app store reviews?
If you’re looking to scrap iOS reviews, there are tons of options: python or node libraries, software solutions, APIs.
This post is aimed at giving you a solid understanding of what each option entails, and how they work at their core, so you can decide on the best option for your case, instead of scrolling endlessly on github repos and medium posts.
When we first set up reviewflowz we were in a very similar situation, and we did the digging. So I thought I’d save you the trouble.
Also, if I’m honest, as much as I love people open-sourcing scraper repos, they’re usually way too complex, with tons of layers of abstraction for no reason whatsoever. It makes them hard to maintain, and even harder to understand. What’s at the core of those libraries is a scraping strategy, and I feel like this is what they should show. The challenge isn’t in writing a couple of fetch calls & a proxy rotation strategy.
A little context on how iOS app reviews actually work
If you’re looking into reviews on the app store, you might, or might not know that there are two different objects at play:
- Reviews
- Ratings
On a platform like Google Maps, there’s a single object called review. And it has, or doesn’t have content.
On iOS it’s a little different. A review without content becomes a rating. And a review object has content and a rating associated to it.
How the App Store deals with countries and languages
You might also know that an app is listed on a select list of countries on the App Store. Some apps will be available in every possible country, while others will only be available say in Korea, or in the US, or in France.
There are a few gotchas around geos on the App Store, the main one being that reviews are associated to a country, but not to a language.
I haven’t found a “global” endpoint, and I suspect there simply isn’t one. This means you need to fetch reviews from each country in order to get all reviews. You can also decide to fetch only a few countries, but you might be missing out on some reviews. There are 116 countries on the App Store. Here’s a list. You can download it as a CSV here.
The undocumented APIs to access iOS app reviews
To scrap App Store reviews, you can leverage this URL
https://itunes.apple.com/{countryCode}/rss/customerreviews/page={page}/sortBy={sort}/id={app_id}/json"It’ll return up to 50 reviews each time, in JSON format. Fairly straightforward really.
There are 4 params to customize your query
- countryCode: See the country list above for a complete list.
- page: You’ll need to implement pagination logic if you want to get all available reviews. Each page returns 50 reviews
- sortBy: The sort order you want. It can probably take a few values but
mostRecentworks, so why use anything else? - app_id: You can get the app ID from the app page on your browser. For example, Shopify’s app ID is 324684580
That’s it.
You can try it yourself
curl https://itunes.apple.com/us/rss/customerreviews/page=1/sortBy=mostRecent/id=324684580/jsonYou’ll get a pretty heavy JSON output with 50 reviews, pagination information under the “link” object, and a few other data points.
Dump the output into your favourite JSON parser to write up the wrapper, and you’re done.
Gotchas to watch out for
If you start hitting the endpoint with no throttling whatsoever, it’ll start to slow down in its responses, which eventually leads to timeouts and crashes the whole thing. You’re left figuring out where it went wrong to restart at the right place. No fun.
With a bit of throttling, you’ll manage to keep fast responses, and it’s ultimately a lot faster. A classic case of put yourself in whoever you’re scraping’s shoes.
Use an IP too much (including clean residential IPs if you’re doing this locally), and you’ll get 403s for a few minutes or hours. You’ll need some sort of proxy rotation logic or service if you want to handle large volumes of reviews. Server proxies work just fine, it’s really just a matter of Apple protecting itself from aggressive scraping.
Other options to scrap iOS App Store reviews
#1 Reviewflowz
I’m a little biased on this one obviously, but if you have $29 to spend and no time to waste, this is really your best bet.
Set up an account, buy a Lite subscription to unlock unlimited CSV exports, and access any App’s reviews. No limit.
I’m all for the fun of writing up a scraper & wrapper, but sometimes it just doesn’t make sense.
As an added bonus, you can set Zapier, Slack, Email, or Microsoft Teams notifications for every new review, or for some reviews, using filters on things like the review score, content, etc.
If you need API access, you’ll need to subscribe to a Premium plan to access both the polling API and our webhooks feature, which comes with built-in deduplication.
You can find out more about our pricing here, or signup for a free trial here.
#2 Use a Python library
There’s a pretty famous Python library that basically leverages the endpoint I mentioned. It essentially takes the same parameters as the endpoint itself and wraps the review objects into different review objects.
There’s a very similar library in Node, which actually comes with a few other endpoints that are interesting. It’s called app store scraper and it’s available here on github.
Another gotcha: the histogram of ratings is inaccurate at best. If you play around with it you’ll notice that the ratings total is the same in every country, while the distribution is different. What the app store does is basically apply a country-specific distribution to the overall number of ratings for an app to build this histogram.
#3 Paid APIs and scraping services
There are also a bunch of paid APIs and scraping services that offer a wrapper for App Store reviews. If I’m honest they don’t handle all that much apart from a wrapper, which… I mean that JSON is already fairly clean.
As much as I’m biased, if you’re open to spending money on this, might as well go all the way to a clean CSV export with us. And if you’re not, you should have all you need in the first part of this post!
If you found this post is helpful, please consider linking to it or sharing it to help us spread the word