Manage the amount requests to the Web API to avoid rate limiting

[JMPerez]

I think you could use something like https://github.com/JMPerez/promise-throttle to limit how many requests per second are made. You have an example of how this is done on https://github.com/JMPerez/spotify-dedup/blob/master/app/scripts/main.js

If you like the idea I can work on a Pull Request to add this.

[watsonbox]

Yes, I like this idea. I guess it would only ensure no throttling errors when using a personal application though, since according to the docs "Rate limiting is applied on an application basis (based on client id), regardless of how many users are using it."

I'm going to add a progress bar for the bulk export soon, too, which would be a pre-requisite for adding this kind of slowed down exporting I think.

[JMPerez]

You can still have rate limiting issues if several users are using the site at the same time, but you would have more control on the amount of requests a given user makes per unit of time.

[pavelkomarov]

My version implements rate limiting, 10 queries per second to the server. This is essential to make exporting really large playlists or all together work. Even for users with insanely large collections, it works in only a few minutes. Progress bar not super necessary.

I've hard-coded my client id, so everyone who uses it racks up traffic under the same limit. So if lots of users at once happen to all try to Export All at the same time, then yes they'll still run in to rate limiting problems. But this is unlikely until there are thousands of users every day, so I've chosen to just ignore it. If we ever get to that point, then I'll have to do something like give everyone their own client id.

exportify.net

[watsonbox]

Terribly overdue 🙈 but I'll see what I can do in updating this repo to support this and using as much of the work that's been done as possible. Thank you @pavel-aicradle for advancing it. I think this issue will best serve as the main one for dealing with this, put there's a lot of useful discussion in these:

• Overcoming "API rate limit exceeded" #6 (includes advice on creating your own client app)
• Exportify 'Export All' not working #56

[pavelkomarov]

#70 gives you my changes, which include the fix to this.