batch operations? retry when Flickr throttles?

Topics: Developer Forum, User Forum
Apr 21, 2010 at 2:55 PM

I've having problems migrating ~8700 pics (using Migratr) with the source being Flickr.

http://forums.callingshotgun.net/viewtopic.php?f=2&t=13833

Watching the calls made to api.flickr.com in Fiddler, it appears that there's an initial fetch of the photo collection (18 pages, 500 per page) that works fine, but then a couple thousand API calls later, Flickr stops responding (no response at all, just no response), presumably because of throttling on their end.  While it would be very nice for FlickrNet to handle that internally (sleep and retry until it works for such a case, opt-in if necessary via app.config), I was wondering if there's a way (existing in FlickrNet or to be added) to batch the operations?

For each of the photos (all ~8700 of them), there are calls to PhotosGetInfo and PhotosGetAllContexts, for instance - so that's 17k+ API calls needed and the throttling (assuming that's it) kicks in around 2k.  Not sure if it's based on API key (Migratr does support the user putting in their own keys) or IP, but if FlickrNet could either allow batching so those 17k calls could drop down significantly (say, under 1k total API calls, perhaps 100 total as a target), then that'd be great.

I'm not familiar with Flickr's REST API, so this the batching may be limited by what they offer, but hopefully the throttling 'detection' and opt-in retry could be done in FlickrNet. After all, I'd much rather this operation eventually succeed (even if it has to sleep now and again) than fail :)

Thanks!

Coordinator
Apr 21, 2010 at 4:14 PM

The Flickr API has a unwritten rule of 1 query per second, per API key - it is not a hard and fast cut off point at a certain number of calls.

I've no idea if Flickr does indeed throttle calls, or start returning blank responses, and without a definitive response from Flickr it would be like trying to herd cats. What happens you use the same API key from two different computers, or even two different threads? What if there is some other error that causes blank responses to be returned (an error at Flickr's end perhaps, or a badly configured proxy server)?

This isn't something I'd like to add to the library. You should be implementing your own throttling of calls to Flickr yourself.