Developers

Flat
File limits?!?
User: mtiller
Date: 7/9/2014 9:01 pm
Views: 5237
Rating: 0

I'm using the API to upload a bunch of files and then build cards out of them.  My script works...to a point.  Eventually, it terminates with "Insufficient privileges for User." as the error message.

I get a similar error if I try to upload all my images by "drag and drop" into the Web UI.  I don't get what is going on here.  Is it a throttling issue, a file limit, or what?

Thanks.

Re: File limits?!?
User: mtiller
Date: 7/9/2014 9:36 pm
Views: 4
Rating: 0

OK, after spending a couple hours on this, I think I narrowed it down.  It seems to be a concurrency issue.  I'm invoking the API through a Node.js script using the Q promises library.  The essential point here is that I'm treating all my API interactions as completely asynchronous (not waiting for one to complete before making the next request).

When I refactored my code to make it less asynchrnous (creating exactly one at a time instead of all of them in parallel), the problem seems to have gone away.

I originally thought this was a limitation on the number of files because it seemed to consistently fail with a big batch and (possibily coincidentally) after uploading the same number of files.  But when I refactored the code to force it to wait until the previous card completed, it got past that apparent file limit without issue.  Perhaps it is also a limit on the number of files you can have that are not "related" to something.  I don't know.

The bottom line is...if you are writing a client that is naturally asynchronous, it appears you have to be very careful about about the number of concurrent requests you make.

Ideally, the API should manage the throttling on its end (by not responding to the nth+1 request until one of the current n is completed).  Barring that, it would be ideal if the API returned an error code that indicated that the issue was related to concurrency.

Re: File limits?!?
User: dfp (designed for play)
Date: 7/9/2014 10:01 pm
Views: 115
Rating: 0

The issue is not directly related to concurrency.  There is a rate limit on the number of requests you can make in some time frame; I don't remember what the time frame is, but I think it's 60 seconds.  It doesn't matter if you put the requests in one at a time or all at once - if you exceed the rate limit, your api requests start failing with error code 450 or 452.

Re: File limits?!?
User: mtiller
Date: 7/10/2014 6:38 am
Views: 2599
Rating: 0

Thanks for the clarification.  That could certainly explain it.

It appears that the web UI is subject to the same restriction but it also gives the same confusing error message.  So it would be really nice if the API was updated to provide a message that more directly states the issue (and ideally headers like GitHub provides).

It would also be better to return valid HTTP status codes.  GitHub returns a 403 in such cases but 450 and 452 aren't real status codes so it is hard for clients to handle them properly.

I've reworked my code to function sequentially.  For now, that seems to be keeping me below the rate limit (although I haven't tested it extensively).  It appears that what was happening was it was starting a whole bunch of parallel onces which exceeded the limit (perhaps because they were in parallel or perhaps because they executed quickly, not sure).

Thanks again.

PreviousNext