Read the rules before proceeding!

Topic: Change log - 1/13

Posted under General


This may end up being a controversial change but I had to do something to fight the explosion in database load lately:

API calls (that is, JSON/XML requests) to post listings are now throttled on a per IP address basis, limited to 1 call per second. If you try to make calls more often than this you'll get a blank 503 page.

Updated by ekkk

  • ID: 72300
  • Permalink
  • RaisingK

    ...Ow, that's going to hurt my scripts...


  • ID: 72311
  • Permalink
  • Coconut

    Oh wow. Why do I have a feeling that this is my fault...

  • ID: 72314
  • Permalink
  • albert

    The saga continues...

    Someone keeps attempting searches like source:http://img35.pixiv.net/img/shikishima_e/21274264* which unfortunately is fairly slow. It's spread across multiple IP addresses which makes me think this isn't some isolated service just leeching the site.

    For this reason, post XML/JSON feeds are now restricted to members only. An easy way is to just pass in login and password_hash params along with your GET request.

    Provided you are authenticated, I've removed the request throttle, although please still limit your request rate.

  • ID: 72440
  • Permalink
  • Coconut

    It was a feature implemented at my request, to link Pixiv thumbnails to Danbooru posts. I kind of had a feeling it'd be pretty database-intensive...

  • ID: 72449
  • Permalink

    I'm sorry to ask, but to those of us who only use the site for commenting and browsing purposes, and who are oblivious to all the terminology of the site scripts and what-not. What does this mean for us?

  • ID: 72455
  • Permalink
  • RaisingK

    Very little has changed period, and nothing has changed for you; don't worry about it.

  • ID: 72456
  • Permalink
  • Graysnake2834

    thanks albert for having restored the date search
    among other things, now the site is significantly faster than yesterday


  • ID: 72473
  • Permalink
  • linoth

    Minor scare as well. I get the idea to work on something Friday, poke the API a half dozen times for output, and see this on Saturday.

    I'll keep load in the front of my mind.

  • ID: 72520
  • Permalink
  • desides

    This is fantastic. For months now I haven't been able to perform any sort of in-depth tag searches (say, searching my favorites for more than one tag), and now I can. No more timeouts, no more failures.

    As far as I'm concerned this is a hugely positive change for actual site usage.

  • ID: 72572
  • Permalink
  • blaze15

    so will this effect me / who rairly uploads and mostly downloads from this site?

  • ID: 72630
  • Permalink
  • glasnost

    No, unless you are using some API-accessing script to do so (if you don't know whether you are or not, you aren't).

  • ID: 72631
  • Permalink
  • KeliraTelian

    Basically, if you didn't understand a word Albert said, this will not affect you in the slightest.

  • ID: 72632
  • Permalink
  • Ryunozora

    I noticed that I can call APIs with an incorrect password hash. Then I experimented a bit and found out that feeding a username that doesn't exist doesn't affect API calls either. It this intentional or a bug?

  • ID: 72708
  • Permalink
  • RaisingK

    This is indeed the case for the post API...

  • ID: 72719
  • Permalink
  • ekkk

    I do have some self written python mirror scripts running, too. I will gladly rewrite them.

    Can you please be more specific as to "limit your request rate". Which queries do hurt the DB? Is bandwith an issue? (Eg. running many curl processes)

    Edit: Maybe i should clarify. I am mirror/backing up my profile _only_. And _only_ actual changes.


  • ID: 72724
  • Permalink
  • 1