Danbooru

Read the rules before proceeding!

Topic: [Userscript/Prototype] IndexedAutocomplete

Posted under General

BrokenEagle98

IndexedAutocomplete switches the Danbooru storage mechanism for saving autocomplete entries, using IndexedDB instead of Local Storage.

Installation

Project page
Main script

https://raw.githubusercontent.com/BrokenEagle/JavaScripts/stable/indexedautocomplete.user.js

Usage notes

It's used exactly the same as autocomplete. The only difference is under the covers, in how the data is stored and retrieved. Debug information is being left on in the console during this development period.

Note: It only enables on browsers that support IndexedDB, which for sure includes both Chrome and Firefox.

Final

After learning about IndexedDB from topic #14474, I've been thinking about transitioning Danbooru from using Local Storage to IndexedDB where possible on Browsers that support it. The reason is that Local Storage is currently hamstringed to a low maximum size tailored to the least capable browsers. This ends up being ~4000 entries, which from my own testing is usually reached in a few days. This necessitates frequent pruning actions to keep the Local Storage cache from overflowing. Each time this happens, the client has to query the server again which is slow and results in higher latency on the autocomplete.

IndexedDB on the other hand has no size limitations AFAIK. This means that there would be no pruning actions required, and therefore network calls would only be needed when an autocomplete entry expires, which is currently set at one week. Additionally, with the extra storage size available, other items could be cached locally as well.

The end goal would be to have a stable proof-of-concept that could eventually be proposed as an issue on GitHub.

Any suggestions or feedback is appreciated.

Latest edits

  • (2018-01-20)
    • Version 13 - Add artist finder to session storage

Versions

Updated by BrokenEagle98

  • ID: 139458
  • Permalink
  • BrokenEagle98

    Pushed Version 3, which stores the pool autocomplete results, i.e. the autocomplete when you start a tag with pool:. This marks a greater departure from the original autocomplete, as with that, only the regular tags were being stored.

  • ID: 139533
  • Permalink
  • BrokenEagle98

    Pushed Version 4, which does the user autocomplete results (user:). This also does the autocomplete for @ mentions.

    Edit:

    Pushed a minor update to this version, which fixes the program loader.

    Updated by BrokenEagle98

  • ID: 139570
  • Permalink
  • BrokenEagle98

    Pushed Version 5, which does the favorite group (favgroup:) and saved search (search:) results.

    Edit:

    Pushed a minor update, which fixes a bug with the user source.

    Updated by BrokenEagle98

  • ID: 139599
  • Permalink
  • BrokenEagle98

    Pushed Version 6, which stores the related tag results. This includes the buttons Related tags, General, Artist, Copyright, Character.

  • ID: 139767
  • Permalink
  • BrokenEagle98

    Pushed Version 7 which varies the expiration time by type and count. Below are the approximate values/ranges.

    • tag (count):
      • < 100: 7 days
      • 100 - 1000: 7 - 14 days
      • 1000 - 10000: 14 - 21 days
      • 10000 - 100000: 21 - 28 days
      • > 100000: 28 days
    • pool (count):
      • < 10: 7 days
      • 10 - 100: 7 - 14 days
      • 100 - 1000: 14 - 21 days
      • 1000 - 10000: 21 - 28 days
      • > 10000: 28 days
    • user: 28 days
    • favgroup: 7 days
    • search: 7 days
    • relatedtag: 7 days

    User was left high as that's not that often to change, whereas favgroup/search could be changed often so were made low, and relatedtag was left low since no post count is returned with the results.

  • ID: 139877
  • Permalink
  • BrokenEagle98

    Pushed Version 8 which stores the wiki page results. Additionally, it fixes an issue where the autocomplete doesn't function on wiki page version pages.

    The expiration schema is the same as the tag category if a tag exists, otherwise it uses an expiration time of 7 days.

    Edit:

    Pushed a minor update, fixing a bug with handling blank wiki entries.

    Updated by BrokenEagle98

  • ID: 140609
  • Permalink
  • BrokenEagle98

    Pushed Version 9 which stores the artist results.

    It uses the same expiration schema as wiki page results.

  • ID: 140699
  • Permalink
  • BrokenEagle98

    BrokenEagle98 said:

    Pushed Version 9 which stores the artist results.

    It uses the same expiration schema as wiki page results.

    Edit:

    Pushed a minor update, fixing how pools save data.

  • ID: 140756
  • Permalink
  • BrokenEagle98

    Pushed Version 10 which stores non-tag pool results. These are the results that come from the pools page and also from the Add to pool dialog box. It uses the same data as used by the pool: tags, so something stored by either can be used by either.

  • ID: 140833
  • Permalink
  • tapnek

    So the reason why someone would use this script is for a faster autocomplete? I don't really get the real hook for using this besides the script utilizing a new technology.

  • ID: 140834
  • Permalink
  • BrokenEagle98

    tapnek said:

    So the reason why someone would use this script is for a faster autocomplete? I don't really get the real hook for using this besides the script utilizing a new technology.

    Not an easy answer without getting too technical, but I'll give it a try.

    Current autocomplete results are stored on your computer using what is called local storage. The reason it does this is that otherwise it would have to make a network call every time which is significantly slower (~250-500ms). Additionally, you might have noticed that this becomes completely non-functional when Danbooru is very slooooow.

    Local storage has the advantage of speed, but is very limited in size. To support all browsers, Danbooru limits itself to just below 5MB. This might sound like a lot, but works out to be around only 4000 autocomplete results. This, again, might sound like a lot but it's not. Every letter and combination of letters that you type in the tag box or search box counts as an entry. For myself, it seemed like I was reaching the 5MB size limit once every 3 days.

    Currently when Danbooru reaches the limit, it purges all results, going back to an empty state. So all the advantage you have gained in speed by storing those results locally are lost.

    IndexedDB on the other hand is slightly slower than local storage (1ms vs 15 ms average), however it has the benefit of being near limitless in size, at least from what I've read and what I've tested (up to 35 MB). This means that results are no longer lost every few days, but instead they can last almost indefinitely, although to prevent stale tag information they currently expire from 7 days to 28 days depending on the post count.

    Overall, this means that on average the autocomplete should perform faster than the current setup. Additionally, to reduce the speed penalty for IndexedDB, the results are cached in session storage which is just as fast as local storage.

    The above covers just the usage of stored autocomplete for the tag box and search box, and does not cover any other usage of autocomplete on this site. Those results are currently not stored, which requires going to the network every time which is very slow. With this script, the results are stored so that the lookup penalty is only encountered on the first iteration. This pretty much guarantees that the results will on average always be faster than without. Besides autocomplete, it currently also stores the related tag lookup, which otherwise would require a network call every single time.

    TL;DR yes it is faster on average.

  • ID: 140835
  • Permalink
  • tapnek

    When will the script support other usages of autocomplete?

  • ID: 140837
  • Permalink
  • BrokenEagle98

    tapnek said:

    When will the script support other usages of autocomplete?

    Well, I've pretty much exhausted all of this site's current usage of autcomplete, so I've started to look where else it could be used. Right now I've been testing the addition of it to the Dmail TO and FROM username text fields.

    If you have any other ideas, feel free to share and I'll see what I can do.

  • ID: 140845
  • Permalink
  • BrokenEagle98

    Pushed some minor fixes to account for the latest deployment of Danbooru (forum #141030).

    • Minor versions
    • (2017-12-27)
      • 10.1 Fixed reliance on missing short category names meta
      • 10.2 Fixed reliance on missing related tags button meta
    • (2017-12-28)
      • 10.3 Fixed for wiki page versions autocomplete inclusion

    Updated by BrokenEagle98

  • ID: 141032
  • Permalink
  • BrokenEagle98

    Pushed Version 11 which adds in user autocomplete to all available username inputs. This adds autocomplete in over a dozen different pages that mainline Danbooru currently doesn't support.

    • Minor versions
    • (2018-01-01)
      • 11.1 Fixed missing autocomplete
        • On post version User searchbox
        • On artist version Name searchbox
    • (2018-01-02)
      • 11.2 Fixed missing autocomplete on post index saved search dialog
      • 11.3 Fixed IndexedDB save of data
    • (2018-01-02)
      • 11.4 Fixed tag antecedent name

    Updated by BrokenEagle98

  • ID: 141123
  • Permalink
  • BrokenEagle98

    Pushed Version 12 which adds in a full client-side data validation in addition to some major code refactoring.

  • ID: 141965
  • Permalink
  • BrokenEagle98

    Pushed Version 13 which adds the artist finder data to session storage. It's not being stored in persistent storage since URLs are so unique that it's not likely to generate very many hits. The primary use case is when you edit a post multiple times, or when you expand/contract the Related Tags section in the detachable edit box.

  • ID: 142263
  • Permalink
  • <<
  • 1
  • >>