Danbooru

What demand would there be for a sort of "uploading" handybook?

Posted under General

As the subject line says.

Over the past few weeks I've compiled a handful of both general and specific tips/tricks/hacks concerning my uploading habits; they vary from maintaining feeds, detecting revisions quickly and efficiently, getting push notifications from sites that typically don't allow you to get them, seeing deleted posts from sites that typically don't allow you to, viewing older versions of posts that are revised, viewing hotlinked images, fetching images on sites that disable Javascript or have deceptive sample sizes... Things like that. I've discussed it in private with a multitude of other users (namely Randeel), and I think doing this much would lessen the barrier of entry to uploading and increase the general quality of the gallery as some users find it easier to upload good content without expending a lot of wasted effort.

I want to note that it will contain some exploits, however, but all of which are permissible. There's nothing "underground" about these or the like; all of these tips I'm compiling together are either usable as is or use scripts/tools/extensions/applications/etc that are all public and easily accessible.

Now I could "publish" it as it stands, but before anything happens, I'd like to probe the community here over how appropriate such a compilation/guide/handbook would be, and if it should be posted here as a forum post (and in which case could go into the useful threads sticky), relevant information in the site howto's, or even outside the site somewhere such as on a blogpost or public github repo (as a wiki). There could be some downfalls to making some of this public, after all. And although competition for posts may increase, I also hope that it'll lead to a better looking gallery and more contributors that will put the effort in to cover the artists and content that some of us regular uploaders otherwise miss.

Updated

I'll just go ahead and throw out my own techniques right now then...

The following include those techniques besides just simple browsing plus use of IQDB/SauceNao... (which I still do to a degree)

Note: For all of the below, the image size just below the largest is downloaded to avoid filling up my hard drive.

Upload by tag

Show

I've developed a custom Python script that downloads all images from Pixiv using their Japanse tags.

  • 1. Images must have a minimum favcount of at least 50
  • 2. Danbooru IQDB matching is done to ensure they're not already uploaded
  • 3. Downloaded images are sorted in different folders by favcount rounded down to the nearest 50
  • 4. Further sorting of images is done manually with PhotoSift starting from the highest favcount to the lowest
    • A. Good enough images are uploaded immediately
    • B. Images that deserve a 2nd, 3rd, or even a 4th look are sorted by priority into a "Look at later" folder
    • C. After a few iterations of that folder, those images get moved to an archive folder for potential future looks
  • 5. Downloaded images are deleted after they get uploaded

Note: For the above, images that have been downloaded get recorded so that they don't get downloaded again.

Since favorite counts on Pixiv change over time, future passes over the same group of posts will pick up new posts that didn't make the cut previously.

Upload by artist

Show

I've developed a custom Python script that will download from all of an artist's image sources with the following currently supported: 'deviantart', 'twitter', 'twitpic', 'danbooru', 'nicoseiga', 'pixiv', 'pawoo', 'tumblr', 'nijie', 'tinami', 'website'.

A separate batch file will run that script with all of an artist's different image sources.

1. Download images adding post ID information to the image filename
2. Record the last image downloaded so that the same set of posts don't get processed in the future
3. Use VisiPics to remove/move similar images from each source separately
4. Use VisiPics comparing each source with the Danbooru source and removing similar images
5. Use VisiPics on all sources at once (minus Danbooru) to find matches across sources
6. For images to be uploaded, do a final check on all matching image sources to find the best quality image

In addition to the above, I keep a weekly log on artists that I've downloaded that week, along with which artists had:

  • no images to upload
  • had some with good quality but the material didn't interest me so I didn't upload
  • had uploads
  • had more uploads than I felt comfortable with uploading all in one go

The above helps me determine how often I should go back and check a particular artist. As of right now, I'm tracking a little over 80 artists.

I'll be publishing it later today in a separate thread.

I intend to also include edit suggestions by other users, which could be made by either bumping the thread or PM'ing me. It would also help if users had any questions regarding the guide, in which case clarity could be added.

In the meantime, this thread could serve as another discussion point about whether any of that information belongs elsewhere, or what else would be helpful.

EDIT: Pushing it back to tomorrow, since there are some things on my end.

Updated

It's been like that for a while now; I've kept it up to date in forum #132211.

Speaking of this topic, I've decided to not make it public in the end if only because of fears that some of the tips I'm about to share would be abused by uploaders I currently am not fond of. This includes users that don't make it a habit to thoroughly tag their uploads or upload content that isn't regularly approved in the queue. Putting some of the knowledge I have to users that might try to regularly exploit the uploading process by such things as mintagging, when the system is already easily gained, is something I found in hindsight that would be rather annoying and demotivating.

So, as it stands, I made a private, invitational discord outside of the site that @Randeel is helping me moderate where I'll be posting it instead. If you're interested, you can pop me a Dmail, but I'm being fairly selective about the whole process. I'm also planning to share the link for users that don't have any intent on joining the Discord, but do keep in mind that it won't be up to date with the other collective tips I share with other users.

If you're on my list of users that I recognize and like, then you'll be getting an invitational Dmail from me soon.

Updated

1