Upcoming Changes for Upload and Approval Complaints

I wanted to detail my plan addressing complaints about uploads and approvals. Previous thread on the issue.

Phase One

• I'm removing the Janitor user level. I will be promoting most of the existing ones to Moderator depending on feedback and how friendly/helpful they have been on the forums and comments.
• In place of Janitor there will be a new user role determining whether or not you can approve posts. This means just because you are a mod or an admin doesn't mean you can approve posts. This is part of a larger move to make rights more granular. Eventually I will move uploads, tagging, forum and comment moderation into separate roles.
• Existing janitors, mods, admins will all start out with this role.
• This will happen within the next week.

Phase Two

• I will start aggressively recruiting new approvers. Every month 3-5 users with good uploads or favorites will be selected and gain the approval flag. After a month their approvals will be reviewed and they'll either keep or lose their approval right.
• This will happen towards the end of July.

Phase Three

• I will start removing the approval flag from older janitors. Specifically I will be targeting low quartile scores in the Janitor Trial Report. If this ends up affecting you please don't take it personally. Consider the next few weeks a grace period to start becoming more selective of your approvals.
• This will happen towards the end of August.

If you have questions or concerns or ideas not listed here let me know. I'll update this original post as plans change.

albert said:

• In place of Janitor there will be a new user role determining whether or not you can approve posts. This means just because you are a mod or an admin doesn't mean you can approve posts. This is part of a larger move to make rights more granular. Eventually I will move uploads, tagging, forum and comment moderation into separate roles.

Does this mean approval of one post will be done by multiple users?

• I will start removing the approval flag from older janitors. Specifically I will be targeting low quartile scores in the Janitor Trial Report. If this ends up affecting you please don't take it personally. Consider the next few weeks a grace period to start becoming more selective of your approvals.

The report says "most statistics are not significant unless there are at least 300 approvals", so even if one approver has high quartile score, will it be rendered moot by it? Will you target said approver as well?

dean_exia said:

Does this mean approval of one post will be done by multiple users?

No. Only one approver is needed to approve a post. What it means is approver status is separate from being a mod. Someone could be a basic member but still be an approver. Or someone could be a mod but not have the ability to approve posts.

dean_exia said:

Does this mean approval of one post will be done by multiple users?

I don't think that's what it means. It's just a new role that can determine whether anyone, even admins like albert, can approve or not.

I'm curious what the cut-off points are for the use of the janitor report in doing the evaluations, including what the minimum quartile, median and +3 scores will be, and if there will be flexibility or a hard set rule.

My quartile score seems to be pretty horrendous, I guess because I've been keeping an eye out for obscure images outside the Touhou/KanColle/Current-Top-Animes ring (though many of my approvals still fall inside it) and mainly using the modqueue which favors old posts over sniping new potential score goldmines. I guess I'll need to switch up a bit.

We have a couple months to work out the kinks in the trial report, don't worry about it for now. I just sent albert a dmail with my concerns with the current state of it but we may be worrying about nothing.

Seems like a good thing. It's more of a pain in the ass for the guy coding it, but when coding this stuff for a living being able to assign very specific permissions is always more useful than roles in the long run. The roles are always insufficient. Different people are good at different things and just because a guy is good at approvals doesn't mean he's good at modding other stuff.

sarusa said:

Seems like a good thing. It's more of a pain in the ass for the guy coding it, but when coding this stuff for a living being able to assign very specific permissions is always more useful than roles in the long run. The roles are always insufficient. Different people are good at different things and just because a guy is good at approvals doesn't mean he's good at modding other stuff.

I agree.

I take it when phase 3 hits, going by the trial report, I'll be one of those losing approval privileges? I guess I'm not really surprised, but I do have a concern with these current values. The values currently heavily punish approvers of comics, which may or may not have been intended. Currently from crunching the numbers of the top 4 approvers (excluding not_one_of_us, so Ephyon, zigzag, z905844, and myself), out of the four I'm the only one who has approved comics in a sizable quantity (~30% of my approvals) and the numbers show that is heavily tanking the scoring of my approvals. When looked at as a whole ~40% of my approvals (in the last 3 months) have a score of 2 or lower, but when comics are removed then my percentage becomes ~29%. In comparison Ephyon, zigzag, and z905844 only have ~1.6%, ~6.4%, and ~7%, of their approvals composed of comics respectively. Their percentages of non-comic approvals with scores of 2 and below are ~30%, ~13%, and ~23%.

Comics inherently have lower scores than non-comic posts, and so approving them in any large quantity greatly damages an approver's scoring. Of the status:active images for the last 3 months we have 74,827 non-comic and 7,323 comic posts. ~26% of Non-comic posts have scores of 2 and below, while ~74% of comic posts have scores of 2 and below.

Yeah comics were the one big thing I addressed in my message to albert, which he intents to take into account.

I do feel somewhat concerned over using picture scores as the measuring stick for an approver's worth. Isn't Score a terrible metric for determining the value of an approver? It's exclusive to paid users and a majority of the pictures with the highest scores lean toward those that are erotic in nature. Not that favcount is a very good metric, either. I don't have any particularly good suggestions of what could be used instead of score, but perhaps changing the scoring system itself to be something along the lines of yande.re's star score system?

I agree. Comics are probably the greatest factor affecting an approver's score, but they're not the only one. So are ratings, copyright popularity, etc. Pretty much any preference for a specific type of post can positively or negatively affect an approver's score.

Some approvers may go out of their way to approve obscure or borderline content when its quality warrants it, but their score will not reflect that. Manually reviewing an approver's performance will still be important, regardless of how well the system will calculate these scores when the kinks are ironed out.

Seems this is major problem. kinda agree with reward and punishment system these days.

Fred1515 said:

I agree. Comics are probably the greatest factor affecting an approver's score, but they're not the only one. So are ratings, copyright popularity, etc. Pretty much any preference for a specific type of post can positively or negatively affect an approver's score.

Some approvers may go out of their way to approve obscure or borderline content when its quality warrants it, but their score will not reflect that. Manually reviewing an approver's performance will still be important, regardless of how well the system will calculate these scores when the kinks are ironed out.

Pretty much as you said. Even time of day can affect whether or not a picture receives a decent score. Copyright and artist popularity really does give certain images an edge over more obscure content. Like, how else to explain how post #2061003 get 14 points in 5 hours while post #1225155 doesn't even get 3 points in 3 years? Simplicity of concept and Shameimaru being cute? Sure, but it's still probably just because it's Touhou.

To address the problem with obscure and comic posts vs very popular posts I'd suggest to lower the janitor score when the posts' copyright count (original omitted) is very high and increase it when the copyright count is very low or the post is a comic. The exact numbers depend on just how high/low the copyright count is / how low the comic average score in relation to other posts is.

It sure would complicate the computation, but without such or similar measures many unpopular posts probably won't get approved anymore.

Artist or character popularity could be considered as well, but I think that might be overkill.

Won't this be an incentive to just approve all the green posts in the queue, regardless of whether or not we personally like it? For example, I don't personally like most explicit stuff, but it does tend to get higher scores and approving lots of it would undoubtedly boost my ranking. It defeats the purpose of selecting janitors for our eyeballs and selective taste if we're just chasing numbers.

Not to mention post score bombing would affect people even worse.

albert said:

• I will start removing the approval flag from older janitors. Specifically I will be targeting low quartile scores in the Janitor Trial Report.

The reason stated for starting to add new janitors is to reduce the issue of "good but worksafe or original or obscure copyright art gets ignored", which is why people favoriting/uploading posts that don't have very high scores are targeted by the automatic invite sender.

If instead janitors approving posts that don't have very high scores are now demoted it would have the opposite of the intended effect, leaving mostly janitors that approve touhou/nsfw/other popular things.

Maybe demoting janitors that approve posts with actually negative score (not just low scores) or janitors that approve posts that later on get flagged and deleted might be a better metric. But there's probably no perfect automatic metric for this and it would be better to handle it on a case by case basis.