It's less about hat's in general, and more about this particular hat being a mark of social status...
Y-...You! Taking off your hat in front of someone like that...how shameless!! W-Wait a minute! Why aren't you wearing underpants!? Borrowed Y-You pervert!! Of course, duh! AH!! Bringing thee down will be simpler than breaking a plate!Underpants? Dost thou mean a loincloth?'Tis necessary?Highschool girlHeian (?) YoukaiThis is so annoying...I will show you the power of the Occult.Very well, little girl!Ow...w, not so bad high school girl.