Bloomers actually are more practical for riding brooms, especially if the broomstick's not polished.
Even if it's polished it's better as you still get more padding and that extra padding can pay off as it'll help prevent getting a bit sore from riding it so much.