I think that Patchouli said it, and that the squiggly tail on the speech bubble indicates a shaky voice, since Patchouli is crying.
Pre-translation, I think that Remilia is the one who asked about how the library cleaning was coming along, to which Patchy and Koa turn the other way. I like Koa's face in the last panel, it gives off a "Crap, they're on to me" vibe.
EDIT: Oh, seems I was wrong. Looks like Patchy is starting to suspect Koa, hence the head turning...
I think Patchouli is simply turning to Koakuma for some comfort. Koakuma may be looking away because she doesn't want to see her mistress's saddened face. She wanted to troll Patchouli, but not go this far.
As for Sakuya, I think she suspects Koakuma for this, but is lying to her mistress as she may believe Koakuma will get what she deserves later.
The reaction flash from Koa in panel 3 suggests Sakuya may have been staring straight at Koa, which is consistent with MezzoDragon's speculation that Sakuya suspects Koa - or, closely related speculah, Sakuya just looks casually at both of them and the reaction flash indicates Koa's guilty conscience.
If that's the case, then in panel 4, Sakuya is covering for Koa in front of the boss (possibly intending to sort Koa out privately), Patchouli is turning to look at Koa because she wonders why Sakuya looked right at Koa, and Koa is trying to avoid looking Patchy in the eye.
If any of that is on target, I guess we'll find out next strip...
Seeing it again, Patchouli would have to be pretty dense NOT to suspect Koa. Ever since they met, Koa mercilessly trolled her. Then Patchouli reveals to Koa that she has a precious book, then bam: the book goes missing, on a day when Koa was in the library when Patchouli was not. The idea that Koa took it should, at the very least, linger in the back of Patchy's mind.
Oh, perfect timing!Pleased to meet you, Lady Patchouli.Patchy lost her precious book. Do you know anything about this, Sakuya?Is something the matter?Remi, who's this...?I will now work here as the chief maid. My name is Izayoi Sakuya.Book 40: The Chief MaidFrom the artist's Pixiv pageI'm sorry, but I'm afraid I don't...I see... Too bad.