IIRC it was explained that when the swords are shot from Gate of Babylon or hand-picked and used by him the moment it dematerializes it gets transferred back to the treasury within the gate.
He also has the option to intentionally leave a weapon he uses outside the treasury if he doesn't want it back, like he did with the weapons he fired at Caster's summoned monster.