The origin of the word is a line in the extra story for Perfect Cherry Blossom (included with the game in omake.txt). Youmu says at one point, "幽々子様！また、みょんな所に居て・・・" ("Yuyuko-sama! Mata, myon na tokoro ni ite..."). The somewhat stilted Touhou wiki translation for this is "Lady Yuyuko! What are you doing in such a myon place...". "Myon na" here is a non-standard word Youmu uses, probably a mangling of two words for "strange", "myou na" and "hyon na", but not further explained by ZUN. This stuck as a very popular nickname for Youmu.
