The short version is; there isn’t one. As far as I know, there isn’t even anything vaguely relevant. Forms like Judo focus on dealing with opponents that are stronger than you, but there are practical limits, and a human being can only take or deal so much punishment.
This is a big part of why there are no hand to hand styles for dealing with bears, or wolves, or lions, or any other apex predator. (No, wrestling alligators doesn’t count.) Putting yourself that close to an animal like that will end badly. In the real world, we’ve dealt with that by using ranged weapons, and polearms; which is why I suggested those earlier. They allow you to kill a creature without getting close enough for it to disembowel you.
If she’s using hand to hand when dealing with other humans, and only using the blade on monsters, I’d suggest aikido, it has a strong focus on non-injury, and while it’s not terribly practical, it might philosophically fit, transitioning into junkyard aikido or jujitsu if she’s willing to harm people who get in her way.
Now, I keep pushing the whole “don’t go into hand to hand” thing, and here’s why: It depowers your monsters. If they’re supposed to be foot soldiers of a greater evil that anyone can deal with, and your character is just one of many people fighting them off, then it’s really fine. And, I’d offer the same advice as above, junkyard aikido or jujitsu.
But, if they’re a scourge upon the world, and no one else can oppose them, having your character take them out unarmed is going to risk doing seriously unfortunate things to your audience’s suspension of disbelief (unless there’s some really good justifications in why everyone can’t deal with them).
I’d recommend looking at The Last Wish, by Andrzej Sapkowski. Though, I’d be slightly more cautious about suggesting any of the adaptations of his work. But, Sapkowski does almost exactly what you’re describing, and has some excellent justifications.
For use of the Katana, I’d recommend Kirasawa’s Yojimbo (though there isn’t much sword combat). Tarantino’s Kill Bill Vol I might be worth looking at, if you want the more mythical version of the blade. (Though, as with all of Tarantino’s work, you’ll need to bring a strong stomach.) Michi’s recommending Rurouni Kenshin. She’s also recommending you look into the underlying cultural history of the katana, that’s The Book of Five Rings, and spending some time looking at Bushido. There’s a lot of cultural context with the katana, so if you’re setting your story in an amalgam of historical Japan, or even just using a Katana, it’s probably worth doing some further research.
If you’re insistent on working with the katana, then The Book of the Five Rings by Miyamoto Musashi (circa 1645), it’s a philosophical text on the kenjutsu arts and conflict. I’d also recommend looking into Iaijutsu: the art of drawing and sheathing the blade (Iaido in a modern context), Kenjutsu (Kendo), and Battojutsu as a study of sword combat in Japan. Do yourself a favor and pick one.
She can pretty much learn the sword or the hands, but she’s only got time for one unless she’s been practicing continually and doing nothing else like in a good old fashioned apprenticeship like in Medieval Europe with pages.
If she’s been training to fight monsters specifically, I’d look into a variety of other supplemental weaponry. If you’re going Japanese, stay with Japanese weapons as supplements. The naginata or some variant of glaive would be her pole arm, they might also train her on the bow, and practice with a wide variety of other useful skills like poison brewing and trap-making, spike traps, pit traps, etc. All the useful extras any good hunter needs to give them an edge. I don’t know if the Japanese ever did actually attach a kunai to the end of a rope and used it as a whip like the Chinese did with the Shaolin rope dart, but you know it’s not a bad idea.
Just try to stay within the Japanese frame and you should be fine, it’s a bad idea to play mix and match with martial styles unless you’re really willing to do the leg work (all the leg work) to understand all the themes they bring into play in your story.
(Edit: an investment of time, if you haven’t already done so, into some of the Anime and Manga that deal with Japan’s mythology and monsters might also be worth it. Digging into the monster ideas used in Claymore and Inuyasha might be helpful.)
For the use of a katana, another good movie might be Ame Agaru (After the Rain, in English). Its fight scenes are considered to be incredibly well choreographed.