The thing that’s honestly kind of frustrating with the Khajiit is, they have a couple named martial arts. We’ve even seen characters proficient in them… and, no animations to go with them, or any real explanation for what they do or how they work. Khajiit use the standard unarmed animations, and get a slight unarmed damage bonus, leading to Punchcat builds.
But, that distraction aside, this is actually a world building question. Though, the Khajiit are a pretty good example of it, so we’ll keep track of them for a minute.
For characters with a non-human biology, any martial arts they develop need to do three things. They need to reflect their physiology, they need to reflect their society, and they need to account for the societies they interact with on a regular basis.
Physiology is the easiest. If your characters can bite, that’s probably going to be a part of any formalized martial style. If they have retractable claws, those will also probably find a home. It’s the same with stingers, barbed tails, horns, serrated plates. If the character can use it safely, and there isn’t a social stigma associated with it, they probably will. And, in an emergency, that stigma might not be enough to keep them from using it anyway.
The Khajiit are actually an interesting restriction on the physiology element. There’s actually seventeen different varieties ranging from bear sized cats, to the bipedal cat men seen in the games, to almost human looking, to intelligent house cats.
If you’ve never looked at it (and it’s not already becoming apparent), The Elder Scrolls is a very strange setting. One that’s passing itself off as normal, but poke it and weirdness starts to seep out everywhere.
I suspect a large part of why they’re so vague about what the martial arts entail is because most of those arts are supposed to be accessible to most Khajiit, and the range of physiologies makes trying to get into specifics impractical.
The characters’ culture will determine a lot of what is or isn’t socially acceptable. If you have a culture that embraces their bestial impulses, or just considers that a normal state of being, then again, they’re probably not going to have an issue using their claws, teeth, or other appendages.
It’s worth remembering that, for any civilized culture, proportionality is still very important in combat. For instance; if you have a race with a venomous bite, biting would be viewed as an attempt at lethal force. With all the associated consequences.
As with humans, the amount of harm done to the victim will be predictable, and their society’s values will influence how their violence is perceived.
If their culture considers their nature bestial, or something they seek to suppress it, then things like using their claws may be viewed as shameful. This would make attacks using them much more extreme.
I said there were external cultural pressures as well. The simplest way to think of this is, as an entire, societies are affected by peer pressure in the same way individuals are. It’s not as immediately apparent, but it can and does happen.
So, if your fantasy race is set apart from those around you by physiology, there’s a very real chance it will affect their outlook on the world, and their perception of self.
I realize this gets into an uncomfortable topic, but it’s one you should probably consider, both in building your world, and your characters.
A couple good things to look at: Though, none of these will be especially useful for the fighting specifically, they should help with world-building in general.
Lucasfilm’s Aliens Chronicles by Deborah Chester: What’s stuck with me, years later, was how distinct Chester managed to make the various alien races. It also does a great job exploring the potential effects of outside influences on a character.
Farscape: I know we’ve plugged this series before, but it’s very good. This is probably one of the best Sci-Fi television series out there. It populates it’s universe with loads of memorable and unique aliens. In part the writing uses that to feed the sense of otherness and alienation that it’s driving, but it usually keeps the strangeness on point.
Since we started with the Khajiit, it’s probably a good idea to look at The Elder Scrolls V: Skyrim, assuming you haven’t already. Skyrim’s tricky to recommend, though. The main story’s writing is fairly standard save the world fantasy fare, and the protagonist is a fantasy, demigod, superhero. But, the world it builds is far more interesting when they player isn’t shouting people off mountains to watch them ragdoll. Things like the treatment of Khajiit and Argonians by the Nords, are fairly easy to find. Digging up the full depth of the setting is a lot harder, and something the game doesn’t ever really suggest you should do. You really should.
The MMO will paint a more coherent picture, particularly of Khajiit and Argonian culture… but it’s also an MMO, so the signal to noise ratio is insane, and I’m not about to say you should spend 200+ hours to dig up it’s insights. Even if it does do a fantastic job of looking at racism and it’s aftermath, which might be relevant for you.
Morrowind is probably still the best game in the series, but, the last 13 years have not been kind to it. It’s wonderful, but incredibly slow. I can honestly say you’ll see a different world in that game, but it’s probably one that would take more of a time commitment than you have (even in comparison to the MMO). That said, if you’ve gone through the later games in the series, but missed this one. It is probably taking a look back.