I’m pretty sure the Khajiit swipe open handed in Skyrm. The ready stance when you have your weapons out, but nothing equipped, is a pair of fists, but their actual attacks are just claw rakes. Which fits with the small amount of lore about Khajiit martial arts in setting.
The Khajiit have at least four defined martial arts. Gout Fang, Whispering Sands, Rawlith Khaj (which may translate to Rain of Sand, I’m not sure), and Two Moons Dance. Though there isn’t much information on any of those.
As for the animations in Skyrim, I wouldn’t read too much into those. I suspect the strike starting from a clenched fist has more to do with creating a unified resting pose for all 10 playable races, rather than because of an in universe reason.
At a more basic level, the animations are configured to give players as much information as possible about the current state of combat (is your opponent attacking you, are you attacking them, ect). This is the exact opposite of what you actually want to do in a real fight, because giving that information to your opponent provides them with a better awareness of what you’re doing. In Skyrim (and most video games, really), it improves the gameplay, but it’s not about realism.
In actual combat, you want to keep your strikes inside your profile (literally, the silhouette your body creates naturally). So the large roundhouse punches and most of the weapon attacks in the game, are techniques that would endanger you in the real world.
Now, there are martial arts that switch between open and closed hands mid strike. Usually making a fist happens while chambering the attack (when you draw back before striking) not during the strike itself.
In actual martial arts, there are any number of reasons to go between a closed fist and open hand when chaining from one strike to another. This is just because you can achieve different things with your fist than you can with the blade of your hand, your palm, or (in rare cases) your fingers. If open or closed is appropriate will depend on what you’re trying to do, and what your training suggests.