I wouldn’t recommend building your own martial art, even in a fictional sense unless you have a few black belts underneath you. The problem is that the innate understanding of how techniques are put together, which techniques are used and taught and how they feed into each other and build off each other as you advance up the tree just isn’t there.
It’s just going to be easier in the long run to find a martial art that fits your purposes and warp it’s history to suit the history of your setting. Preferably, it’ll be one whose history already mirrors the themes and philosophies your story supports. The quickest way to figure that out is by deciding what you want from the martial art, like everything it may require a lot of research into different martial arts and their backgrounds.
You might also want to look where you’re drawing your inspiration from for your story to begin your search. For example: while monks in both Europe and China did go into battle and learn the fighting arts, if you’re pulling primarily from D&D you want to go with Shaolin. The reason is that the basic philosophies of Shaolin, The Tao, and Confucianism are already present in the way D&D structures and puts together the monk class. Your monk may subconsciously end up reflecting those tenets even if you didn’t intend for that to happen.
When you know what went into something and the inspirations it took from , you can extract what you need back out. You can strip away the superfluous elements that are unique to the setting you took inspiration from and keep the idea you wanted to take without the risk of someone pointing to you later and saying: oh, this came from X. Most monks in modern fantasy fiction are drawn from the D&D mold, because of the Tao and the use of Chi (energy). Because of Star Wars chi is often accidentally translated into magic. If you want a Christian monk, you need to use European styles of fighting. Christian monks often carried staffs, cudgels, maces, and even swords (Friar Tuck is an example) because they would often be facing armed and armored opponents even among the peasantry. By the time the Catholic Church had spread across Europe, hand to hand techniques would have been mostly useless to them. This doesn’t mean they were less skilled, different circumstances call for different tactics.
Remember: martial combat is reactionary, what feeds the creation of a style is the challenges the practitioner will face in the world around them. By figuring out what those challenges are, you can then turn to a society that also faced similar challenges and find a martial art with a philosophy that will fit your setting.
Most writers start at the end point, the results are what they see. Don’t start there, back up to the beginning: what are the pieces at work and what sort of world do they build together to create. Know your world and you’ll find your combat style.