I tend to think of it as overall physical mobility. But, that’s almost more of a word association thing, or a general language preference than anything profound.
Basically, the same reason I don’t like using the word dexterity outside of talking about game stats.
So, second part of this is size doesn’t really matter. At least, not in the way you’d think it would. Spend enough time dealing with martial artists, cops, or even former soldiers, and you’ll find some really heavyset individuals who can move. It’s not something we tend to think of, but someone can be in both really good condition and overweight at the same time.
Games like D&D, tend to give Elves a Dex bonus because they’re pulling from the Tolkien model of fantasy races. That is to say that elves are, by default, supernaturally graceful. So, they get a +2 Dex bonus. I wouldn’t read much more into it than looking at how much Tolkien has influenced our general concepts of fantasy.