One note- In the post I use Stardock as an example (instead of "the company"), since we are all familiar with the company, the game and everything.
Legal issues could indeed be a problem, though in many cases, not.
In your example of taking a story from a book, that's a common thing to do. How many worlds (Elemental included!) are based on "the great empire fell to a catacalysm some hundred years ago."? Elemental, Dragon Lance saga, just to name a couple. Does that means that Elemental have to pay revenues to the Dragon Lance writers for the idea?
There might be some such problems, but in many cases it can be solved.
Paying for mods- Assuming we are not all pirates, if I can allow myself to pay 40$ for a brand new game (say, civ5), and later on 10$ for its DLC content, I could just as well pay the same 10$ for a mod-content.
Money distribution (among the team) would most likely go as a contract. Most works has a credit section- everyone's signature will have to be put, and arranged between them. If a problem arise, any of the members may put a hold, as well as copyrights, to remove the mod from the market until the issues are cleared.
This means that if I use the works of a composer who dictated his work to be used non-commercially only, and he dislikes my use, he can put a hold on everything I do until Stardock, me and him clear the issue.
Money ruining the mod community- just one thing I want to say is, that in many cases, each game has only a few "spectacular" mods, the rest, to the community mind, are either "bad" or not in the stage to be used. FFH was a spectacular mod that got as a mod example to Civ4 (FFH1, not 2). While the effects might have unforseen results such as degrading or uplifting the community, I don't believe we can just state it.
A way to make the whole money for mod payment can be as follows: Stardock releases Kumquat (I believe this is the engine name), with an official mod named E:FE, for 40$. People who "own" the engine, may build maps and mods, as well as download them (a slightly different version of this is the CryEngine, where the company released the Crysis games as a trailer to the engine, with the plans to earn the big bucks from selling the rights to the engine). Maps could be sent to Stardock for approval. Depends on the value a map is given (from 1c to 1$, I have no idea) it's sold independently/in a bundle or an other system. This system can go around many ways, such as selling it directly to Stardock, or any number of ways.
Mods are a little trickier- I'd say that mods will be free to use, and if it gets popular the modder may sign a deal with Stardock. In a sense, he is working for Stardock. We can see how a person who made a great mod (Derek) got into a high position. The same way, the company can hire the modder, as it sees him as an asset to the game.
An option for the "I don't want to pay for something I know nothing about" is the Android Market system, with their 24h 1 time money return (if you buy an app and dislike, you can cancel the order in 24h, money returned, no questions asked, once per app).
DRM- Most pirates are cracking the game, and maybe the patches. SOTS2 for example, is a game I won't touch without testing firsthand, but there is no crack from the past half a year out. (Note- I do not support piracy, but badly released games are not something I will ever support, either) It's relatively safe to assume that the modding community will be protected like the online-play.
One last thing, regarding the money degarding the quality: In a different entertainment brunch we got Comics, and the Web Comics. Many comics cost a small amount (5$? There are no such shops near my house to know the prices), and web comics are usually free for all. Web comics usually earn revenue either by adverts, or donations. There are people the work full time as web cartoonists, and earn their living in such a way. In some cases, I do believe that we could get some great mods from people that cannot dictate their full time to the project, but would if they could.