I'm not pretending to be an AI expert here, I'm just developing a piece software with somewhat similar real-time decision making and these are my educated(?) guesses of what may be going on...
It's difficult to develop an AI that's good in all situations, i.e. that can generalize its skills from the cases it was developed and tested on to the random galaxies of all the zillion galciv2 players. But it's the only way to prevent massive loopholes and cheese to even some degree and provide some illusion of intelligence and strategy skills. If you just hard code some "expert" rules (do A if B and C etc.), you end up with very predictable, machine-like AI that works only in very specific circumstances and is easy to trap with some specific tactics.
Even a war-optimized AI will have some loopholes for sure and you'll win it every time when you discover its weakness, i.e. the simplistic rules it's based on. There is no free lunch, no silver bullet. Trying to develop a more general model is the only way to go even if it's futile with the resources available.
Frogboy, your posts indicate that you have been focusing on generality, not special cases and it shows. The AI
feels more varied and fun to me than maybe any other TBS AI I have tried!

Maybe it could beat me easier if was optimized with only that in mind but I wouldn't like to be forced to "build the the same kind of perfect fleet ASAP in every single game" all the time. If I want challenge, I try to beat the game with weird ability settings, find new ways to win&lose etc. I like to create a kind of RPG-like story of each game, not necessarily repeatedly hone my skills of rushing the AI in some specific circumstances. That's what fun to me. YMMV.
Of course, there are bugs and weird things happening. Take the "AI builds way too much embassies"-bug for example. My wild guess: it would make no sense to fix the bug by just limiting the number of embassies the AI can build in a single planet. You would never learn what the AI was doing wrong in the first place. It's better to find out the root cause and fix the AI's general way of thought instead of duct-taping the problem away. It's still good to complain "please fix the annoying embassy bug, I hate it" so that the developer knows what is going wrong from the players' point of view. I'd like to see the bugs fixed, too. Just
please don't forget to say please! Stardock devs are remarkably open about the product they are developing and making themselver vulnerable to all sorts of flaming and abuse. I don't want their dev journals and quick replies to these forums to stop. (BTW, the memory management and AI related journals have been great.)
Some of Precursor10's suggestions sound good to me; from "use case", not "force this rule to AI" point of view. There are areas where the AI plays too dumb. I guess the devs know about them because they seem to read these forums actively and can't possibly be so stupid to not spot the glitches in the game itself.

It just takes time and is
really not trivial.
Oh, and "AI:s should gang up against the human player" is a very very bad idea, IMHO.