I think there are lots of things that can be done to expand the diplomacy aspect of the game
Here here. This was one of the most interesting aspects of the game, and it has, in my opinion, been neglected to near death. As a matter of fact, other aspects of the game have been enhanced, or nerfed, to the point where diplomacy is an inconsequential factor.
don't get me started on the abortion they call the new espionage system.
I have to disagree with you on this point. The ability to do damage to an enemy without engaging in full fledged war is a good one. I do agree with you in that some have found ways to take advantage; dropping a spy onto a farming tile to decrease population prior to invasion should be disallowed, but all in all, I like the system. My one complaint would be the lack of 'soft' espionage. IE, the ability to gain information about another civ without actually causing any damage.
My problem with this context is that so far we don't get nearly enough info about *why* an AI might not have another AI on its list of possible attack targets.
Yes, this can be quite frustrating. I can understand the obvious reasons; alliances, treaties, and the like, but as often as not, the reason is a mystery. I don't necessarily need this to be spelled out in the game, but I would like it to be written down in black and white somewhere.