IIRC, the game biases high-level monster lairs away from the player's starting point. It doesn't do that for the AIs. Which means that the AIs start with clambercoil dragons and all the other nasties right on their doorstep. And no number of club-bearing militia is going to take down a dragon.
I think that factor alone could account for the inability of the AI to be competitive at the mid/late game stage. It's really hard to get your civilization off the ground if the monsters are constantly stumbling into your fledgling towns.
Meanwhile, expert players are happily clearing fields of darklings and wolves and spiders and wondering, "What's wrong with the AI?".