Ahh. I had never before heard of the poor cities denigrated down to just their build queues, and that designation was throwing me off, but I guess that makes sense if that's all your interested in them for. *grin* FYI, my units used lightning pikes (about 120 attack per stack of 5), so you know I took my sweet time researching and building (rather than going for the quick domination victory).
As a related note, I promise you do not need very many cities for their build queues (1 fort and 1 conclave served me well--the rest were redundant), but what you need them for is Faction Power. At times, I had a score a mere 2-5 point ahead of the AI opponents (early game), and yet I was able to get tribute from them just fine. I probably had 3-4 cities at that point, and no noticeable growth penalty (those initial cities were huge by the end of the game, as I said, giving me more money than I needed to spend). I played an enchanter, so used the scrying pool for an extra city buff and switched the buff to growth (Sovereign's Call or something like that) from Inspiration (which is only good for the initial start, so don't be afraid change that buff to match your area of pursuit).
I loaded up an old save to check, and I was at about 190 seasons when I demanded the rest of the world give in to me (430ish power when the remaining 3 opponents were under 200). I played a bit more just to see what a level 5 town's choices were (and a couple choices are great).
Heh, it sounds like you pushed military at the neglect of Civ tech and building a flourishing civ, so it seems odd that you'd complain about growth (as you can see, I had zero problems with it, building the well/inn line right after the material boosting buildings, as well as the occasional consular from protected outposts). It could also be telling that I only used a single army of 4 units to own the world, with 4 other units in various frontier cities "just in case" (and they guarded more against wandering mobs wanting to enter my territory than the easily cowed opponents who never really put up a fight).
It seems our styles/strats varied a little even though we ended with the same result (military victory)--and it's nice to know the game can look vastly different based on our choices.