That is the beauty of the system. Specialization is great, but without the people (pop) to "run" the facilities, it doesn't matter if you have a gazillion factories to produce that wonderful armada. You need farms to drive everything, as farms equal people, and people equal money.
Flipping the coin however, too many people means overpopulation, which will bring down morale, which can, and certainly will, lead to people not working. This will cut into your production. So, you end up spending more money on social buildings to keep up morale, and/or spending valuable tech upgrades toward it, or spending actual cash to appease the people, or the ever dreaded "lower that tax rate", which in turn defeats the whole purpose of that high population in the first place. Lots of check and balances on running the economy in GalCiv2, which is why it is such a killer strategy game.