I think that you generally do much better if you have at least one farm on decently large planets, which for me is any planet with about 8 or 9 Planet rating.
First, I do not put a starport on every planet. In the games I play, I have not needed it. It may be wise early if you want to put cheap defending ships on every planet. But by mid game, its pretty stupid to worry about having a starport on every planet when you cannot even use it to build reasonably useful constructors.
To that end, the only decent uses of any square on any planet are super projects, trade goods, research centers, and economic buildings. I tend to designate any given planet as either an economic center or a research center. Economic centers only increase economy by a given percentage of the tax income. So building 1 farm has a much greater effect then just one more stock market will.
Anyway, I like to have at least 2 factories on a planet, so that upgrades happen in a reasonable time frame. Its simply retarded to have a farm and no morale structure. So you will need at least 4 or more squares. Now, a planet wtih 1 morale structure, a farm, and 2 factories, is just stupid. If your going to build a farm, its best to do so on planets with at least 7 squares. Also worth considering is that Embassies also work with populations to increase influence, assuming you care to use them.
Population has the benefits of extra invasion defense, increased influence, and more impact on the United Planets. It gives you something to fill your transports with when you decide you like your neighbors planets, and would like to have them.
The only negative effect of large populations is that they may adversely affect your approval rating when you crank up the tax rate, and that they force you to either research the morale tech, or grab morale mines, or morale trade goods. The benefits outweigh the problems.
END COMMUNICATION