If you have a ton of cities, your prestige is spread, and because high grain only helps once you're at the cap, it's going to take a long time to get there because your growth is slow. High grain does exactly nothing until then. Even with prestige it still takes awhile to reach higher levels.
High materials are very useful immediately, both for building early game armies, and for rapidly developing a city to its current size cap.
Again, that's the whole point of the prestige mechanic - you're not being penalized when it can't grow, you're being penalized for choosing to build the extra city.
Removing the penalty or softening it would remove the point of the mechanic in the first place - whether it's a good mechanic or not is up for debate (I'm fine with it, it's a simple and elegant way to differentiate between large or small empires), but it's not a bug.
The prestige bonus is _huge_ compared to the growth buildings that are available early (and even mid) game. An inn or pub gives +0.5 per turn, you can be giving many more times that rate if you have just a few cities.
I still suspect that a sprawling empire is superior, but it does make sense to develop a few large cities early in the game, because many cities early can't support the double whammy of troop support and building maintenence costs without going into the red. Whether a few (or 'fewer') cities is viable on larger maps is another issue entirely.