No buildings at all? What? Or did you mean not building them? Building Maintenance is a pittance, so surely that can't be the reason right? (I tend to pick the trait that increases it for bonus points in my custom factions ^^)
It works if you're doing extreme colony rushing. Each new colony adds production that you can balance so it costs nothing while adding research and manufacturing to your empire. Once you rush colonize, you can start building and specializing the colonies without putting stress on your budget. I've never done this because it doesn't sound fun to me and doesn't work so well on higher difficulties.
I don't do extreme colonizing. My games are more built up and based on a defensible position.
I prefer to build my homeworld with 3 factories and fill it up the rest of the way with labs, rush buying until I have at least 3 labs. If I have a secondary planet, it becomes all factories. I generally build 3 factories as my first buildings on all of my planets and fill them out with whatever they're being specialized for. No hard and fast rule, though, since each game is unique and staying flexible is how I beat the higher difficulties.
I don't bother with farms until my planets start to bump up against the pop limits.
Don't forget the production slider. My global one starts at 50/50 manufacturing and research, 1% econ with 100% social. Yes, that's 101%. Any planet that I want to add to a shipyard or specialize gets its own setting. Shipyard planets are generally 99% manufacturing, 1% econ and 1% research.
Once you get the higher terraforming techs, you can go heavy factories and econ. Money is real nice because you can upgrade those ships that are floating around close to the enemy instead of having to wait until a fleet is built and moseys across the galaxy. That becomes pretty important when you're racing towards a newly discovered planet or resource. If your survey ship finds a small fighter, it can quickly become a constructor and extend your range.
I found the hard way to pay attention to the economy. You don't need bundles of cash, but you don't want to go negative. If you do, you can't produce anything until you're out of the red. Two or three turns of that can set you back dozens of turns over the game span.
I don't build military until I need them. Most of the time, that means medium capital ships as my first fighters, turning to carriers as soon as possible. If the AI's getting aggressive, it might mean small missile only ships to keep them calm. I specialize my military in missile ships. it's better to have one decent weapon rather than multiple weak ones. I could just as easily go beams, but the AI likes them and I can trade for the tech later. I don't bother with defense. Missile ships can lob away at the enemy from long range.
My armed survey ships go after ship graveyards and my unarmed ones go for artifacts, capsules and space junk for the most part. Those anomalies come in handy. I eventually have a fast large survey carrier ship scooping up the remaining ship graveyards. There are only two small pirates guarding those, so one carrier module and a missile on a large hull will do the trick.
Good hunting.