(Note: All "costs" below are production costs, and obviously will not directly apply if you're rush buying.)
So long as you're upgrading them to buildings that cost more (which for a [basic] slave pit is damn near everything), you're okay. But for instance upgrading a manufacturing center (150) to a stock market (120) is going to wind up costing you 200 total production-150 total for the manufacturing center, and then 50 more to upgrade it to a stock market because the "minimum" upgrade cost is 50-except when the upgrade costs more.
So if I upgrade an embassy into a research lab, it'll cost me 30 for the embassy and then 10 for the research lab (because the research lab costs 40). But then if I decide I want to replace it with an embassy again (for whatever reason), it's going to cost me 50 production to do that.
One other thing to be aware of is that social overflows. If I have a planet with 45 funded social build an embassy, the embassy is built and 45 production is spent on it. Then when I go to overbuild it with a research lab, I'll have to pay 50 production because 40 is less than 45-but if I overbuilt it with an entertainment network (cost 55), it would only be 10.
Since most of us don't have our improvements built with exactly how much production they cost, keeping the difference to at least 20-25 when upgrading is a good rule of thumb.
One last note-we had this discussion in another thread and I complained about overbuilding not getting any credit for previous buildings. I then of course tested it in TA and it worked fine. However upon testing it in DA just now (because I had it open testing something else), only rush buy gets the overbuild discount there, so this is yet another thing that is Fixed in TA™.