Orbital Refineries are a bad deal in most cases. Trade ports are a far better use of your time and efforts. In most cases, it's better to build a trade port to earn extra credits and then spend the excess on the black market. It's only late game when you have lots of extractor upgrades that refineries become useful, and even then I almost always place them on planets that would break my trade chain.
One thing you have to remember to do is upgrade your planet population infrastructure. Population infrastruture increases the maximum population of a planet, allowing more people to live there and generate tax revenue for you. As well, if your population infrastructure is too low, you pay underdevelopment tax, so it's a good idea to get rid of that as soon as possible.
Typically, planets are a poor source of metal and crystal, their extractors producing very small amounts. If you have high upkeep levels (fleet research reduces their efficiency, unfortunately), this can reduce your incomes of raw materials to very low levels. However, neutral extractors in asteroid fields or other uncolonizable systems will produce far more than their planetary equivalents. These are the important ones to capture, and these are the ones you want to build refineries adjacent to. If you're on a map with few or no neutral extractors, you're basically forced to use the black market because it's completely infeasible to get enough metal and crystal through extraction alone.
