I didn't say it was a guarantee, but a lot of times you can get lucky and at least distract them long enough to make a mistake that costs them the fleet or force them to retreat. At the expense of a few ships I'll take that risk, especially if its a choke point.
You could send in a single Vulkoras (or generally - siege cap) level 4-5 and it'll be more effective and not so easily killed. I for one always have at last one or two Hangar Defense structures in later game at any planet of mine, which is enough to wipe 5-7 siege frigates before they do too much damage. Also, building two light frigs is enough of a counter to your attack.
As for the battle itself, I would concentrate on killing your fleet than losing one planet in the meantime, since with your fleet destroyed I can conquer a dozen more. Plus, those 5 siege frigs have eaten away 70 fleet points, which is circa 10-12 LRM frigs, which could have helped you very much in the battle.
Siege frigs have little use as they are, simply because they're too fragile, too limited, very expensive and basically do the same thing the caps do, but aren't so good at it.
On the other hand, they cannot be more effective, simply because killing planets would be way too easy.
They are an expensive alternative if you don't want to waste your cap ships' time on bombing. Just as colony frigs are an expensive alternative to Colony Cap for colonizing (for Vasari, for others they are moderately more useful for captuing extractors).
