You can tell him to leave your territory the same way he tells you to leave his.  The AI is just generally pretty vigilant about always threatening you and maintaining border integrity.  The mechanic (the ability to violate borders and try to sneak through) is nice, but the implementation is clunky.  Really, there should be an option to tell the unit to leave right in the unit window when you select a foreign unit that is in your territory.  Having to go into the diplomacy screen for it every time gets old.

Or just a general open/closed borders option with a certain faction.  

That could be a good improvement for the diplomacy

Add it an option to trade units and cities and that would be a great combo

The lack of an ability to tell the AI that my borders are closed is one of the reasons why I tend to simply go to war with every AI I meet that doesn't have something I want in their shops which I could not get otherwise (which basically means they are Altar, New Pariden until my champions have as many spellbooks as I want them to have, and Capitar if I want the warhorses - assuming that Capitar sells warhorses in its shops instead of regular horses). That, and I hate it when they send a unit across my territory (because I really don't want to enter the diplomacy screen every single turn to tell them to get out) and beeline towards a hero I've been waiting to recruit due to funds or research priorities when they shouldn't know that that hero even exists.

The only times in history when nations have sent armies wandering through the territory of other nations is when they are at war, made prior arrangements with the nation over whose territory their army is passing to allow that passage to occur, are allies and have a reason to send armies into their ally's territory, or don't care what the other nation thinks of having their army pass through the territory of the other nation (trying to provoke a war, think that the other nation isn't strong enough to be a concern even if they take enough offense at the unsanctioned passage to go to war, etc).

Well, there's also the "we aren't TECHNICALLY at war, but I think that that bit of land is mine and you think it's yours, so I'll keep an army camped there to keep your army out of there, and if you send people by we'll kill them, but we aren't prepared to invade what we agree is your territory over it" situations, i.e. ill-defined or disputed borders - which don't visibly occur in E:FE. All national borders are clearly defined, with the potential for no-mans lands where anyone can travel.

The game already has a means of checking if something is within my territory. All that needs to be done to establish a closed-borders situation is to set up a flag that tells the AI to check to see if the path of a unit ends inside of another faction's territory (i.e., would trigger an 'enemies are in your territory' alert if the AI and the other faction were at war), and then decide if it wants to provoke that player by moving its units into their territory or not. Territory violation by passing completely through another player's territory in a single turn would be more difficult to prevent, but because most units can only move two to four spaces in a single turn it also isn't as likely to occur as moving into another player's territory and ending the turn while still in that player's territory.

I also don't mind the territory violation by crossing into and out of my territory in a single turn as much as the AI ending a unit's turn inside of my territory, since at least if they crossed into and out of my territory in a single turn I don't have to tell them to leave via the diplomatic screen.

I tend to see the AI push with pioneers to the opposite side of my territory one of two ways.

If it's just one pioneer running around, I'll just rush my own pioneers and set up outposts so that they can't snag any resources on my side of the border.  If they do manage to avoid being eaten by monsters, I can take back one outpost no problem.

If it's two or three or more pioneers shamelessly gobbling up the resources and trying to build cities on the other side, I'll declare war and remove them by force.

Yes, it's not always a good idea to declare war on someone, but chances are that by the time these last pioneers are crawling all over your land to get at the good stuff on the other side, you have a fairly good economy going, you just need to give your research and military a kick in the pants.

I tried the blocker feature but only blocked the one tile where they could build a city. THey built and outpost next to it and kicked me out and built the city anyway.

Personally, I usually go for the non-aggression pact which lets me enter the other sovereign's territory and stay there.

(I tend to not leave any resources available for pioneers to gobble up, and the few that they do get I try to feel philosophical about.)

You should also be able to tell some know, I am just passing through and it not always result in war being declared.

Perhaps, an option-- I was just passing through, would you accept 10 gildar as a toll. 

They also get unhappy with you when they attempt to send a pioneer through your territory to steal your shit from under your nose, and you tell them to get the hell out every damn time.

I've found that their happiness with me has little to do with how long they tolerate my units' presence behind their borders.  In fact, I've had opponents who like me issue warnings at once, and opponents who really, really dislike me keep quiet for 2-3 turns after I entered their turf.  I've even reloaded to test what's going on, and could make no sense out of the AI's response: sometimes I'm kicked out at once, at other times nothing is ever said (I've literally gone more than 20 turns behind one opponent's lines, my sovereign's stack aiming for Wildlands).

Just another example in the game of behavior that, however well defined it appears to the developers, certainly appears undefined or confusing to those who play.

You can return the favor via the Diplomacy screen.  The problem isn't that you can't kick their units out, but that the interface for doing so is clunky given how often they send units to your territory.  To have to go into the diplomacy screen for each faction each time they send units into your territory is tedious.  

This seems to be a cumulative issue - if you have had longstanding warm relationships you get different behavior, but you are right that there also seems to be a random element (which might mean that my impression of a cumulative effect is just me fooling myself).

Note also that pioneers cannot settle in your territory.  (Some people in this thread have expressed this idea already, but i think it is worth repeating.)