Once you've located some wild horses on the map and built the harvesting improvement on them, you'll see horses coming in as a resource (in the resource tray at the top of the screen). After that you should just need to design and train units with horses equipped.

Heroes and Sovriegns can buy mounts at the shop, but you will need to train new troops to have mounted units. You will need the mounted warfare upgrade to use the horses.

Note: Kingdom uses horses, Empire uses wargs.

+1 move and combat speed. This is very worth while as it keeps your forces from moving at a crawl.

As others have said, +1 move speed is nice ...

In the current iteration, a base unit gets a combat speed of +2 for a total of 3.0 in tactical combat. Give them a mount and one more speed item and you get a combat speed of, say, 4.2. For, say, archers, this means 3 attacks instead of 2. Handy.

Yep - tactically anyone on a horse has the old "SLOW" animations.

It's already fixed internally and will be pushed next patch.