So, how to board? Basically, you fly up to a ship, preferrably a hostile one, and grapple it first. This has a hotkey, the "," on the numpad (usually between numpad 0 and numpad enter)
Then you have to call up the context menu on the enemy ship (3rd mouse button or pressing the mousewheel) and order the boarding from that menu (upper right). This part can be a bit tricky, since you have to view the enemy ship to do it. Using turretview (v) may help.
Ordering the boarding action will only send over the first troop and also open the boarding window. When your transporter cycles, you can send more troops over (the button with the arrow towards the enemy), which is highly advisable.
Your marines will then duke it out with the enemy crew, which might take a while. The enemy ship will still be firing at you. However, do not shoot back since you might kill your own troopers or even destroy the ship. After the enemy troops are reduced to zero, your people will then proceed to sweep the enemy ship and secure it. More of your troops may be lost in this phase, presumably due to traps. Then, if you still got some people remaining, the ship will be yours. You can't take control at once however. You must tow the ship to a station or planet. Once you anchor there a window will pop up allowing you to sell the towed ship or put it in the garage for later use.
While you are there, remember to hire new troopers to replace combat looses.

At first, you are quite a competent boarder. Your ship can carry 22 troopers while the usual enemy gunship (which you will be facing most often in the beginning) only carries 16 or 20.
Having more troopers is very important for winning in boarding. It is also helpful to have a large transporter capacity und quick cycle time so you can get your guys over there in a hurry, so enhancing the transporter might be worth it. Until you have deployed all of your troops the enemy might have advantage in numbers and decimate the boarding party. There are also some things (officers, perks etc.) that increase trooper effectiveness or hamper enemy effectiveness.

But there are also things you can do in gameplay to optimize your chances of boarding success. You can damage the enemy ship. In my experience, this will reduce the value of the ship only slightly, but causing hull damage also has a chance to kill enemy troopers. Less enemies = easier win. Just be careful not to cause too much damage or the enemy ship will blow. Also be mindful of the situation. Oftentimes you will be boarding a pirate or other criminal and there may be other lawful ships nearby that also shoot at your target. And they won't stop that until you have captured your prey. If you cause too much damage before boarding, their fire might blow the ship up.

- you do not need to grapple first, but it helps to keep the ship in transporter range
- you need to grapple a ship to tow it
- how fast you can go without loosing tow seems to depend on relative ship sizes (the smaller the towed vessel and the larger the towing vessel, the faster you can go)
- if you loose grapple, just go back, grapple again and go slower this time
- reloading seems to refill the troopers for hire, useful to replenish troopers on larger ships
- I am not sure the bonuses and penalties to trooper strength work correctly at this time
(so far I didn't manage to cap a ship with more troopers than me, but I got large bonuses from perks and can impose large penalties to the enemy from an engineer skill)
- Damaging a target ship does not actually kill off troopers but makes boarding easier anyway (maybe via the effectiveness mechanic)
- using the chase camera is most effective to call up the context menu; however this camera mode does not seem to have a hotkey per default (although you can assign one); this camera mode will make you look at the locked target automatically