Have you read the entire modding documentation? Because it contains individual explanation of how the tools work and then another section that goes step by step trough expanding game files, using the individual tool, making changes etc.
I don't remember any steps being out of date, especially considering that there's rarely any changes to the mod support.

The step by step guide for the rooms starts at the bottom of page 13 of the documentation. So I would suggest looking at that.
You need to first use the tmx tool to convert the binary room files into tmx that tiled will be able to work with.
Trying to load the binary rooms files (what's in the res.pak by default) with tiled will not work.