Maybe first player is supposed to be controlled by Keyboard/Mouse?

Well, I wish this issue had been resolved (my reason for resurrecting the post), because, after years of having no issues whatsoever playing couch co-op games on my Samsung TV through Steam Link with two XBOX One wireless Controllers (connected via Bluetooth to the Samsung TV), tonight we've hit this issue.

The game is Conan Chop Chop. We've played it twice with zero issues. Tonight, however, we go to play, and I (first player) am suddenly controlling the second player while player 2 can't control anything. After much confusion, we learned that the remote control was controlling the first player. As OP said, taking the batteries out of it (even before starting Steam Link) doesn't solve the problem. It is still in control (somehow).

If anybody knows a way to stop allowing Steam Link to use the TV remote as a controller source altogether, they would be literally saving lives by explaining how. Note, there are no actual human or animal lives at stake here.