Hello,
Go to your Steam folder (default C:\Program Files (X86)\Steam)
steamapp --> common --> RollerCoaster Tycoon Deluxe

Right-click on the RCT.exe, select "Properties" and click on the compatibility tab and select compability to Windows 95 or Windows 98.
Run the game and lanch a new game, now you can change the resolution in the game by clicking on the floppy-disk Symbol --> options and then change it to 1024x768, confim the changes

Have fun =)