Yes, 100% it is the map causing this issue. Players are disconnected by the server because of this map at random. They have said it disconnects them with weird characters and sometimes crashes their Garry's Mod.

Pretty nice map, but it needs some bug fixing before I add it back to my server.