On the western island below the larger snow biome, there is a small cave in the side of the hill. Red gems are found there. I can also think of two other places, The Redwood Island Cave which you access by diving underwater near the north edge of the island, and the cave in the non dune desert that has metal in it, which the red gems are in a secret place you can access from diving in the water pool and swimming to the left. Careful of the megalania.

red gems are also found on aberration island in the west behind wyvern vulcano island.
You can find them in the river that divides the island.

Red Gems are also located on Carno Island...only 3 spawns I've found so far.

Red gems 54.8x12.1 Inside cave.
Blue Gems 82x8 Several spots on that island have them.