you can find green in redwoods, just laying around
you can find blue in caves and on Ab Island or chop white trees for sap
you can find red on monster island or chop red foliage for sap

i found red gems in a secret underwater swimming spot in one of the desert sulfur caves

All 3 can be found on ab island.

Are you sure @Nayr438? I could find all of them except for the green gems on aberration island? Where are they?

So far I've found Green in the cave of the Devourer, Blue on the surface of the Aberrant Island in the south-west and Red in the cave beneath the Aberant island but that's it.