If you use Devour Land on another Empire territory it should come with a reputation/Public Opinion penalty on their side.

Right now I can freely devour the lands of my opponents all around their cities without penalty.