At least h264 and nvenc support recording modes with variable bitrate which are configured by a "quality factor".
OBS, ffmpeg and more already have them.
This would fit as an option and not a substitute