While I'm still not sure of the cause, I did run a game with just the RimVali mod, and it didn't happen, so whatever is causing it, its something interacting with another mod, I'll keep testing though.

I think i know the mod its conflicting with cause i have my avali turning invisible but for good.

And i figured it out. its mega fauna or alpha animals

Edit: Alpha animals makes the avali permanantly go invisible

It also seems vanilla framework does too