I have fallen and I can't get up

Okay, I found it out. After a long time (possibly when vassal integration reached maximum level) I got a special casus belli that allowed me to declare war on my daimyos. Of course that came along with a massive hit on the stability, -2 because they are my vassals and again -2 because of our great relations. With consecutive insults every year and a couple of years more, I might have been able to get rid of the latter stability hit, but eh. Once I annexed a vassal this way, the diplomatic insults from my vassals kept rolling in which gave me Casus Belli with which I was able to annex them without a stability hit.