Yes they seem to keep spawning very close to my base sometimes in it. It could be because I'm in the giant mansion from the Crowlake mod and even though the property is walled in, it covers a lot of area

Did they spawn inside the building?

Yeah, twice! But It didn't happen again after that. Sometimes they spawn, I see the marker but they don't move towards my location until I get closer to them first, which is the opposite of spawning on me I guess haha

Ok so I'll check that

had the same issue, but i was just in the middle of the street wihout any zombies or bandits nearby.
a saw the icon of hostle assault and three guys with the crowbars just appeared around my character.

I've encountered a similar issue with friendly wanderers. They seem to have an innate knowledge of my location and an unnatural desire to reach me.

I've had several spawn homes lose several of their doors and windows before I can even finish looting them. Wanderers seem to spawn right outside the building and immediately attempt to path straight through it. Conversely, I don't notice them attempt to do so nearly as aggressively towards unoccupied houses.