im having a similar issue, but its when we leave the game and rejoin. Ive also noticed when my wife leaves the game (myself being the host), and joins back sometimes NPCs I see are zombies for her.
We get the icons telling us there is friendly's or bandits with guns but it ends up being just zombies and killing them doesn't clear the icon. We also started in Louisville and going back all the remaining NPCS are gone.

Im going to try a fresh clean local game today with a lot of mods ive been using turned off.
I did read on the bandits mod page that any mod that alters the zombies other than cosmetics might cause conflicts. ive also seen someone has made a collection of mods that seem to work well with this one.

I also see mentions of tweaking settings for the mod, but on my end I don't see anywhere in the sandbox settings for the server that allow me to tweak the bandits at all.

Ill give a fresh game a shot first, and cherry pick my mod selection carefully today and absolutely provide an update with the results.

if you could, maybe share which mods you have enabled on your game.

EDIT:
So im a big dummy, and I learned that I need to enable the mods in the main menu for it to show up in the multiplayer sandbox settings.


UPDATE:
After a good two hours of spider boarding. I did notice that the Bandits mod itself was updated December 19th and today. Its name changed in the steam workshop list. having recently installed this mod dec 17th there's a possibility that might of broke things because it lines up when I started having these issues.

Once I start a game, survive day 2 ill DC and see if there's issues.