can you fight this guy to get positive relations when your at -100

I think so, fighting anyone inside town does if i remember correctly?

Thats why he prompts you to head outside, and he follows you.

So run ouside and fight him away from town

Does fighting this guy piss off the guards?

---Updated ----
Quick changes to where dueler's live, priority and added chance they appear in bar to see if it improves the mod and conflicts less w base game. Possiblity that having 2 duelers in town, 1 resident, 1 bar could cause issues.
Also removed redundant line that added more shek warriors to residency, possibly conflicting w base game mechanics.
feedback on change welcome

Awesome! I will be tracking all your mods for updates so I can incorporate them to my current play through.

Good point , the original idea was to make them a bar squad. I'll look back into this as it should fix that issue

Upon exploring the mod further I was able to take the duelist NPC far away from town and fight him. Although, by giving the duelist NPC a residence in an already sparse Squin, it makes the police station dissapear and be replaced by one of the shops, therefore negating the ability to turn in any bounties in that town.

Is it necessary to give them a residence there at all?

I downloaded the mod and I trying it. The vanilla duelist NPC does not give me the option to initiate far away from town but rather, right outside the gates. I will test it again and see if it changes.

So in vanilla the duelist should have been discontinued . It may be you fought shek warrior as some may have this diag.
I modified the duelist to be turned back on with new behavior.
He warns you that u must be far from that guards and checks before attacking you .
However it is kenshis problem that wandering guards may happen by as you're fighting and join in , this is an issue I can't help.
So to answer your question.. Both?

Prior to seeing your mod, I accepted the duel and met the vanilla NPC outside the gates but the guards attacked me once the fight started. First time I tried it.

Does this mod add a new duelist NPC or modify the existing vanilla behavior?

I agree with Rinklestein. Your on to something

I think this could be a great framework for a companion (or more companion focused mod), something fleshed out with custom dialogue and etc. may really suit it. Really like the idea!