In Escort Merchant Caravan quest, if you engage the raider before the caravan does and defeat it, it will usually start fleeing with few troops remaining. These few troop will then spawn on map and relentless chase the caravan. If you try to engage them again, they will fleet at battle start (yield some loot), and re-spawn again for infinite times..
In base game, two things are different, remaining raider after defeat no longer carry the quest marker and stop chasing the caravan; also, when there is only small amount of troops, those troop generally disappear directly.

Workaround atm: auto resolve the combat (send troops). However, the result is usually really bad for this (usually get few death even when 60 cavalry vs 5 type of battle with 10 heroes..)