Here are my findings after studying source code.

The most important thing is that number of stealth kill bars scales linearly:
1 bar: 1%-25% chance of instakill
2 bars: 26%-50% chance of instakill
3 bars: 51%-75% chance of instakill
4 bars: 76%-100% chance of instakill

Stealth kill is possible only with single hit attacks. UI is confusing as it shows stealth kill chance for burst and autofire, but the chance is ignored.

Factors affecting chance to instakill:

• min chance is 1%
  
• attacker's Dexterity: +1% for each dexterity point
  
• target's Wisdom: -1% for each wisdom point. Yes, each unit has wisdom, for instance Goon has wisdom 30 while Elite Raid Leader has wisdom 85.
  
• if at this point the chance is negative because attacker's Dexterity is less than target's Wizdom, the chance becomes 0
  
• -40% for Crocodile and a few story characters
  
• +10% if targeted body part is head or neck
  
• +15% for attacks with 3+ Aim levels if attacker has perc Assassination (Dexterity tier 3)
  
• +10% if attacker has perc Ambusher (Dexterity tier 2)
  
• +10% if attacker has trait Stealthy. Also final chance to kill instantly cannot be below 30% if attacker has this trait.
  
• +2% per Aim level if weapon has Tactical Device modification

After all those factors are totaled the game can multiply the calculated chance based on:

• target state:
  a) *0.33 if target is Aware
  b) *0.5 if target is Suspicious or Surprised
  
• *0.5 if attacker's weapon penetration class (light/medum/heavy) is worse than target's armor class (light/medum/heavy)

Extra information from TheSherman:
"if it is a successful stealth kill (you passed the check) there is no gun noise whatsoever. if you missed your check but still kill your victim, they will hear the gunshot and pain grunts"

Example:
Fox with Dex 92 and Stealthy trait uses weapon with light penetration class to shoot at Goon's medium armoured body part, the Goon has Wisdom 30 and is Surprised after his teammate has just been killed by Fox.
So we have 92 (attacker's Dexterity) - 30 (target's Wisdom) + 10 (Stealthy trait)=72.
Then we multiply 72 by 0.5 because Goon is Surprised and get 36.
Then we multiply 36 by 0.5 again because Goon has better armour class than Fox' weapon penetration class and get 18. But since Fox has Stealthy trait the chance cannot be less than 30% so Fox has 30% chance to instakill Goon.
As you can see difference between 72% and 18% is huge and it is caused by different enemy state and armor.

Even if enemy has a perk to avoid first hit during each turn, it still dies to instakill. But if instakill roll fails and enemy gets just a normal hit, the perk is activated and enemy avoids all damage, even if the damage would be 100% HP.

Another example:
Grizzly with Dex 76 shooting at Elite Raider Leader with Wizdom 85. Calculated chance is 0%. Min chance is 1% so we have 1% of instakill. It does not matter if target is aware and what armor class uses. The game shows 1 stealth kill bar and you decide to try your luck. Don't, in this specific case it is just 1%, not 24%.

Also while studying the code I noticed that Machete (melee weapon) has a chance to kill enemy instantly too, even if it is not stealthy attack. It does not stack with stealth chance, game uses whichever chance is greater. Machete needs to target neck for that and the instakill chance is calculated as 5 + max(0, (attacker's Strength stat - target's Health stat)/2).

Example:
Igor with Strength 98 uses Machete to attack Goon's neck with Health 28, chance to kill instantly is 5 + max( 0, (98-28) / 2) = 5 + max(0, 70/2)=5 + max(0,35) = 5 + 35 = 40, that means 40%

To see the chance of stealth kill in game, just use my mod.
https://psteamcommunity.yuanyoumao.com/sharedfiles/filedetails/?id=3012152757

Buns has 1% vs unaware Crocodile:


IMP character with Stealthy, Assassination and Ambusher has 30% vs the same Crocodile: