I think it has the same behavior than Humans. When an AI kills a scout of yours, you're angry at him, and adds him to the list of "next races I'll declare war on", which means it worse this race's relation with you. When you do kill a scout of an ennemy, it also worsen your relations...
But note there is a difference between scout and fregates. You can kill any scout without going at war. But if you are about to attack a fregate, you will have a confirmation message warning you it would declare war to the owner.
If you want someone not to attack your scout, you have to make a "Non Agression Pact".
If you don't have any "Non agression pact" with a race, I'm not surprised if he kills my scout, it's not my friend after all...

Last I checked it is not normal human behaviour to attack and kill anything that is clearly unarmed and defenseless even if it's unknown simply because it is there.
Sure there are exception, but as a general rule humans tend to be friendly or at least prefer to avoid confrontation.

Someone else suggested a Non-Aggression pact in another thread as well and as I said there it may sound good on paper, but is not always feasible.
I have yet to get any race to accept that right after I met them as my relations are not good enough yet.

If I was in their territory I could understand it but it's not, it's anywhere and I even had them do it in my very own territory.
Now why would the AI enter a system owned by someone else and blow up any and all unarmed ships, but only unarmed ships and not he colony or military ships or structures unless they are looking to provoke a war?

Sure they are not my friend, but they are not my enemy either and I see no reason to make and enemy of someone I don't know especially when I don't know what that someone is capable of should I try.

As for attacking frigates, I was actually able to attack the fleet of another race without getting any warnings or war declarations, though that could possibly be a bug.

You are allowed to attack enemy fleets of any type without a war declaration if they are in a system you own, one that is unclaimed, or they are on an empty warp point in a system the race you are attacking owns. That's why it's advisable to park your fleets over planets you own or by outposts in systems rather than on empty warp nodes unless you feel confident your fleet strength is enough to deter attacks.

Add.0.0.6.2a: Added a new AI control value to determine whether scout ships will be attacked, DISPOSITION_THRESHOLD_KILL_SCOUTS. This only effects scout ships other civilian ships will be attacked as normal.

:::: 4.7: GlobalsAI.yaml

:::: 4.7.1: AI.T_DiplomacyMajorCiv

DISPOSITION_THRESHOLD_KILL_SCOUTS: -0.25 (Default) or F (9.9 = disable)
This controls whether a major civilization will destroy other race's scout ships. If current disposition is less then ((territoriality - pacifist) / 10) + DISPOSITION_THRESHOLD_KILL_SCOUTS other race's scouts would be attacked

Talking about that, I just got the case for a military fleet (my main fleet actually) in my current game. I was on an Alkari system, and he attacked my fleet without declaring war.
It feels strange. I understand not declaring war when you attack a non military fleet, if it is not in your own system, but attacking a (large) military fleet looks like a war, anywhere it happens...
I'm not saying he shouldn't have attacked me (since I was going to attack him 10-20 turns later anyway, we were rivals...), he nipped me in the bud, that was a good decision. But I would have expected this attack to be an automatic declaration of war... don't you agree ?

No, I totally wouldn't.