Yes, thats a great idea, more things to be able to upgrade in your fiefs overall, so villagers will possibly have some guards to protect them, castles better fiancial upgrades and towns the above

Indeed! both of you great ideas!

Well, examples from other mods I've played.

For Towns

1. Caravan Escort Outpost. Basically double size of a caravan. So, usually adds 1-2 fully upgraded units from kingdon(like fully upgraded archer or cav). So, Caravan Master, all regular bells & whistles, & 2 fully upgraded X.
2. Blacksmith. Allows full ammo refills between sieges. But this also could possibly allow your Weaponsmith/Armorsmith to refine your gear.
3. Merchant's Guild increases town Prosperity
4. University can increase town relation
5. Prison Tower
6. Festival Square(Something when you rest there can raise troop morale)
7. Watch Tower
8. Message Tower

Then for Castles

1. Prison Tower
2. Watch Tower
3. Blacksmith
4. Messenger Tower
5. Brewery(Adds relation/morale for resting troops)(just idea)
6. Guard Tower(Allows small patrol to randomly go out patrol around castle, bout 12-14 Cav units, all basic with 1 upgraded.)

Villages
1. School
2. Mill
3. Watch Tower
4. Message Tower
5. Palisade(takes longer to raid a village)
6. Militia Camp. Has trained Peasants you got, max say dozen that can assist you on fights in/near your village.

Yea I'll add buildings for sure, thanks for the comments guys.