You have to use the weapon attachment bench in order to attach and detach attachments. It's a crafting table that you can craft as an Aegi, or if you've obtained the Alliance Crafting Guide if you're playing as some other race. There's also a weapon attachment bench available in the Creon Embassy, near the teleporter store, which is free to use for everyone.

Do note that attachments can only be attached to compatible weapons. The weapon will say "Attachment Compatible" where the special ability is usually listed in the weapon tooltip. Make sure to match the weapon types too: it's a small icon much like the element on elemental weapons. An assault rifle attachment only fits on an assault rifle, for example.