Hello! It is possible you may be running into the engine limitations. Homeworld's engine unfortunately is hardcapped at around 2~2.5GB RAM, which means that even if you have a good pc you may start to encounter graphical issues and other problems as you approach that limit.

The FXmod adds 4 new races compared to the base game, along with various new models and fx, so on (very) large maps with lots of different races and/or high unit caps you may more quickly start to reach the limit of what the game can handle.

I would also recommend reading the "Graphic Issues" section in this link, and see if any of these help:
https://psteamcommunity.yuanyoumao.com/app/244160/discussions/1/368542844489248718/