1) Delete the current desktop shortcuts and create new ones via the following steps:
- Open the Steam client and go to your "Library"
- Right-click on the installed game
- Select "Manage"
- Select "Add Desktop Shortcut"

If that doesn't work try to:

2) Manually edit the shortcuts' icons:
- Right click on the blank icon and select "Properties"
- Change the icon by clicking the "Change Icon..." button
- Browse to the games' directory and select a similar icon from the installation folder

If that still doesn't work, try to:

3) Re-associate the .url (shortcuts' extension) to a web browser:
- Settings > Apps > Default Apps > Choose default apps by file type
- Scroll down to .url file extension
- Set the default app for .url to "Internet Browser"

EDIT (below): Users seems to report that the guide doesn't work on Win11. I am not currently running Win11 so I can't be 100% sure of the correct steps to fix this issue under that OS. Based on the post from OAT on elevenforum.com webpage named "How to associate .url with Internet Browser?" and from video on YouTube showing the extensions of Edge, the procedure seems to be the following.

4) Win11 fix (URL equivalent):
- Settings > Apps > Default apps
- Scroll down and click on "Microsoft Edge"
- Set HTTP and HTTPS to open with Edge