Do this:
try running a few ping and tracert tests to the server you are having problems with and also to a known good server (such as one of Google's.) The tests should be run when everything is working okay so you have a control and then again when you are experiencing problems. You need at least 3 samples of each to get a fairly accurate data sample.
Edit: Also, run the same tests as described above to your modem/gateway's IP.
Copy and paste this into notepad, replace the text with IPs where necessary, and save with a .bat extension (assuming you are running Windows.) Then run a few times when the connection is good and a few times WHILE you are having problems.
date /T >>networktest.txt
time /T >>networktest.txt
ping router IP >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
ping gateway/modem IP >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
ping ISP DNS >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
ping -n 10 game server IP >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
nslookup
http://www.google.com >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
nslookup
http://www.yahoo.com >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
tracert game server IP >>networktest.txt
date /T >>networktest.txt
time /T >>networktest.txt
You will need to rename the output files to avoid having them overwritten when running subsequent tests.