I always start troubleshooting issues like this by making sure the platform is solid first. I hate spending a bunch of time only to find out it was bad ram...
So start off by running memtest for a few passes to make sure you have a stable machine. If that passes, give us the specifics on your hardware, like CPU, and nic, and how much memory.
Note that I have not mentioned the USB stick at all. That is because once you boot, everything is in memory and the stick is irrelevant.