Another wall of text by Leslie.
So I finally got around to setting up my new 2.5gbit network.
TLDR
What should have been a simply upgrade turned into a 2 year project. I got 250-270MB in both directions but had to ditch Windows.
The beginning
This actually started the Black Friday before last, yes, it took nearly 2 years to do this!
Prices had finally come down on 2.5Gbit and I had found a 2.5Gbit card for SUPER CHEAP so I grabbed it, and while that fits my file server, my desktop lacks any spare slots as both are mini-ITX and the desktop has a GPU. So instead of buying another 2.5Gbit card I had to find an alternative, and frankly, I got frustrated and just gave up on it. Too little information and too many band aids to the problem were needed and it just wasn't a big deal (something you need to keep in mind with this sort of thing). The recent Internet speed thread by TP4tissue got me to go and just finish out the part I needed, but even with more time for things to settle it wasn't so simple, in fact some of it became more complicated due to time and things that have happened since.
As for a switch or router, that was easy, there would be none.
An old networking trick is you run a normal network, gigabit in this case, and then run a direct line between the two systems that need speed, in this case 2.l5Gbit between the desktop and the server. This sort of network has been around for decades and has many uses. Just setup a normal network, then on the direct link you assign an IP (different from the normal network) but no DNS or gateway. Easy, or at least it used to be.
First problem was the lack of slots in the desktop,
If you lack slots adding a card means getting creative, the good thing is these days we have better options, they're just not always good ones. The most simple is USB, but USB is going to create a bottleneck, and in some cases, it's significant. The other option is to use the wifi/Bluetooth card slot if you have one to spar (I do not), and use that slot with a mini PCIE to PCIE adapter, this seems good as it's just a PCIE slot, the bad part is that it means finding somewhere to mount the network card. These adapters are not always small but they do allow remote mounting of the network card. If you mount it externally you need a box and a way to get the fragile cable out of the case, internally I have no way to mount it securely, heck, good luck finding space for it in most mini-ITX cases. I would prefer nothing external, let alone some fragile cable connecting to an external box but at least I would get full speed. Another option is to use the second NVME slot, you can use an adapter to make it a conventional PCIE slot and then use the above pcie adapter. While it can work, the NVME slot on the back of my board would have no clearance in my case and even if I could fit it, there's again the problem of mounting the network card.
There are some other issues here to contend with as well, using the second NVME slot on most boards divides the NVME PCIE bus lane, I.E. half speed, so your NVME drive may be cut down some. Honestly, this isn't the huge cut you may think as those numbers are theoretical and even benchmarks do not reflect real world. Odds are you wouldn't notice the speed drop. I have dual drives and while I can tell once in a great while, it's pretty rare. Unfortunately that also meant no slot to spare.
As for USB there's problems here too,
While USB 3.0 will allow 5Gbit the overhead for it is 20% compared to USB 3.1 needing only 3%. When sending data to a drive it's one thing, but networking has far more overhead, so that 20% becomes more like 35%. In real world terms this means:
USB 2.0 is half duplex, leading to gigabit speeds, don't even bother.
USB 3.0 with a 2.5Gbit card is going to have an average high of around 150-175MB
USB 3.1 with a 2.5Gbit card is going to have an average high of around 220-250MB.
Then there's power, these are extremely power hungry and not all ports, especially type-A can power them well enough for stability. You may need a hub (especially on type-A) and if you do, you should be aware it may make the hub useless for anything else under heavy transfer due to bandwidth constraints.
A note about 5Gbit over USB
While you can go 5Gbit it costs more and there's fewer options but with a peak of only around 3.5Gbit you're getting diminishing returns on it and unless you're running raid or SSDs with a powerful processor at both ends you will struggle to flood that anyhow. An Atom, Celeron or 2nd Gen I5 probably won't cut it. The 5Gbit adapters are even more prone to overheating than the 2.5Gbit, which are known to do so on occasion (you want a Realtek chip) and power is an even bigger problem. My advice, if you're going to spend the money on 5gbit, skip USB, 2.5Gbit and 5Gbit and just go buy used enterprise 10Gbit cards, it's going to cost the same more or less, offer more options and be more future proof.
In the end I went with a Type C 2.5Gbit Realtek based adapter.
Finally, I can set this up!
So like I said, there was to be no switch or hub, just a direct line bypassing my main network. While a switch makes it plug and play they're not cheap, in fact during the Black Friday sale when I got my card they tended to sell for around $140-170 for a 4 port, which is just insane, you could get a used 10Gbit switch for not much more, though it would be power hungry and loud. Also, before you buy any 10Gbit switch, make sure they support 2.5Gbit, I'm not sure they all do, and if not those that do are likely newer and going to be more expensive. Luckily, you can now find 8 port 2.5Gbit switches for around $100 but I only needed to connect two systems and I knew a way around that. Or so I thought, imagine the theme from Jaws playing here because sh*ts about to go down.
The server...
This should be simple, you set a static IP, no dns or gateway on both ends, reference the share based on the IP and it simply works. Done it many times. Easy right? Well it used to be. I popped the numbers in Linux and it was ready. Windows, not so much... Win10 didn't like setting a standard subnet mask, turns out, depending on who you ask, MS either screwed up netmask or switched to an enterprise method (use 24 instead of 255.255.255.0). Fine, whatever once I figured that out, easy right? Nope, Win10 now refuses to take a static IP address. MS decided we no longer needed it and apparently deprecated it. This explains why last time I was trying to set a static IP I gave up and just added it to the router. I never bothered to look into why it failed, I just worked around it and moved on. Seriously MS, wtf.
So at this point my options are buy a switch/router, roll back to Win 7 or 8 or move to some form of Unix (BSD) or, Linux.
Win7 software is becoming an issue, forget drivers, have you tried installing Direct X, Dot Net or any of the other requirements by hand lately, it's kind of a mess. Things may last forever online but the bility to find it and working links do not. Win8 is simply dead, GPU companies (and more) have actually abandoned it entirely. So you may find a network driver for Windows 8 but no GPU driver and you may find a GPU driver for Win7 bit no network driver, and good luck with the rest. It's doable, just not practical anymore and getting more difficult by the day and if I'm going that far, why not just go all the way.
I've been wanting to ditch Windows on my server anyhow but boy howdy is that a lot of work but I also like a challenge. I've used several NAS and file server software in the past, they all had one or two things I disliked. Three things I wanted if I was switching was Virtual Machine hosting, that would be nice, however the last NAS software I tried and liked had deprecated that in favor of containers (good job!), containers are great, but you can't run a Windows container on Linux. Another issue is my cloud storage, I use Mega and Google, Google promised a Linux client since debut but never delivered. There are replacement clients, but the best one, while not free it's not expensive, but wow is their sales team aggressive and I refuse to support companies like that. It took 2 years for them to stop messaging me after a trial. I don't care how good it is, you aren't getting my money. The last thing was remote access, I have long used Teamviewer for this but I was willing to deal with that later as there's more options there and it was less important.
I ended up with Truenas, which actually works better than I expected. I'd prefer something Linux based (BSD Unix vs Linux) but this runs well and surprisingly supported the 2.5Gbit card out of the box which was a concern as BSD is not great at hardware support. It supports remote Virtual Machine hosting and a lot more. The last issue was Google drive but at this point I was angry enough with MS I was willing to ditch Google Drive if it meant getting rid of them. While looking into setting up Mega I found Truenas actually supports almost all cloud storage right out of the box, maybe not as smooth, it has to run on a schedule not on file change, but that's much better than I expected.
So we're up!
Almost. Sending to the server was (really) fast but receiving was gigabit speed, considering the server has a spinner and the desktop has the NVME SSD it should be the other way around. Odd. Resources were high and ping was also astronomical but I attributed some of that to it being USB.
I was feeding my USB backup drive into my desktop then through the USB 2.5 to the server, so I figured maybe it was flooding out the USB. Watching the data rates seemed to confirm this as it was reading then sending, reading then sending, not a constant flow. Different cable? Nope. Maybe it needed Thunderbolt enabled in bios, nope. In my digging I did find it was in half duplex mode which explains the cyclical read/send, older USB was half duplex but this was 3.1 with full duplex, this was what led me to find that there's two drivers for 2.5Gbit and one (the default) is not so good, especially for the Realtek based ones. Like most things I.T. there's an easy fix, once you find it.
Finally!
Pings were now actually better than the network (there's no switch), transfers are now averaging peaking at around 250MB (Bytes not bits!), with some managed to hit 270MB. A 3.5GB file sends in about 14 seconds compared to 45 seconds, a massive bump in speed. As for CPU overhead, a concern for me, I'm shocked, it's actually on par with the built in gigabit. I can't express how impressed I am for such a cheap no name adapter.
Was it worth it? Is it worth it?
Yes and no. I only spent $40 on this whole project so it was cheap, but what I saved in money it cost me in time. I had to redo my server, ditch Windows and that meant backing up and reinstalling everything. All told I spent days researching this when it really should have been a drop in had I just used a switch. I also spent an entire weekend switching out my server software. I didn't mind ditching Windows on it, I had been wanting to be rid of that for a while and the time spent researching USB network adapters had to happen anyhow. Plus, I like to experiment so it wasn't a waste. Note that I do eventually intend to get a 2.5gbit switch, just not right now.
But was the faster speeds worth it?
Well I doubled my network speed, I can even run networked drives as if they are installed in my desktop and while I do a lot over my network 99.99% the time this upgrade is completely useless and that's why it took so long and why I refused to spend a lot. It really has almost zero impact in my life or computing and I'm the person it's geared towards. It's nice, I'd totally do it again but I wouldn't run out and buy a bunch of new networking gear to accomplish it.
Parts for those curious:
Random cat5e cable, Cat5e is rated for 100meters @ 2.5gbit.
Inateck 2.5G USB model ET1001, about $18 on Amazon. These known to have the Realtek chip and not overheat.
2.5Gbit Rosewill RC-20001 PCIE card, $20 at the time, currently sold out. Expect to pay $40-$60 for similar today.
