Anyone using either infiniband or 10Gbps Ethernet at home?
- Thread starter NeverDie
- Start date
pete_c
Guru
Here I purchased Intel Pro 2 and 4 port server GB NICs for my PFSense box. They work fine. (historically always used Intel for added NICs on my other boxes - well vm boxes).
Currently still running with 6 GB NICs on the box and haven't gotten around to updating the NICs from 2 port to 4 port to give me 10 Gb ports on the PFSense box.
Initially purchased them on Ebay as refurbs. Just noticed there are 4 port Intel NICs now new cheap (well ~$60 USD).
BTW here is some interesting reading stuff brought to you by a Serve the Home forum post from 2011.
10-Gigabit Ethernet (10GbE) Networking - NICs, Switches, etc.
Prices for refurbs have dropped even more now.
A snippet from OP above (from 2011).
Currently still running with 6 GB NICs on the box and haven't gotten around to updating the NICs from 2 port to 4 port to give me 10 Gb ports on the PFSense box.
Initially purchased them on Ebay as refurbs. Just noticed there are 4 port Intel NICs now new cheap (well ~$60 USD).
BTW here is some interesting reading stuff brought to you by a Serve the Home forum post from 2011.
10-Gigabit Ethernet (10GbE) Networking - NICs, Switches, etc.
Prices for refurbs have dropped even more now.
A snippet from OP above (from 2011).
NeverDie
Senior Member
Possibly my facts or my math is wrong, but my back-of-the-envelope calculation to get a frame-of-reference was this:
Is that about right, or does my reasoning contain a fatal error?
Side-note: I hadn't thought about it previously, but if the 4 Gigabits per second number (above) is correct, it would seem that putting a Samsung 850 Evo on a Sata II rather than a Sata III would be a terrible waste!
- A Samsung 850 Evo has a best-case read/write speed of around 500Megabytes/sec.
- Equals about 4 Gigabits per second.
- Assume 50% overhead for ethernet.
- Then, about 8 gigabits per second is needed to read/write over ethernet before more speed doesn't matter.
Is that about right, or does my reasoning contain a fatal error?
Side-note: I hadn't thought about it previously, but if the 4 Gigabits per second number (above) is correct, it would seem that putting a Samsung 850 Evo on a Sata II rather than a Sata III would be a terrible waste!
There are many potential bottlenecks to performance.
Samsung quotes 540 MB/s for sequential reads. But it's hard to guarantee that a file of any size will occupy sequential blocks. A better number to base performance on is random reads. There, they quote up to 98,000 4KB ops per second for a queue depth of 32 (~400 MB/s) and as little as 10,000 ops per second for a queue depth of 1 (~40 MB/s). So to figure out what you will really get, you need more information about how the hardware connected to the disk and the software driving it will handle a file read. The number of ops that get queued up at the disk may depend on how much buffer space the system allocates and how quickly the buffers get emptied as the data is moved to its next destination (often another buffer).
Other potential bottlenecks:
1. The SATA adapter connected to the disk: what are its bandwidth and IOPS limitations? What is its data path width to the rest of the system (e.g. PCI Express gen1, 2, 3; x1, x4, x16) PCIe gen2 x1 will give you 500 MB/s, less some overhead.
2. Similar questions for the ethernet adapter.
3. Processor performance limitations. e.g. memory bandwidth, processor speed, number of cores, etc.
4. Software stack performance.
Many times, the only way to figure out what you can actually get is to configure a system and measure it. I've worked with many systems where all the specs looked good on paper, but it took a great deal of work to get the performance to even come close to what the individual specs led you to expect it should be.
Samsung quotes 540 MB/s for sequential reads. But it's hard to guarantee that a file of any size will occupy sequential blocks. A better number to base performance on is random reads. There, they quote up to 98,000 4KB ops per second for a queue depth of 32 (~400 MB/s) and as little as 10,000 ops per second for a queue depth of 1 (~40 MB/s). So to figure out what you will really get, you need more information about how the hardware connected to the disk and the software driving it will handle a file read. The number of ops that get queued up at the disk may depend on how much buffer space the system allocates and how quickly the buffers get emptied as the data is moved to its next destination (often another buffer).
Other potential bottlenecks:
1. The SATA adapter connected to the disk: what are its bandwidth and IOPS limitations? What is its data path width to the rest of the system (e.g. PCI Express gen1, 2, 3; x1, x4, x16) PCIe gen2 x1 will give you 500 MB/s, less some overhead.
2. Similar questions for the ethernet adapter.
3. Processor performance limitations. e.g. memory bandwidth, processor speed, number of cores, etc.
4. Software stack performance.
Many times, the only way to figure out what you can actually get is to configure a system and measure it. I've worked with many systems where all the specs looked good on paper, but it took a great deal of work to get the performance to even come close to what the individual specs led you to expect it should be.
NeverDie
Senior Member
RAL said:
Thanks for enumerating some of the variables that might affect the result.
I tried to give an upperbound (best-case) number, which you correctly point out may not be achieved. Agreed. I'm left wondering, though, what might be achievable. What's the highest lower-bound number? I'm not sure how to answer that one. Something more than 1Gbps (as I've already established that number), but a lot more, or just a little more? Roughly how much? Without at least some framework (such as an upper-bound and a lower-bound), it's hard to know whether it's worth the cost and effort to try building it. I'm guessing maybe someone here has already tried it through aggregation (above), and if so it would be great to have that datapoint.
It would be interesting to have a home SAN. I have no urgent need, and I'm not sure how practical it would be, but it might be interesting, even if it were just at a toy scale, to, say, quickly migrate VM's from one platform to another while they're still actively running.
I'm probably about 3 months away from having my 10Gb network setup so will post some results then.
I'll be using a Mikrotik CRS226-24G-2S+ which has 2 SFP+ modules in it for 10GB over fibre. The trick I found with this device to get the highest throughput is to ensure the ports are configured into the switch group, else the performance is pitiful. I don't have the NIC yet for the server so I'm still researching that piece. Sorry I can't be help now, but I'll post back when I've got some results to share.
I'll be using a Mikrotik CRS226-24G-2S+ which has 2 SFP+ modules in it for 10GB over fibre. The trick I found with this device to get the highest throughput is to ensure the ports are configured into the switch group, else the performance is pitiful. I don't have the NIC yet for the server so I'm still researching that piece. Sorry I can't be help now, but I'll post back when I've got some results to share.
NeverDie said:
My rule of thumb from when I used to work on this sort of stuff was, for an initial estimate, to take the best case bandwidth I thought I could reasonably expect from the weakest component, and use 50% of that for what could be achieved. Sounds harsh, but it's based on many years of experience working with ultra high performance file systems. Sometimes the end results were better than that, but more often they weren't.
Frequently, right out of the box with everything running using default configurations, we'd get only 10% of our target. Then, with lots of tuning, we inched our way up.
In general, you want to use the largest units of data transfer you can. PCIe adapters often default to 64 or 128 byte packet sizes on the PCIe interface. That increases overhead and greatly reduces performance. Some adapters (and systems) have no ability to increase the packet size, but the better ones do.
On ethernet, use the largest frame size that all of your hardware will support. Default is often the minimum of 64 bytes.
NeverDie said:
That's going to require more research on your part to figure out.
You need to look at the SATA adapter, the system it plugs into, the ethernet adapter in that system, the ethernet switch, the ethernet adapter on the other end, and the system that it plugs into.
It's a lot of stuff to get your head around. Many card and system vendors don't publish the data you really need to know, but if you can find out what chip sets they use, you might be able to find some data from the chip set vendor.
That's partly why I said that it's often easier to configure the system and measure it. But then, whatever you do measure, you don't really know if it could be doing better.
NeverDie
Senior Member
Not sure this is entirely relevant, but sometime after July, Google Fiber is supposed to be available to my neighborhood. $70/month for 1Gbps internet access. It's already deploying elsewhere in town.
I stopwatch measured the throughput I'm getting on my Cat6: the Windows 10 technical preview takes 40 seconds to transfer from one computer to another. That's about 820Mbps of actual throughput between two computers, going through the router. Maybe the router is slowing it down, or maybe it's Windows, or maybe it's the SSD's on either end or their drivers, or who knows. I've read that with typical ethernet frame sizes, there might be around 6% overhead (not 50% as I had supposed). With jumbo frame size, overhead could in theory be reduced to <1%. So, that leaves about another 10% or so unaccounted for. I'll need to find some better tools than a stopwatch and a big file. Any favorites for that?
I stopwatch measured the throughput I'm getting on my Cat6: the Windows 10 technical preview takes 40 seconds to transfer from one computer to another. That's about 820Mbps of actual throughput between two computers, going through the router. Maybe the router is slowing it down, or maybe it's Windows, or maybe it's the SSD's on either end or their drivers, or who knows. I've read that with typical ethernet frame sizes, there might be around 6% overhead (not 50% as I had supposed). With jumbo frame size, overhead could in theory be reduced to <1%. So, that leaves about another 10% or so unaccounted for. I'll need to find some better tools than a stopwatch and a big file. Any favorites for that?
wuench
Senior Member
You will never get full 1Gbps We usually recommend an upgrade when we see sustained peaks of 75% or greater. 82% is pretty close to saturated for a windows file transfer using CIFS (like I said initially it is actually exceeding expectations).
Jumbos is an easy thing to try but like I said before CIFS will hold you back being a TCP based protocol (it is the most likely culprit), not to mention Windows and VMWare being involved nether of which are efficient at using network bandwidth. Windows is an interactive OS so it is tuned for that not file transfers, you can probably find some info online on how to tune it and VMWare and other virtualized environments are forced to do a lot of things in CPU that are normally done on the NIC hardware because they have to be shared. This is why VMWare has a hardware compatibility list, they actually have their own custom drivers written for them by the likes of F5 to optimize communications so that you can run virtualized network equipment on their platform.
So you can do a lot more work trying to tune the windows stack, turn off unnecessary services/apps that are also communicating, turn on jumbos, switch to a UDP protocol like NFS and tuning it, making sure you are using VMWare certified hardware and/or committing dedicated hardware resources or NICs to the VM, etc. But you aren't going to get much more out of that link after all of that work.
Jumbos is an easy thing to try but like I said before CIFS will hold you back being a TCP based protocol (it is the most likely culprit), not to mention Windows and VMWare being involved nether of which are efficient at using network bandwidth. Windows is an interactive OS so it is tuned for that not file transfers, you can probably find some info online on how to tune it and VMWare and other virtualized environments are forced to do a lot of things in CPU that are normally done on the NIC hardware because they have to be shared. This is why VMWare has a hardware compatibility list, they actually have their own custom drivers written for them by the likes of F5 to optimize communications so that you can run virtualized network equipment on their platform.
So you can do a lot more work trying to tune the windows stack, turn off unnecessary services/apps that are also communicating, turn on jumbos, switch to a UDP protocol like NFS and tuning it, making sure you are using VMWare certified hardware and/or committing dedicated hardware resources or NICs to the VM, etc. But you aren't going to get much more out of that link after all of that work.
Similar threads
