960 Pro 1TB x 4 in RAID 0?

[iLLucionist]

So.. what would the performance benefits be if you put 960 Pro 1 TB 4 times in Raid 0 configuration?

I'm not even sure that would be possible on stock Intel boards given the relatively small amount of PCI-e lanes...

[Parak]

Threadripper

[iLLucionist]

Threadripper

mmmm.... looks nice. I have a 950 pro and already I'm thinking about this..

I do a lot of virtualization, and virtualization in virtualization (vagrant = virtualbox in ubuntu in vmware in win10)... and photo editing (which maxes around 6 core in terms of multicore performance).

Unfortunately threadripper does not excel in games..

[Vittra]

None. It's difficult to even perceive speeds differences outside of synthetic benchmarks between high speed SATA III drives and PCI-E based NVME.

When you throw RAID into the mix, things get worse - you gain very little compared to what you actually lose (increased drive wear via writes). Virtualization itself is not a viable reason for this type of setup per se - if you have a heavy I/O (iops) workload, then it could be viable. Of course at that point, you've now got a separate problem where 950 Pros running at full speed for longer periods of time will start to thermally throttle themselves. 960's attempt to address this somewhat with a thin copper layer residing underneath the sticker. If you have active airflow on the drives and/or heatsinks, you'll be much better off.

On top of that, you have insufficient CPU lanes on both Intel and AMD mainstream platforms to accommodate such significant storage configurations. PCI-E/NVME drives getting their lanes from the chipset will inherently experience latency issues over the more direct path of PCI-E lanes traced directly to the cpu, thus slowing down their speed, and worse, they communicate rom the chipset to the cpu via a single PCI-E 3.0 x4 link that is shared with all over devices doing the same. That link is called DMI 3.0 for Intel. AMD used to called it UMI, not sure what the current name is for Ryzen/TR.

[iLLucionist]

None. It's difficult to even perceive speeds differences outside of synthetic benchmarks between high speed SATA III drives and PCI-E based NVME.

When you throw RAID into the mix, things get worse - you gain very little compared to what you actually lose (increased drive wear via writes). Virtualization itself is not a viable reason for this type of setup per se - if you have a heavy I/O (iops) workload, then it could be viable. Of course at that point, you've now got a separate problem where 950 Pros running at full speed for longer periods of time will start to thermally throttle themselves. 960's attempt to address this somewhat with a thin copper layer residing underneath the sticker. If you have active airflow on the drives and/or heatsinks, you'll be much better off.

On top of that, you have insufficient CPU lanes on both Intel and AMD mainstream platforms to accommodate such significant storage configurations. PCI-E/NVME drives getting their lanes from the chipset will inherently experience latency issues over the more direct path of PCI-E lanes traced directly to the cpu, thus slowing down their speed, and worse, they communicate rom the chipset to the cpu via a single PCI-E 3.0 x4 link that is shared with all over devices doing the same. That link is called DMI 3.0 for Intel. AMD used to called it UMI, not sure what the current name is for Ryzen/TR.

That's very insightful, thanks!

The thing I notice with virtualization on my i7 6700K with 4 cores (I know... I should buy a Xeon or something or Threadripper I guess for these kind of workloads), is reduced responsiveness and overall latency, not necessarily UI, but also for all things I-O related, including cpu cycles. Perhaps that is VMWare Workstation not being efficient enough, or Win 10 subsystem not efficient, or just my CPU. But I would like to have virtualization where both OSes perform near identical... switching between them is too cumbersome (win10 for design and ms office, linux for everything else).

[tp4tissue]

Heaven's gate shattered by - iLLucionist's - computing prowess, he drinks champagne from the ample bosoms of angelic females..

[iLLucionist]

Heaven's gate shattered by - iLLucionist's - computing prowess, he drinks champagne from the ample bosoms of angelic females..

And I eat coq au vin. And I drink gin. And cognac. And gin tonic. IT IS THAT TIME.

[SBJ]

Heaven's gate shattered by - iLLucionist's - computing prowess, he drinks champagne from the ample bosoms of angelic females..

And I eat coq au vin. And I drink gin. And cognac. And gin tonic. IT IS THAT TIME.

The amount of classy here exceeds my capacities.

[iLLucionist]

Heaven's gate shattered by - iLLucionist's - computing prowess, he drinks champagne from the ample bosoms of angelic females..

And I eat coq au vin. And I drink gin. And cognac. And gin tonic. IT IS THAT TIME.

The amount of classy here exceeds my capacities.

My dream: have a big classy house, with wood trim panels on the wall, and panel doors. Then have a personal library with cigars, humidor, cognac, chesterfields, wood everywhere. A nice rug on a wooden floor.

And then... BAM: a super powerful computer with 6 monitors, 5 from left to right, 1 above. The sickest build ever, I donno, 10 TB nvme ssd, 4x 1080 ti (because you can), 128 db ddr, 2x threadripper (or xeon, whatever), supersilent. And then play csgo in your bathrobe with a cigar in your mouth and a good cognac sippin.

They should make a movie about this.

[tp4tissue]

Heaven's gate shattered by - iLLucionist's - computing prowess, he drinks champagne from the ample bosoms of angelic females..

And I eat coq au vin. And I drink gin. And cognac. And gin tonic. IT IS THAT TIME.

The amount of classy here exceeds my capacities.

My dream: have a big classy house, with wood trim panels on the wall, and panel doors. Then have a personal library with cigars, humidor, cognac, chesterfields, wood everywhere. A nice rug on a wooden floor.

And then... BAM: a super powerful computer with 6 monitors, 5 from left to right, 1 above. The sickest build ever, I donno, 10 TB nvme ssd, 4x 1080 ti (because you can), 128 db ddr, 2x threadripper (or xeon, whatever), supersilent. And then play csgo in your bathrobe with a cigar in your mouth and a good cognac sippin.

They should make a movie about this.

Real rich people considers computer stuff  [Hired Help] , they regard them with no more respect than their mexican laborers doing the yard work.


By the time you yourself get there, you will understand what I mean.. hahahaha..


It's ok to have passions.. and the times are changing.. but the prejudice is pretty strong from the top down at the moment.


Thinking about it depresses me.

[HotRoderX]

o how technology has faltered. I miss the day's that raiding 2 Maxtor Fireball Sata 150's would increase your performance by leaps and bounds. Our the old AMD Barton's and how they where total power houses!... Though am excited to see AMD finally back in the game after 6+ long long years of Intel Domination!

[Leslieann]

None. It's difficult to even perceive speeds differences outside of synthetic benchmarks between high speed SATA III drives and PCI-E based NVME.

cut

To expand on this...
You get diminishing returns the faster you go, and that is ignoring the fact that the fastest NVME ssd is no longer the bottleneck in modern systems like drives used to be. Even if you had the data throughput, Windows doesn't respond well to raid. You will see it on a benchmark, but really, it just doesn't gain much from it unless you go to insane levels, and you can't because it lacks the throughput and connections. You aren't even going to gain access time because of how fast SSDs are at this.

So where is the bottleneck? This used to be an easy question to answer but today it's more dependent on your system and your intended purpose. If you are a gamer, it's the GPU, a video editor it's the cpu, for the average office user, it's usually a combination of low end components. For Mac users, it's often the os itself.


Another concern is failure rates.
With 4 drives you just drive up your chance of failure by 400%, SSDs are reliable, but raid 0 is only as good as the weakest link, if any single drive fails, all your data is toast because every file spans all 4 drives (except really small files).

[TacticalCoder]

It's difficult to even perceive speeds differences outside of synthetic benchmarks between high speed SATA III drives and PCI-E based NVME.

I agree re-RAID, chances of failure, NVMe no longer being the bottleneck etc. but... A good NVMe is quite faster than the theoretical max speed of Sata 3: no matter how high-end the Sata 3 drive, it's still hitting the max theoretical speed of Sata 3, which is way lower than what a good NVMe drive can do (if I recall correctly). I don't remember the numbers, but when I saw them, I bought a Samsung Pro NVMe SSD instantly and haven't been disappointed.

I'm probably not the "average user" as I don't play any games and my PC is more of a "workstation": Linux system running lots of dev software running, many VMs, etc.  But I clearly remember that when I switched from a Sata 3 SSD to a Samsung Pro M.2 / NVMe SSD is was night and day.

I've never had a machine boot that fast (I don't boot that often as I typically let my PC on with all the tools / VMs open but when I boot, it's nothing like previously) nor feel that fast in every use.  Granted, I do manipulate big files (VMs / containers) and tend to do probably "searches' among hundreds of thousands of files more than most but still: for regular use I've never had a system feel that fast and I attribute it to the switch to NVMe.

So although I don't think Raid-0'ing 4x M.2 SSDs would be worth the gotchas, I still perceived upgrading from a high-end Sata 3 to a fast NVMe as quite an upgrade.

I may be wrong but there's no way I'm ever going back from NVMe to high-end Sata 3 ; )

[tp4tissue]

It's difficult to even perceive speeds differences outside of synthetic benchmarks between high speed SATA III drives and PCI-E based NVME.

I agree re-RAID, chances of failure, NVMe no longer being the bottleneck etc. but... A good NVMe is quite faster than the theoretical max speed of Sata 3: no matter how high-end the Sata 3 drive, it's still hitting the max theoretical speed of Sata 3, which is way lower than what a good NVMe drive can do (if I recall correctly). I don't remember the numbers, but when I saw them, I bought a Samsung Pro NVMe SSD instantly and haven't been disappointed.

I'm probably not the "average user" as I don't play any games and my PC is more of a "workstation": Linux system running lots of dev software running, many VMs, etc.  But I clearly remember that when I switched from a Sata 3 SSD to a Samsung Pro M.2 / NVMe SSD is was night and day.

I've never had a machine boot that fast (I don't boot that often as I typically let my PC on with all the tools / VMs open but when I boot, it's nothing like previously) nor feel that fast in every use.  Granted, I do manipulate big files (VMs / containers) and tend to do probably "searches' among hundreds of thousands of files more than most but still: for regular use I've never had a system feel that fast and I attribute it to the switch to NVMe.

So although I don't think Raid-0'ing 4x M.2 SSDs would be worth the gotchas, I still perceived upgrading from a high-end Sata 3 to a fast NVMe as quite an upgrade.

I may be wrong but there's no way I'm ever going back from NVMe to high-end Sata 3 ; )

I don't disagree that you yourself perceived a difference.

But I've done this test myself between a 850pro 512gb and a 960pro 512gb.

They booted at exactly the same speed and opened  photoshop and solidworks at the same rate as well from start.

I tested this several times from a fresh install on both drives..



A lot of things could've influenced difference you felt.

Data fragmentation
Old software bloating boot time
Windows registry conflicts
Windows update bloats
Other drives taking time away from sata controller
Network loads taking up drive time
OS differences..

[TacticalCoder]

They booted at exactly the same speed and opened  photoshop and solidworks at the same rate as well from start.

But opening Photoshop and Solidworks on Windows is hardly representative of how I use my system!

For example I'm using lots of VMs and containers and "images", which I often manipulate as "big files" that I need to copy around.

I do both regularly copy huge files and search among about 500 K files for "stuff" (not files that are 500 K big: 500 000 files that I need to search things): either just by name of by content.

I've seen test done on YouTube, not by me, showing the Samsung 960 EVO 500 GB (NVMe / M.2) vs the Crucical MX300 525 GB (Sata 3, 530 Mb/s read / 510 Mb/s write) where a big file (8 GB I think) copy was taking 37 seconds on the NVMe and 1 minute 30 seconds for the Sata 3 drive. I can find the link if needed.

That's a 3x difference to copy a big file. I do regularly copy big files. I'm not sure it's only "perception". Same test for gaming gave identical game startup time, so no gain there.

Data fragmentation

No, I'm on Linux: fragmentation typically ain't an issue.

Old software bloating boot time

Nope, similar setups. Super slick init files which I'm always in full control: I hardly ever any launch any service / software at boot.

Other drives taking time away from sata controller

Nope a single drive, nothing on SATA controller, not even CD/DVD readers/burners.

Network loads taking up drive time

Nope, my system boots in exactly the same time whether the network is available or not. Fixed IP address (no DHCP, no nothing).

OS differences...

No, same Linux version on both, same install. I basically have got a "scripted" install procedure which I reproduce on my workstation everytime so I always have basically the system configured the same way.

Basically: it's not because for some or most uses on Windows (like gaming or opening Photoshop) there's no speedup that nobody is seeing speedups in their own use cases...

Maybe I should take the time to buy a high-end Sata 3 SSD and install the exact same system and do a few comparisons. Then we'd see if it's perception or not. But I'm pretty confident that I'd see noticeable differences both when copying huge files around and when doing searches on the entire user(s) directories, which is where I'm "perceiving" the difference.

[tp4tissue]

File copy is a rare speed up event, because you would need two drives in the same system with equivalent speed.

Do you have 2x 960 pros ?

As for copy to self,  that is almost completely drive controller dependent.


As for data fragmentation.. There's this rumor going around how fragmentation doesn't affect SSD etc etc, However this is not true.

As you continue to use your system, EVEN ON  SSDs,  the i/o could be 100s of times more on the same file vs a fresh image.

That i/o could slow down the drive's storage controller significantly, OS has nothing to do with it.

[TacticalCoder]

File copy is a rare speed up event, because you would need two drives in the same system with equivalent speed.

Are you telling me that in a single drive system, a big file copy from/to the same Samsung 960 pro is not going to be faster that the same big file copy from/to a fast Sata 3 SSD?

Because I do have a use for big files copy on the same SDD: actually I do it all the time. In addition to VMs, containers, images (like full Raspberry Pi images), I've also got for example cryptocurrencies blockchains (these are massively big file / files / DB) that I sometimes need to "rollback", backup etc. and restart in an older state from the "backup" file.

"People" may not copy big files but I do, very frequently.

Do you have 2x 960 pros ?

Not in the same system but on my next rig the mobo shall have 2 M.2 NVMe port (Asus Z370-A) so I may, why not.

And I just saw that between two 960 pro on two different systems you can copy them over 10 Gb ethernet at 2x the maximum theoretical Sata 3 speed, which is kinda very cool. There are vids showing just that.

I mean: we're literally talking about copy of big files over the network which are 2x faster than the max Sata 3 theoretical limit.

Overall I really don't buy the "just use Sata 3 because you don't need more", which kinda sounds like the good old "640 KB of memory ought to be enough".

[iLLucionist]

File copy is a rare speed up event, because you would need two drives in the same system with equivalent speed.

Are you telling me that in a single drive system, a big file copy from/to the same Samsung 960 pro is not going to be faster that the same big file copy from/to a fast Sata 3 SSD?

Because I do have a use for big files copy on the same SDD: actually I do it all the time. In addition to VMs, containers, images (like full Raspberry Pi images), I've also got for example cryptocurrencies blockchains (these are massively big file / files / DB) that I sometimes need to "rollback", backup etc. and restart in an older state from the "backup" file.

"People" may not copy big files but I do, very frequently.

Do you have 2x 960 pros ?

Not in the same system but on my next rig the mobo shall have 2 M.2 NVMe port (Asus Z370-A) so I may, why not.

And I just saw that between two 960 pro on two different systems you can copy them over 10 Gb ethernet at 2x the maximum theoretical Sata 3 speed, which is kinda very cool. There are vids showing just that.

I mean: we're literally talking about copy of big files over the network which are 2x faster than the max Sata 3 theoretical limit.

Overall I really don't buy the "just use Sata 3 because you don't need more", which kinda sounds like the good old "640 KB of memory ought to be enough".

Yeah, agreed. I actually push around a lot of files, also like you say, containers, vms, etc. And back-ups. Don't forget production system backups. Those can be huge, depending on the databases.

I want to up my network infrastructure to 10gbps, but my ISP provider only delivers routers with 1gbps. I guess i should go glass..

[TacticalCoder]

I want to up my network infrastructure to 10gbps, but my ISP provider only delivers routers with 1gbps. I guess i should go glass.

I've got "only" 250 Mbps symmetrical fiber here (that's the max here for home/small-office-home-office as far as I know unless you shell out insane amounts) but you can still already upgrade your home / work infrastructure to 10 Gbps 

[tp4tissue]

I got gigabit..

I actually have drives on fiber now.  Buhhhhh that's still only 4GB/s top. because my server is pcie 1.1

10gbe is not useful for home use.