Striped drives with periodic local backup, am I a dumby?

Computer Hardware & Server Infrastructure Builds
1

Hi,

TLDR - Is it a terrible idea to stripe consumer SSD’s with no redundancy and schedule rsyncs to back it up?

I’m planning on using freenas more like a glorified external drive. I want a fast, medium sized share that I can access from Windows, Linux and Mac OS without fussing over file systems. Use cases include VFX (which involves streaming many very large image files), video editing, light software development. This isn’t for professional use, I’d invest in a better solution if my livelihood was on the line.

So priorities are performance on a tiny budget, ie sacrifice resilience! I’m thinking the following layout:

  • SFP+ direct link to my workstation (which dual boots windows and linux). Later, a Mikrotik 10G switch to add a render box and the mac. 1G connection to the rest of the network.

  • Some number of consumer SATA ssd’s, striped to give me enough bandwidth to saturate the 10g connection.

  • Some spinning rust and a scheduled rsync to back the SSD’s up to it frequently.

  • This in turn backed up to a cloud service less frequently.

I understand that a single failure in my SSD pool will destroy all my data since the last rsync. If I’m OK with that, is this plan sound? Or is there a more sophisticated arrangement that gives me some redundancy and still ~10G speeds without spending significantly more money?

Thanks!

2

Striped is probably the fastest way to go until your budget get’s bigger.

3

Fast plus backup to big might be OK depending on how often you sync, but how many drives are you talking about and what sizes are you looking at? Seems that with a combination of fast and big you might be able to just settle on medium sized and have the redundancy all in one. Then do cloud backups for off site storage. If you have 8 drives, you could run two 4 drive vdev and combine those together in a “stripe”. Then you could lose 1 drive per vdev without losing data and have the speed from the simple stripe. If you are going to use all SSD, then the speed should be decent enough. More drives = better performance but higher cost.

And if you are not making money from this, do you really need to saturate a 10gb connection? Time is money, but when there is no money coming in, decreasing the time will cost a fair bit of expense. I’d like to saturate a 10gb connection in my test lab, but the reality is that there is no need other than curiosity. I have not purchased what I would need to do this, even though I have 10gb optical cards for each VM host and an extra for the storage server. I’d need to upgrade drives in a very big way to get this done, and that starts eclipsing the learning value. I’d also need to upgrade the hardware for the storage, the older server does not have fast enough IO for that type of drive. It would be easy to spend $3000 to $10,000usd just to make the storage happen, and that’s a ton of money for something that doesn’t bring value beyond curiosity.

Summary is that while a 10gb network might be a good way to go, because getting 5gb of saturation might be somewhat economical, I don’t think I would build out the storage with the aim of trying to hit that full bandwidth. Or maybe my pricing is wrong and there are cheaper solutions that I’m overlooking. If there are cheaper solutions, I’d certainly like to know because I might build out my lab for SSD instead of the little spinning drives, currently sporting some massive 250gb drives at SATA 3gbs max split into 2 vdev (not striped) with 1 vdev for VM and the other for HA heartbeat and ISO storage. Certainly not the fastest beast I could create, but lets me learn without costing too much money. If I could replace all 8 drives with decent size and speed SSD for not very much money, I’d do it because I could use the storage for other things where saturating a 1gb connection would be useful (like video editing). Though that said, you can get faster with USB connected drives so maybe not.

This is getting long, but how big are these animation still frames? With lots of small files, the process of transferring the data can really slow things down. There are a lot of things that do not (in my network) saturate even a 1gb connection. Example is that when you extract the installation files for Avid Media Composer and copy them to a client workstation, there are a ton of little icon images, XML files, HTML files, etc. All these little files require the transmission to acknowledge the receipt before proceeding and the file system to place them in storage, this causes the overall transfer speed to be much lower than say transferring a very large compressed ZIP file or video file. Moving video files I commonly see 100MBps to 112MBps across my network, each client is a 1gb connection, that is essentially saturated as far as I’m concerned and probably a limitation of the client SSD. But I’m not sure how big a 4K (or larger) TIFF or EXR image will be so you may want to look at where the bottlenecks might be happening. Yes a 10gb network will make them a little faster, but it might not be what you are expecting. A sequence of HD PNG images would certainly be slower than a big ZIP file.

Finally this is not to discourage you from pursuing your desires, but just some notes on what I see on my network at work. We teach (among other things) video editing and for many years we used shared storage to accomplish this. We are currently on user owned USB drives, but that flops back and forth with the shared storage every few years. My 8 drives and dual 10gb connection seem to be fine for the compression we use on the files, I’ve tested past 40 streams at 35mbps and the server is barely working (4 streams on each of 10 clients). Typically we are only 2 streams for short duration with 1 stream normally working (each). Specifically we are using XDCam EX codec because that’s what our current cameras shoot, and our playback server is good with XDCam so it is an easy export. I need to test DNxHD at 145mbps one of these days and see where things go bad. Should be able to get 2 streams on each client for a decent number of clients, but I won’t speculate on that until I try it. I’m just using SMB shares with Windows clients for this.

1 Like
4

Hello!

Thanks for the detailed response. Roughly in order…

I think about 4 TB of the fast storage would be enough. So I was thinking just 4x 1tb drives striped should get close to saturating the nic (don’t know the real world ratio of performance gain / drive with ZFS, am I right that 4x ~500Mb/s drives should hit the mark or get close?). I know there’s also consideration with cheap SSD’s relying on SLC caches for the published read speeds and so struggling to sustain the speeds. Mostly seems to be with regard to much faster NVME drives though, so hoping an old SATA one will maintain consistent speeds (if anyone can tell me how accurate that is please do!).

Regarding your suggested layout, assuming that would be more like 2x performance? Very probably could be enough for much of what I would use it for. Would it be raid-z2 to get that level of redundancy?

The large files are EXR / DPX for Nuke and Flame. For a straight image, they are usually about 25 MB / frame which would mean less than 5Gb/s to stream 24 / sec. But CG renders with a bunch of 32bit AOV’s can be literally hundreds of MB / frame (although granted there’s other bottlenecks just using those frames for anything). Also, people are starting to ask for 8K / higher frame rates (groan).

And regarding the time vs money, you’re right of course but this is sort of a dry run, Nuke and Flame is my actual profession and more and more people are going it alone. If I bring in a job I would probably load the thing up with drives and bring in proper redundancy but very curious to see how far I can push it with relatively cheap components (amazing that any of this is possible really, I still remember when a “Stone” for Flame cost $100k). In that instance I may need more than one client, also.

If I never bring in a job it will remain as a glorified USB drive!

1 Like
5

Well I did a test with the two sata SSD’s I already own over the 10G connection and it seems two striped is enough to saturate with sequential reads at Q8 on crystal diskmark. Which is great! Q1 is about half that though.

6

If two striped is OK, do you have the budget for 4 drives? Then you could do 2x2 “raid” 10 (mirror and stripe) which would give you a little protection while maintaining at least the 2 drive stripe speed.

I also think you might be able to hit that speed with 4 in a RAIDz (similar to raid5) and may be worth a test if you can afford the drives to do this. At least you could lose 1 drive and continue working.

Someone I went to high school with makes a good living with Nuke, Flame, etc. I forget all the clients he has worked with, but Discovery is one of them. Mostly he’s a company owner now so only selected clients get his work, employees get the rest. I think they are mostly Media Composer for editing and he went to Nuke and Flame when Avid stopped doing anything with the old Symphony software.

I’d also suggest Davinci Resolve, if you buy a license you now get the full Fusion and Fairlight packages, you still need to install them separately, but you get full features. The bits included in Resolve are slowly migrating to fully integrated, but it takes a bit of time. I bought a dongle a couple years ago, but I think the online activation allows for all of these full applications now. Also be aware of cheap dongles, their is a big counterfeit market for them. Real dongles support all versions going forward or back (so far). $300usd for the dongle from any real reseller. Look for discounted (real reseller) Fusion dongles, they will also activate the other applications, lots of details on Blackmagic’s website.

And I should mention the Blackmagic has their official training materials on their website for free, certification exams are also free. You can also buy those books on Amazon if you want versions on paper. Loads of videos, and the training media are also in the training section of their website.