Author Topic: Managing Hard Drive space, files and the cloud.  (Read 1382 times)

0 Members and 1 Guest are viewing this topic.

Offline mrduul

  • Thread Starter
  • Posts: 118
Managing Hard Drive space, files and the cloud.
« on: Thu, 15 June 2017, 18:23:14 »
So there was a time where I was really organized in my early 20s...

I have been so busy with my life and business that my files have gotten sloppy.

Also I edit a lot of 4K Video now and need even more file storage. Thinking about building a 100TB server however even that might not be enough for long term storage of 4K content.

So what is the best solution for:

1. Storing 4K video.
2. Cloud services (Which cloud services are reliable)
3. Software to help organize and navigate thousands of folders and files.
4. Security, best encryption software?
5. For long term storage is magnetic tape still reliable?

Thanks.

Note:
My operating system is installed on my SSD.
They have the M.2 cards now looking into that.
« Last Edit: Thu, 15 June 2017, 18:46:14 by mrduul »

Offline kolec94

  • Posts: 111
Re: Managing Hard Drive space, files and the cloud.
« Reply #1 on: Thu, 15 June 2017, 18:35:20 »
Ok so ill answer a couple that ik
1. Nvme ssd's or ssd's
2. I personally use One drive and after i got through some setup issue it works well


Sent from my LGLS755 using Tapatalk


kbparadise v60 blues

Offline Leslieann

  • * Elevated Elder
  • Posts: 4519
Re: Managing Hard Drive space, files and the cloud.
« Reply #2 on: Fri, 16 June 2017, 01:58:40 »
SSD (and especially NVME) is not going to work for 4k video storagte, the price per GB would destroy you. NVME also brings the additional limitation of port numbers on motherboards (which most top out at only 2 or 3 ports). No one is editing 4k on SSD, at least not outside a major studio.

Cloud storage brings even more problems as even gigabit transfer rates will be an issue for 4k video. You really need 10Gb ethernet to work efficiently with it, which isn’t cheap, and you don't want to know what that will cost you per month for cloud 10gb internet, if even possible to get that speed end to end.

While a NAS or file server would work, if you are doing 4k video editing, you need more than Gigabit. So your options here are lots of drives either in the system, a 10Gb ethernet system (which will cost what your system does) or a Thunderbolt NAS. Another option is to use dual gigabit connections tied together, but I'm not aware of any reasonably priced NAS that have those so you would need to build a sever with that in mind.

Personally, I would buy a big case and a put lots of fast SATA drives shoved in it, hidden out of the way, then use longer cables and usb hubs to put your monitors, keyboard, mouse etc away from the noise or setup a dual gigabit file server. Either way, I would recommend WD Caviar Black drives as they are faster, more reliable, and not much more money than normal drives.


Organization is on you.
Create a folder for all your data, sub divide it. Stick to it religiously.

Security.. meh.. What’s in your videos that needs encryption? It makes it almost impossible to recover in a failure and slows things down.


Tape works for long term storage, however it’s NOT fast, needs quite a bit of manual labor and unless you’re checking it regularly, it’s not the simplest or most reliable system. Most companies who do use tape, also have something else as well and have a person who spends a lot of their time dedicated to maintaining it. Basically, it’s an enterprise solution, not a small business or personal one. I’ve known people who tried to use it and it failed them regularly.


The simplest and reliable system I’ve seen for backups is an online system like Backblaze or Crash Plan (which do encrypt the backup data).  And for the money, they rock, but are slow if something goes wrong, you are either stuck waiting to download a ton (bandwidth caps!) or spending money and time waiting for these services to send you hard drives with the data.

Crash Plan offers a second alternative and that is to backup to another computer, using the same software. You can backup to another system on site, and/or their cloud, and/or to another location (a friend or a computer you own). I would recommend a combination of these, either local and cloud or just a nearby friend or business you trade backups with. This way if things go bad, you can simply bring the system home and restore the data without using your bandwidth and a whole lot less time.

Using this allows you a not only reliable system (Crash Plan will even warn you if it hasn’t backed up in 3 days), but also gives you an off-site backup and if you do multiple places it works as long terms as you aren’t likely to lose all three places at once allowing you to copy it to another source before others fail.
Novelkeys NK65AE w/62g Zilents/39g springs
More
62g Zilents/lubed/Novelkeys 39g springs, HK Gaming Thick PBT caps, Netdot Gen10 magnetic cable, pic
| Filco MJ2 L.E. Vortex Case, Jailhouse Blues, heavily customized
More
Vortex case squared up/blasted finish removed/custom feet/paint/winkey blockoff plate, HID Liberator, stainless steel universal plate, 3d printed adapters, Type C, Netdot Gen10 magnetic cable, foam sound dampened, HK Gaming Thick PBT caps (o-ringed), Cherry Jailhouse Blues w/lubed/clipped Cherry light springs, 40g actuation
| GMMK TKL
More
w/ Kailh Purple Pros/lubed/Novelkeys 39g springs, HK Gaming Thick PBT caps, Netdot Gen10 Magnetic cable
| PF65 3d printed 65% w/LCD and hot swap
More
Box Jades, Interchangeable trim, mini lcd, QMK, underglow, HK Gaming Thick PBT caps, O-rings, Netdot Gen10 magnetic cable, in progress link
| Magicforce 68
More
MF68 pcb, Outemu Blues, in progress
| YMDK75 Jail Housed Gateron Blues
More
J-spacers, YMDK Thick PBT, O-rings, SIP sockets
| KBT Race S L.E.
More
Ergo Clears, custom WASD caps
| Das Pro
More
Costar model with browns
| GH60
More
Cherry Blacks, custom 3d printed case
| Logitech Illumininated | IBM Model M (x2)
Definitive Omron Guide. | 3d printed Keyboard FAQ/Discussion

Offline tp4tissue

  • * Destiny Supporter
  • Posts: 13568
  • Location: Official Geekhack Public Defender..
  • OmniExpert of: Rice, Top-Ramen, Ergodox, n Females
Re: Managing Hard Drive space, files and the cloud.
« Reply #3 on: Fri, 16 June 2017, 12:28:15 »
Is cloud service fast enough to edit from ?


Video bitrate isn't much higher than 100Mbit on prosumer 4k cameras.   


Buhhhh wouldn't latency make it impossible to edit.




I didn't know cloud storage even guarantees a speed,  they're always trying to sell you m0ar gigabytes, but not m0ar Gbps, which is imho far more important.

Offline tp4tissue

  • * Destiny Supporter
  • Posts: 13568
  • Location: Official Geekhack Public Defender..
  • OmniExpert of: Rice, Top-Ramen, Ergodox, n Females
Re: Managing Hard Drive space, files and the cloud.
« Reply #4 on: Fri, 16 June 2017, 12:38:37 »
just tested my --free-- google drive, it downloads at 300mbit, and uploads at 200mbit..   that seeeeems pretty fast.  not sure if it's all file sizes all the time

i tested this with only a 500mb size file..

Offline kolec94

  • Posts: 111
Re: Managing Hard Drive space, files and the cloud.
« Reply #5 on: Fri, 16 June 2017, 14:03:23 »
If ur serious about editing 4k video u need atleast a good ssd with probably 500gb or more in storage

Sent from my LGLS755 using Tapatalk


kbparadise v60 blues

Offline xtrafrood

  • formerly csmertx
  • * Elevated Elder
  • Posts: 2715
  • Location: Gainesville, FL
  • wildling
    • csmertx
Re: Managing Hard Drive space, files and the cloud.
« Reply #6 on: Fri, 16 June 2017, 14:37:03 »
Isn't it possible to use a dual 10GbE network motherboard as a single 20GbE connection? I think I've read something about this or at the very least watched a video on the subject. That and ZFS would be a lot of fun
Chris Schammert

Offline tp4tissue

  • * Destiny Supporter
  • Posts: 13568
  • Location: Official Geekhack Public Defender..
  • OmniExpert of: Rice, Top-Ramen, Ergodox, n Females
Re: Managing Hard Drive space, files and the cloud.
« Reply #7 on: Fri, 16 June 2017, 14:52:03 »
Isn't it possible to use a dual 10GbE network motherboard as a single 20GbE connection? I think I've read something about this or at the very least watched a video on the subject. That and ZFS would be a lot of fun

what would you do with so much speeeeed..


Offline mrduul

  • Thread Starter
  • Posts: 118
Re: Managing Hard Drive space, files and the cloud.
« Reply #8 on: Fri, 16 June 2017, 15:15:55 »
Thanks for the replies.

Leslie I will try Crash Plan service. The encryption is for other things like personal docs which I really shouldn't upload to cloud anyway more for offline encryption.

I will try my go at storing some important files on magnetic tape for long term storage.

Read this article on 4K uncompressed raw files...It's insane the amount of space you need.

"If you choose the option to shoot your project natively in ProRes HQ 1080p (23.976fps) with two BMCCs or with two Blackmagic Pocket cameras for 25 days and you have 2 hours of material each day.

Then you’ll end up with 50 hours of ProRes HQ footage that is 3.78 TB
"


http://www.4kshooters.net/2014/06/25/how-much-hard-disk-space-do-you-need-shooting-4k/

Going to invest $3,000 in hard drives to run in a RAID 5 server array. I need like 500TB to archive my 4K video wtf. holy ****.  :eek:  :p

https://www.amazon.com/Seagate-BarraCuda-3-5-Inch-Internal-ST4000DM005/dp/B01LNJBA50/ref=sr_1_2?ie=UTF8&qid=1497644148&sr=8-2&keywords=4tb+seagate
« Last Edit: Fri, 16 June 2017, 17:26:51 by mrduul »