Tell me about building a RAID

[mtbakerskier]

I need some way of storing a shit load of very large (think 150mb plus tiff files). I am already backing everything up to an external drive and dvd's but I am running out of space fast. What do I need to build to seperate raid arrays, and wich type of raid is best?

[Free Range Lobster]

Mass storage would be the best way.

http://en.wikipedia.org/wiki/RAID

kinda technical but you can get the general idea

[Nobody Famous]

RAID is not too well suited for home use yet. And 150mb is not that much data. Sams Club and Cotsco both have 400 Mb external drives for ~ $175. That's cheap.

If you want RAID 1 or > with that capacity, plan to spend $ 800 to 1200. Plus you may need a rack and power supply, and the fans will make noise when they're powered on.

[sfotex]

You'd typically use RAID for redundency and speed - however you still probably want to have some offsite storage to guard against fire, theft, etc. esp. if you're putting your livliehood on there (photos, etc.) At the minimum you probably want a disk controller that supports raid 5, and a couple of drives. Make sure you set up for redundency if you need it vs. striping across 2 drives for pure speed, unless this is all you're after.

[Free Range Lobster]

RAID is not too well suited for home use yet. And 150mb is not that much data. Sams Club and Cotsco both have 400 Mb external drives for ~ $175. That's cheap.

If you want RAID 1 or > with that capacity, plan to spend $ 800 to 1200. Plus you may need a rack and power supply, and the fans will make noise when they're powered on.

RAID 1+0

massive but backed up.

honestly if you wanna go serious on this you need to build yourself a file server with approx 500gigs

its going to be expensive but you're going to be mirroring everything.

i'd also look into large tape backups that you can ship to an offsite storage facility if you're worried about having sure backups + data integrity.

[Storm11]

Mtbakerskier,
I work as an assistant photog and just set up a RAID system for my boss. Here's the low down from a photographers standpoint...
If you want the most storage for your buck, go with a RAID 5 set up
If you want safety of files, go with a RAID 10 (or RAID 1+0, same thing)

Now, here's where it get's kinda tricky. Depending on what kind of OS system and computer you're using (We're on OSX 10.4.6 with a G5 quad) there may be limited options (especially if you have a new G5) for SATA RAID driver card support. You want to try and find a true hardwear RAID card for the best in speed, but they don't really exist for the new G5's yet. We had to set up a software RAID on ours, but we are running RAID 10 with 2000 GB (2 Terrabytes) of storage. We went with a multi-harddrive enclosure from www.macgurus.com
The guys there are super helpful and know their shit. I would suggest giving them a call...
Also, stay away from a company called RocketRAID (they are cheap POS that don't work like they are supposed to).
Feel free to PM me and I'll give you more detail that I am just too lazy to put down right now.

P.S. I would also recomend backing everything up to DVD (or wait until Blu-Ray discs come out sometime later this year) as well and then storing them offsite (i.e. your house, or if you work from home, a safety deposit box. Better yet, send them somewhere far away (there are companies who store offsite goods for you) in case of natural disasters.

[Cirquerider]

I have RAID 0 and RAID 1 arrays using 4-250 GB disks. But for extra security, I also have a 300 GB external backup that I can keep offsite in case anything happens.

RAID 0 is a mirror array. Two disks contain identical information so if one fails, you can restore all data from the survivor. I use this for my essential work and operating system files. 2 disks with 250 GB each = 250 GB total storage.

RAID 1 is a striped array. The data is written and read from two disks simultaneously which makes it very fast. I use this mainly for video capture and rendering. 2 disks with 250 GB each = 500 GB total.

You need a computer with a motherboard with RAID capabilty. Mine is an ASUS P4C800 Deluxe, and is a bit over a year old, with SATA drives on a Intel RAID and Promise Raid array. Raid arrays can be built off card based controllers, but they won't be as fast. This shit is expensive. The safest and least expensive option for you is the external hard drive. If you have questions though, I will be glad to prove my geekness by trying to anwer.

[mtbakerskier]

I should clarify. All of my images are 150mb or larger Tiff files. and I have a lot of them. I.E. one third of this years images alone will not fit on a 300gb external drive, and I have 5 years worth of images that need to be able to acces quickly. So what I am looking at doing in additon to the dvd backups that I am already doind is building two seperate raid arrays. Each would mirrior the other one, so I would have redudant backups. What is my best config for doing this?

I am currenty using a 20in G5 Imac not intel and a 17 powerbook. Both computers need to beable to acces the raid system.

[Snow Dog]

It sounds like you're after NAS -- Network Attached Storage, not RAID. RAID is about fault tolerant storage but you want a separate backup too.

[Storm11]

I am currenty using a 20in G5 Imac not intel and a 17 powerbook. Both computers need to beable to acces the raid system.

Well, here's the problem. In order to see the increased speed from a RAID, you really have to have a SATA array, not USB 2.0 or Firewire. In order to get that, you need to have an available PCI slot. I don't know whether or not you can add a PCI card into an iMac. Ontop of that, you need to have a PCI slot that isn't PCI-Express (you need PCI-X, etc.) in order to get a RAID card which will run natively on a Mac as a hardware RAID. If you don't have an empty, or PCI-X slot, you're out of luck. Sounds like a NAS is the way to go for you in that case. (Which is connected via ethernet connection)

[Cirquerider]

You have professional data storage needs that differ from mine. The solution might be Network Attached Storage in terabyte plus quantities. Check this out: http://www.anthologysolutions.com/products/index.htm You will spend about $1000 per terabyte, but the capabilities and protection are impressive.

BTW, I'm a camera JONG, and I'm watching the Canon V NIkon thread with interest. Nice posts.

[mtbakerskier]

Ok, once I set up a storage system such as a raid or a net work mass storage device can it grow as my database grows? Can I just ad more drives to increase the storage or will I have to re-format everything each time???

I thinking it would be best to instal Aperature on both computers, and then set up the master library on the first storage device and then use the second just for backup. That way, both computers will have the same acces and ability to edit the libary and still beable to back everything up. I would just have to make sure that only one computer is accesing the libary at any given time wich shouldnt be a problem.

Man it looks like this is going to be fucking expensive.

[Storm11]

With a RAID, you cannot just simply add more storage.... You can, of course, create another RAID from 2-3+ disks if you want, but a single RAID is unscalable unless you transfer everything to separate disks, format the RAID and then re-transfer....In other words: A major pain in the ass.

I'm not too sure about NAS however...

[RShea]

Raid is a group of drives used for either increased speed or increased protection (redunancy) or both. You have multiple levels of raid as talked about above with numbers assigned to them Raid 0 and 1 are the basic first level. They need at least 2 drives only- with Raid 0 using both drives as 1 big drive for increased capacity. The disadvantage of Raid 0 is if 1 drive goes out then all the data is lost on the other drive (unless you can do high level data recovery or send the drive out) so it is not recommended for important data- usually used for faster access in software and operating system which can be recovered or reloaded. So 2 400 gig hard drives look like 1 big 800 gig drive.

With raid 1 you use 2 drives and mirror them- everything written to the first is also written to the 2nd- you do not gain large storage, as a matter of a fact you loose capacity but gain redunancy if 1 drive fails, the data should still be on the 2nd drive still working. With 2 400 gig hard drives, you still only get 400 gigs of storage available.

Next most popular level is raid 5- it requires at least 3 drives or more. It has some higher overhead because it writes data to the drives and also writes the error checking info. If a single drive is lost then the system continues to run and you do not loose any data due to the error checking info. You replace the drive and the info is recovered automatically. It is both redunant and increases the capacity since you can use larger drives and the data does not all rest on a single hard disk. Raid 5 can be increased in capacity by adding more drives (assuming the controller and case can handle more than 3 or 4 drives inside). SCSI and SATA or PATA IDE are the 3 drive styles popular. If high capacity is needed look at either SATA or PATA IDE as they are more cost effective and higher capacity than the SCSI drives out there. If speed and performance is an issue- then SCSI is the way to go, but it is pricey compared to the IDE type drives out there.

Raid 10 is a very high end solution for critical data- it offers the best of raid 0 (higher capcity striping of data across drives) with duplication and reduncancy of the info in raid 1.

There are a few other raid types out there but for your application you have to ask yourself how important is the data (redunancy and back-up of the data to other sources) how much do I need access to (versus having some stored off line- on tape, DVD, or other removable media) and what is my budget for this(to do this right it may not be cheap at all)? Once you have these thing sorted out then the plan can be easier to come up with- Raid 5 or Raid 10 etc. would be the most likely choice if reduncancy and larger capacity than a single drive are both needed.

[RShea]

Ok, once I set up a storage system such as a raid or a net work mass storage device can it grow as my database grows? Can I just ad more drives to increase the storage or will I have to re-format everything each time???

I thinking it would be best to instal Aperature on both computers, and then set up the master library on the first storage device and then use the second just for backup. That way, both computers will have the same acces and ability to edit the libary and still beable to back everything up. I would just have to make sure that only one computer is accesing the libary at any given time wich shouldnt be a problem.

Man it looks like this is going to be fucking expensive.

You have to look at the controllers and the management software to answer this. There are a few controllers that will allow you to add more drives to the Raid 5 and expand the overall capacity, but for the most part the lower end solutions do not offer this feature.

See this link for talk of physical vs logical expansion:
http://www.pcguide.com/ref/hdd/perf/...pansion-c.html
Example of Physical expansion would be replacing the 300 gig hard drives in an array with newer 400 gig hard drives (it can't expand the extra 100 gigs automatically with every solution- youy have to reload the data in MOST cases with the common controllers out there. Or some allow you to take the extra 100 gigs and make another logical drive (you end up with a split hard drive with close to the full capacity but not in 1 single drive letter).

Logical expansion is taking say 3 300 gig hard drives in a raid 5 and adding a 4th 300 gig hard drive to the array and having the extra drive included to expand the raid 5 to the hgher capacity.

[RShea]

With a RAID, you cannot just simply add more storage.... You can, of course, create another RAID from 2-3+ disks if you want, but a single RAID is unscalable unless you transfer everything to separate disks, format the RAID and then re-transfer....In other words: A major pain in the ass.

I'm not too sure about NAS however...

With MOST raid you can't just add more storage. SOME raid controllers do allow this, they are the higher end and more expansive versions but they are out there (depending on the type of raid used of course).

[RShea]

You may be wantling to look at something similar to a product like this:

http://www.digistor.com.au/products....t=10&subcat=22

(they are an Ausie company but there should be something like this available in the US. It just happened to be the first one I found googling for expandable, stand-alone storage that is network compatible.) I know nothing about this product the company or the reliability but a box along those lines may be on your short list- otherwise you will be looking at a network server with some sort of raid described in posts above.

[Snow Dog]

I have 5 years worth of images that need to be able to acces quickly. So what I am looking at doing in additon to the dvd backups that I am already doind is building two seperate raid arrays. Each would mirrior the other one, so I would have redudant backups. What is my best config for doing this?

Not to posthole your skin track but with 250GB drives (cheap) you'd need 4x5x2=40 drives. That's a lot of electricity running 365x24. At least you won't have to heat your place in the winter.

A pile of USB or Firewire drives is an option too. Just plug in the ones you need. Not as cool as big mutha fckn server but it'll work.

[Meathelmet]

And as fellow photog, i have to ask for what critical printwork do you have to keep you TIFFs unpacked for? Like,why not using the LZW for example?
I amnot taking a piss or anything,just curious.

Just that i have a lot of collegues that do their main archiving that way, and state that the loss of data in compression is neglible compared to non-packed.
Even for some superlarge format printing. (in wich case the client still would use some third party programs to do the rezizing/RIP)

Just intrested.

[Nobody Famous]

Think I'll take back earlier statement that RAID is not ready for home use yet. Last year I read RAID via USB technology was 'right around the corner' and now I see there are some products out there, such as this RAID 1 system that connects to your computer via a USB port. The upper-capacity limits on those are smaller that what you're looking for, but if you search for USB RAID you'll probably find bigger.

Question for RShea, or others; I recall from past experience that if you ever change out a drive in a Raid 5 box, and want the new drive to work hands off (i.e., formatted, data copied from the other operating drives, etc., used to call this hot swapping) there were some restrictions on the new drives hardware, such as the geometry had to be the same, possibly had to be by the same manufacturer, or some limitations like that. Do you know anything about that?

[Snow Dog]

Like,why not using the LZW for example?
I amnot taking a piss or anything,just curious.

LZW works best with text. Programs can compress well because there's usually large blocks of repeating data which is easy to compress. Images don't compress very well so there's nothing to gain.

Just that i have a lot of collegues that do their main archiving that way, and state that the loss of data in compression is neglible compared to non-packed.

I hope so since LZW is lossless. The common archiving tools like ARC and ZIP are lossless too. The before and after should be identical.

Meaningful image compression is usually lossy so every time you decompress-edit-compress you lose a little more of the image.

[fez]

If you are already backing everything up to dvd, then really you dont need raid. keep backing up to dvd, if you lose a drive it will be a pain, but i doubt you will really be able to justify the expense and headache of raid. just get the cheapest p4 tower you can get your hands on. buy 4 of the biggest hard drives you can find and a gig of ram. put two of drives in the tower and install the ram. install the linux of your choice. pull the cd-rom or rw drives from the tower and put the other two hard drives in there. Set up the new linux box on your network (not sure about apple networking, samba?) but make sure the server is wired (not wireless) for better transfer. there you have it a 4 drive cheap NAS server. need to scale? build another box.

[Ski Monkey]

Here is a NAS solution with RAID options I was looking into largely for backups.

http://www.tomsnetworking.com/2005/08/19/head_to_head/

[Meathelmet]

LZW works best with text. Programs can compress well because there's usually large blocks of repeating data which is easy to compress. Images don't compress very well so there's nothing to gain.

Umm, you get me confused.

A test TIFF of 150MB gets shrunk to 30mb when using Interleaved and to 36MB when using PerChannel types of packaging.

So, in short Interleaved = 1 : 5 ratio
and PerChannel = 1 : 4.8 ratio

What am i missing?

The pictures are now one fift of the original size..

[Snow Dog]

What am i missing?

More to the point, what am I missing.

A TIFF can contain more than the raw image so maybe parts of it are very compressable.

From scantips.com:

TIFF image files optionally use LZW lossless compression. Lossless means there is no quality loss due to compression. Lossless guarantees that you can always read back exactly what you thought you saved, bit-for-bit identical, without data corruption. This is a critical factor for archiving master copies of important images. Most image compression formats are lossless, with JPG and Kodak PhotoCD PCD files being the main exceptions.

Compression works by recognizing repeated identical strings in the data, and replacing the many instances with one instance, in a way that allows unambiguous decoding without loss. This is fairly intensive work, and any compression method makes files slower to save or open.

LZW is most effective when compressing solid indexed colors (graphics), and is less effective for 24 bit continuous photo images. Featureless areas compress better than detailed areas. LZW is more effective for grayscale images than color. It is often hardly effective at all for 48 bit images (VueScan 48 bit TIF LZW is an exception to this, using an efficient data type that not all others use


If you can get 5:1 (lossless) compression then compress away.