Setting up a RAID on a Linux server


This page describes how to setup and operate a RAID (Redundant Array of Independent Disks) on a Linux machine. More general information about the different kinds of RAIDs can be found here.



Hardware
Features:
  • 2 ATA drive interfaces (master+slave) for up to 4 hard drives
  • support for RAID 0,1,1+0 and JBOD
  • Hot spare capability
  • S.M.A.R.T.
  • Native Linux kernel support  (2.2.15 and above)

  • 3DM remote disk management through the WWW.
Price:
  • MSRP: about 100 EUR (EC); 85$ (US)
Promise had released a driver module for Linux. Unfortunately it was not (yet) open-source, and the binary was available only for a small set of kernel versions, which did not match ours. So we prefered to go for the pure Linux version using the RAIDTools, although it is somewhat of a pity to use a RAID card for this. Using the RAIDTools is straightforward; after installing the RPM,  we set as a RAID 0 our 8 disks by creating a /dev/raidtab ASCII file containing

raiddev /dev/md0
        raid-level      0
        nr-raid-disks   8
        chunk-size      32
        persistent-superblock   1
        device          /dev/hde
        raid-disk       0
        device          /dev/hdf
        raid-disk       1
        device          /dev/hdg
        raid-disk       2
        device          /dev/hdh
        raid-disk       3
        device          /dev/hdi
        raid-disk       4
        device          /dev/hdj
        raid-disk       5
        device          /dev/hdk
        raid-disk       6
        device          /dev/hdl
        raid-disk       7

next we actually create the RAID with
% mkdraid /dev/md0

On powerful systems, a fairly high performance can be obtained with software RAID. With a good controller and 5-6 disks or more,  RAID0 performance will generally be limited by the bandwidth of the PCI bus (~130MB/s on a 32bit/33MHz system, and twice this number with a 64bit/33MHz bus). RAID5 is another story. Although sequential write speeds exceeding 100MB/s and read speeds >150MB/s can be measured in benchmarks, we noticed a dramatic drop in performance (by a factor 2 or more) once the machine was put under modest computing load.
 

  • The 3Ware Escalade cards

  • The 3Ware Escalade products are some of the few hardware RAID PCI products with excellent support in Linux. We chose these cards to equip our machines because of their good performance in RAID5. A review of the 7450 model is available here.  5 of our cluster nodes run the 7850 model (8 drives). 4 other nodes are equipped with the more recent 7500-12 model, which support up to 12 ATA disks.
     
    Features:
    • 4 to 12 ATA drive interfaces (all masters)
    • support for RAID 0,1,5,1+0 and JBOD
    • hardware XOR for RAID5, improving write performance
    • Hot spare capability
    • 64-bit/33MHz PCI bus
    • Native Linux kernel support  (2.2.15 and above)
    • S.M.A.R.T. support
    • 3DM remote disk management through the WWW.
    Price:
    • MSRP: about 770 EUR (EC); 600$ (US)
    Installation is a breeze, just make sure that the 3Ware driver is activated in the Linux kernel, or installed as a module. Once the card and the disks are installed and configured, the array(s) are seen as /dev/sda, /dev/sdb,... like standard SCSI devices.
     
     
    Inside view of one of the processing/storage nodes. The full length 3Ware Escalade 7500-12 card occupies a 64bit slot of the PCI bus. The 12 250GB ATA disks are on the right of the picture. Special ATA ribbon cables (with no socket for a slave drive) from 3Ware help keeping the inside of the box relatively "clean". The fans for the disks are located beneath the front slots (so they are invisible unless the slots are removed).
    The power supply (top left corner of the image) is an AMD-certified 550VA model. One can also see the 2 Athlon XP/MP2400+ CPUs (horizontal fans), and the two 1GB DDR-ECC-registered memory modules.
    Front view of one of the processing/storage nodes. Each slot (from 3Ware) has space for 3 disks, so that a total of 12 RAID disks plus the system disk and a CDROM device fit within 19''. The box itself is a cheap model which has been rotated to be used in a rack. Note the dust filters inside the front cover.


    Filesystems

    Benchmarking and Tuning
     
      Last edited 2003-02-18 by Emmanuel Bertin <bertin@iap.fr>