SmartOS Stuff from @cwvhogue

My collection of SmartOS/illumos software, how-tos, configurations and troubleshooting notes. chogue {at} blueprint.org
Videos & Slides

Manta Unleashed BigDataSG - Part 2




Photos

Why oh Why?
 



How-Tos:


  • SmartOS isomerge replace files on the iso image, swap device drivers, make USB.
    5 March 2013



Software



Tested/Working Hardware Configurations


Tiny (2 of these):


  • Lenovo Ideacentre Q110 10032
  • Intel(R) Atom(TM) 230 1.6GHz (64-bit Atom)
  • 320Gb HDD.
  • Updraded to 5GB RAM (~4Gb useable)
  • 64-bit SmartOS ok, does not support KVM
  • Used to test boot USB, non-kvm zones, LXz (lx-branded zones), some zfs send/recv with Medium machine, builds.
Small (2 of these):


  • HP Z210 Workstation (Small Form Factor)
  • Intel(R) Xeon(R) E31225
  • 16Gb RAM
  • On-board Intel 82579LM GigE
  • Intel(R) Gigabit CT Desktop Adapter (82574L) PCIe 1x
  • 2x 2.0Tb WD Caviar Black HDD
  • update to latest BIOS 01.44 (01.41 causes a Grub Error 28: Selected item cannot fit in memory)
  • usb boot issues - BIOS requires the usb key partition flag must be marked "bootable" - I do this from CentOS Disk Utility 
  • Used to test zones, build software.

Medium (1 of these):

  • Lenovo ThinkStation D20 4158CM1
    • Dual Intel Xeon X5650 2.67GHz - Total 24 Cores.
    • Flashed MB Bios to latest (01/14/14).
    • Disable CStates (all!). Disable NUMA-aware! Enable Hyperthreading.
    • Onboard Broadcom NICs are functioning.
    • 48GB RAM (Kingston KTL-TS316S/8G, 240pin Reg ECC Single Rank DDR3-1600MHz PC3-12800 1.5V)
    • 6 Sticks in CPU1 bank - CPU2 bank does not work with this RAM
    • 96GB is unattainable with cheap RAM
    • LSI MegaRAID 9211-8I 8-Port 6Gb/S PCIEx8 SAS/SATA HBA
    • Disable onboard SAS from BIOS. Flash LSI MegaRAID BIOS to IT mode driver
    • 1 x Intel 530 120GB SSD
    • 6 x WD RED 2TB SATA3 HDD



Large (1 of these):


  • White Box Server
    • 2x Intel Xeon E5-2640 2.5GHz cpus,
    • 64GB DDR-3-1600 ECC RAM,
    • 16 x3Tb Hitachi SAS Drives (6Gb/s, 7200RPM),
    • 1x120Gb SSD Drive, DVD-RW
    • Supermicro X9DR3-F Motherboard with 2x GigE LAN, IPMI
    • ARECA ARC 1882-ix16 SAS 6GB/s RAID Controller,
    • 3U Chassis rackmount with 900W redundant power supply.
 
This server arrived from the vendor with an unregistered copy of Windows Server on the SSD drive, which they used to verify the hardware was working. They also initiated a hardware raid build, which was incomplete when the box arrived. 

Now anyone with Linux experience dating back to 1996 will tell you that bleeding-edge drivers are always an issue with non-Microsoft operating systems. In this case, the ARECA ARC 1882 RAID driver was not on the hardware list for SmartOS/illumOS, but ARECA does make FreeBSD and Solaris compatible open-source device drivers. 

I had to do what I have done in the past for Linux device drivers, figure out how to get the existing device driver code modified to work with this new hardware.  With a few iterations of figuring out the device driver organization, and help from Robert Mustachhi, we figured out that support for the ARC 1880 was already in SmartOS, just not listed on the driver lookup table. But no support was coded for the ARC 1882.

Knowing that drivers are already built on FreeBSD and Linux, some digging found a nice diff from the DragonflyBSD project on the driver which showed that 1882 support is simply another case statement and there is no board-specific code in the driver.  After passing this tip to Robert, he sent me a test version of the driver, and it was up to me to figure out how to get it onto a SmartOS USB boot drive image with the rest of the distro.

With the right driver on the USB key, deployment of a 43Tb ZFS filesytem took about 10 minutes, 4 spent on configuring the RAID controller to run as a JBOD device, then 6 minutes of typing in configuration information, then a reboot.
Troubleshooting Notes

23 March 2013
Fixing the broken SmartOS build. After ./configure
#sed -i 's/fPIC/KPIC/g' /opt/fake-subset/usr/perl5/5.12/lib/i86pc-solaris-64int/Config_heavy.pl 
Oh wait - issue appears to have been closed - no need.


21 March 2013
Areca Arc 1880,1882 Now Supported
Areca Arc 1880,1882 RAID controllers are now supported on SmartOS as of the 20130321 build. Thanks RM!


7 March 2013
Server Working, Some Torture Testing

Areca ARC 1882 Raid controller is working with RM's arcmsr and the SmartOS isomerge exercise, booked from my own modded USB key.

Torture test time. Wrote a simple manyfiles script to use the Solaris mkfile(1M) command to write a number of serially numbered fixed sized 0-filled files to disk.

Fetch the script from github raw on a new server with something I can type easily:

# wget --no-check-certificate http://tinyurl.com/aws966c

As my server has 43Tb of storage, I can use this to fill it with lots of small files or very larger sized files (e.g. 430 ish 100Gb files). Then can use pbzip2 to paralell execute bzip2 compression on these, all should return the exact same filesize.