March 5 2013
This How-To covers extracting files from the SmartOS ISO
image and replacing them, which is automated with the
node.js script isomerge.
If you are trying to modify/change a device driver in a SmartOS image,
you are in the right place.
In this case I am updating the Areca ARC driver arcmsr
for the 1882 and 1880 RAID cards, including compiled
code/drivers not yet committed to the SmartOS image.
This How-To is written for vi imparied SmartOS noobs
who are trying to get some funky piece of bleeding edge
hardware working with a new device driver that some
nice person from the SmartOS/illumos dev community
patched up for you.
BONUS - isomerge doesn't make a bootable USB image,
but all the commands required are listed at the end.
This How-To covers extracting files from the SmartOS ISO
image and replacing them, which is automated with the
node.js script isomerge.
If you are trying to modify/change a device driver in a SmartOS image,
you are in the right place.
In this case I am updating the Areca ARC driver arcmsr
for the 1882 and 1880 RAID cards, including compiled
code/drivers not yet committed to the SmartOS image.
This How-To is written for vi imparied SmartOS noobs
who are trying to get some funky piece of bleeding edge
hardware working with a new device driver that some
nice person from the SmartOS/illumos dev community
patched up for you.
BONUS - isomerge doesn't make a bootable USB image,
but all the commands required are listed at the end.
Install SmartOS From the top. You need a machine that actually boots SmartOS (64 bit cpu, kvm not required). See the SmartOS Install Wiki. You will need access to the Joyent SmartOS VM images (aka datasets), BUT on a bare SmartOS install /var/db/imgadm/sources.list is missing/empty, so fill it with the images location as follows: # cd / # echo https://datasets.joyent.com/datasets >> /var/db/imgadm/sources.list you can quickly install pkgin, and nano into the global zone # curl -k http://pkgsrc.joyent.com/sdc6/2012Q2/x86_64/bootstrap.tar.gz | gzcat | tar -xf - THIS puts everything onto the global zone zfs volume in /opt These commands set up pkgin and then use it to install nano. # pkg_admin rebuild # pkgin -y up # man pkgin # pkgin help # pkgin in nano OK, now we can grab the appropriate SmartOS Zone base64 image. # cd /var/db/imgadm # imgadm # imgadm update # imgadm avail | grep base64 This lists the images, most recent at the bottom. Using the following SmartOS base64 image: fdea06b0-3f24-11e2-ac50-0b645575ce9d base64 1.8.4 smartos 2012-12-05T21:59:37Z # imgadm import fdea06b0-3f24-11e2-ac50-0b645575ce9d Importing image fdea06b0-3f24-11e2-ac50-0b645575ce9d (base64 1.8.4) from "https://images.joyent.com" 100% [=============================] time 158.2s eta 0.0s Imported image fdea06b0-3f24-11e2-ac50-0b645575ce9d to "zones/fdea06b0-3f24-11e2-ac50-0b645575ce9d". JSON for the Zone (yes, this is where I need nano): { "alias": "smartos-isomerge", "hostname": "smartos-isomerge", "brand": "joyent", "max_physical_memory": 32768, "tmpfs": 8192, "fs_allowed": "ufs,pcfs,tmpfs,hsfs", "image_uuid": "fdea06b0-3f24-11e2-ac50-0b645575ce9d", "quota": 15, "nics": [ { "nic_tag": "admin", "ip": "dhcp" } ] } Save to the file Smartos_Isomerge.json Create the Zone with vmadm # vmadm create -f SmartOS_Isomerge.json Successfully created 14a9d390-79bc-4331-8e33-d892295c150c - will report the [uuid] of the new zone (not the same as the example above, a new uuid). Tip: these long [uuid]s can be tab-completed with vmadm and zlogin commands, just type a few chars! # vmadm get [uuid] --will report the JSON state of the zone. # vmadm list - will list the available VMs # vmadm update [uuid] max_physical_memory=32768 - - will update the max memory of the vm # vmadm start [uuid] -restarts a halted zone Log in to the new zone as root # zlogin [uuid] - to get in.. - escape sequence to get out of the zone is: # ~. Looks like this: [root@2c-27-d7-2e-93-55 /var/db/imgadm]# zlogin 14a9d390-79bc-4331-8e33-d892295c150c [Connected to zone '14a9d390-79bc-4331-8e33-d892295c150c' pts/4] __ . . _| |_ | .-. . . .-. :--. |- |_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; SmartMachine base64 1.8.4 `-' http://wiki.joyent.com/jpc2/SmartMachine+Base [root@smartos-isomerge ~]# | Set up for isomerge within the smartos-isomerge Zone Grab a few packages: nano, wget and git wih pkgin. [root@smartos-isomerge ~]# pkgin in nano [root@smartos-isomerge ~]# pkgin in wget [root@smartos-isomerge ~]# pkgin in scmgit-1 Git clone smartos-isomerge [root@smartos-isomerge ~]# git clone https://github.com/jclulow/smartos-isomerge Cloning into 'smartos-isomerge'... remote: Counting objects: 22, done. remote: Compressing objects: 100% (18/18), done. remote: Total 22 (delta 5), reused 20 (delta 3) Unpacking objects: 100% (22/22), done. [root@smartos-isomerge ~]# Locate the SmartOS Image you want to manipulate by version datestamp: I will modify the 20130222 iso listed at the bottom of the listing (at time of writing). smartos-20130222T000747Z-USB.img.bz2 22-Feb-2013 02:04 155179890 smartos-20130222T000747Z.iso 22-Feb-2013 02:03 268290048 smartos-20130222T000747Z.vmwarevm.tar.bz2 22-Feb-2013 02:06 155270475 Grab the SmartOS image with wget [root@smartos-isomerge ~]# cd smartos-isomerge [root@smartos-isomerge ~/smartos-isomerge]# wget https://download.joyent.com/pub/iso/smartos-20130222T000747Z.iso --2013-03-05 05:23:50-- https://download.joyent.com/pub/iso/smartos-20130222T000747Z.iso Resolving download.joyent.com (download.joyent.com)... 165.225.154.118 Connecting to download.joyent.com (download.joyent.com)|165.225.154.118|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 268290048 (256M) [application/octet-stream] Saving to: `smartos-20130222T000747Z.iso' 100%[=============================================>] 268,290,048 524K/s in 8m 30s 2013-03-05 05:32:22 (513 KB/s) - `smartos-20130222T000747Z.iso' saved [268290048/268290048] Gather The New Files for your Use Case I am merging a new version of the Areca ARC 1882 device driver arcmsr provided by RM. So I grab the device driver where he left it form me... (which will go into /kernel/drv/amd64) [root@smartos-isomerge ~/smartos-isomerge]# wget https://fingolfin.org/tmp/arcmsr --2013-03-05 05:50:24-- https://fingolfin.org/tmp/arcmsr Resolving fingolfin.org (fingolfin.org)... 72.14.186.115, 2600:3c00::f03c:91ff:fe96:a264 Connecting to fingolfin.org (fingolfin.org)|72.14.186.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 77912 (76K) [text/plain] Saving to: `arcmsr' 100%[=============================================>] 77,912 89.4K/s in 0.9s 2013-03-05 05:50:26 (89.4 KB/s) - `arcmsr' saved [77912/77912]] I also need to update the /etc/driver_aliases file to include a line for 1882 support. The file lives here on GitHub: https://github.com/joyent/smartos-live/blob/master/overlay/generic/etc/driver_aliases Use wget with the "raw" version of the file URL to grab the file: [root@smartos-isomerge ~/smartos-isomerge]# wget https://raw.github.com/joyent/smartos-live/master/overlay/generic/etc/driver_aliases --2013-03-05 05:57:35-- https://raw.github.com/joyent/smartos-live/master/overlay/generic/etc/driver_aliases Resolving raw.github.com (raw.github.com)... 199.27.75.133 Connecting to raw.github.com (raw.github.com)|199.27.75.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 23894 (23K) [text/plain] Saving to: `driver_aliases' 100%[=============================================>] 23,894 105K/s in 0.2s 2013-03-05 05:57:37 (105 KB/s) - `driver_aliases' saved [23894/23894] Now I have everything I need. I will edit the local copy of driver_aliases to add in the line for 1880 and 1882 arcmsr support using nano and save it.  Set up the input JSON for isomerge - modified from the sample provided. [root@smartos-isomerge ~/smartos-isomerge]# cp sample-input.json arcmsr-isomerge.json [root@smartos-isomerge ~/smartos-isomerge]# nano arcmsr-isomerge.json  This is the edited JSON I am using as input for isomerge. Note the arcmsr driver gets 0755 permissions according to the directory contents (-rwxr-xr-x ) and goes into the appropriate directory. { "workdir": "/var/tmp/ISOMERG", "inputiso": "smartos-20130222T000747Z.iso", "mergefiles": { "/etc/driver_aliases": { "owner": "root", "group": "sys", "perms": "0444", "src": "driver_aliases" }, "/kernel/drv/amd64/arcmsr": { "owner": "root", "group": "sys", "perms": "0755", "src": "arcmsr" } } } Tip: How did I known where to put my file in the merge JSON? What are the right permissions? As root, from the global zone of my SmartOS box: # cd / # find . -name "driver.aliases" -print ./etc/driver_aliases # ls -al /etc/driver_aliases -rw-r--r-- 1 root sys 23894 Feb 22 01:34 /etc/driver_aliases # find . -name "arcmsr" -print ./kernel/drv/amd64/arcmsr # ls -al /kernel/drv/amd64/arcmsr -rWxr-xr-x 1 root sys 77912 Feb 22 01:34 /kernel/drv/amd64/arcmsr NOW we are almost ready to run isomerge isomerge is a node.js script that requires additional modules to be fetched with npm. [root@smartos-isomerge ~/smartos-isomerge]# npm install async [root@smartos-isomerge ~/smartos-isomerge]# npm install mkdirp [root@smartos-isomerge ~/smartos-isomerge]# npm install colorize isomerge also calls mkisofs, requiring the cdrtools package to be fetched with pkgin. [root@smartos-isomerge ~/smartos-isomerge]# pkgin in cdrtools Go forth and isomerge. [root@smartos-isomerge ~/smartos-isomerge]# ./merge.js < arcmsr-isomerge.json * mkdir /var/tmp/ISOMERG path.exists is now called `fs.exists`. * lofi add smartos-20130222T000747Z.iso - device: /dev/lofi/2 * fstyp /dev/lofi/2 - type: hsfs * mkdir /var/tmp/ISOMERG/iso * mount hsfs /dev/lofi/2 on /var/tmp/ISOMERG/iso opts ro * cp from /var/tmp/ISOMERG/iso to /var/tmp/ISOMERG/isounpack * umount /var/tmp/ISOMERG/iso * lofi remove /dev/lofi/2 * lofi add /var/tmp/ISOMERG/isounpack/platform/i86pc/amd64/boot_archive - device: /dev/lofi/2 * fstyp /dev/lofi/2 - type: ufs * mkdir /var/tmp/ISOMERG/root * fsck /dev/lofi/2 * mount ufs /dev/lofi/2 on /var/tmp/ISOMERG/root opts rw nologging * mv from /var/tmp/ISOMERG/root/usr.lgz to /var/tmp/ISOMERG/tmpusr.lgz * lofi uncompress /var/tmp/ISOMERG/tmpusr.lgz * lofi add /var/tmp/ISOMERG/tmpusr.lgz - device: /dev/lofi/3 * fstyp /dev/lofi/3 - type: ufs * fsck /dev/lofi/3 * mount ufs /dev/lofi/3 on /var/tmp/ISOMERG/root/usr opts rw nologging * merging files... * cp from driver_aliases to /var/tmp/ISOMERG/root/etc/driver_aliases * chown root:sys /var/tmp/ISOMERG/root/etc/driver_aliases * chmod 0444 /var/tmp/ISOMERG/root/etc/driver_aliases * cp from arcmsr to /var/tmp/ISOMERG/root/kernel/drv/amd64/arcmsr * chown root:sys /var/tmp/ISOMERG/root/kernel/drv/amd64/arcmsr * chmod 0755 /var/tmp/ISOMERG/root/kernel/drv/amd64/arcmsr * ...done merging files * umount /var/tmp/ISOMERG/root/usr * fsck /dev/lofi/3 * lofi remove /dev/lofi/3 * lofi compress /var/tmp/ISOMERG/tmpusr.lgz * mv from /var/tmp/ISOMERG/tmpusr.lgz to /var/tmp/ISOMERG/root/usr.lgz * umount /var/tmp/ISOMERG/root * fsck /dev/lofi/2 * lofi remove /dev/lofi/2 * mkisofs /var/tmp/ISOMERG/output.iso * done Congratulations, output.iso is the merged image. But my server has no CD/DVD drive, so... BONUS - Make the bootable USB image! # pkgin in pbzip # cd /var/tmp/ISOMERG/ # mkfile -n 2000000000 output-USB.img # ls isounpack output.iso output.usb # lofiadm -a output-USB.img /dev/lofi/2 --Tip: Raw device here is /dev/rlofi/2 used for fdisk, mkfs # wget https://raw.github.com/joyent/smartos-live/master/tools/usb_fdisk_table --2013-03-05 07:54:34-- https://raw.github.com/joyent/smartos-live/master/tools/usb_fdisk_table Resolving raw.github.com (raw.github.com)... 199.27.75.133 Connecting to raw.github.com (raw.github.com)|199.27.75.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1232 (1.2K) [text/plain] Saving to: `usb_fdisk_table' 100%[===============================================>] 1,232 --.-K/s in 0s 2013-03-05 07:54:36 (33.6 MB/s) - `usb_fdisk_table' saved [1232/1232] # fdisk -F usb_fdisk_table /dev/rlofi/2 # mkfs -F pcfs -o fat=32 /dev/rlofi/2:c Construct a new FAT file system on /dev/rlofi/2:c: (y/n)? y # mkdir temp # mount -F pcfs /dev/lofi/2:c ./temp # cp -r ./isounpack/boot ./temp # cp -r ./isounpack/platform ./temp # grub --batch Probing devices to guess BIOS drives. This may take a long time. GNU GRUB version 0.97 (640K lower / 65536K upper memory) [ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename. ] grub> device (hd0) output-USB.img device (hd0) output-USB.img grub> root (hd0,0) root (hd0,0) Filesystem type is fat, partition type 0xc grub> setup (hd0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/fat_stage1_5" exists... no Running "install /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded Done. grub> quit quit # umount -f /dev/lofi/2:c # lofiadm -d /dev/lofi/2 # pbzip2 output-USB.img The merged image output-USB.img.bz2 is now in bootable USB form as per the usual SmartOS image. |