Automating Setup on a Dell R630, Part 1 // Human as a Service

Recently I was setting up a new server rack with several Dell R630s. The out-of-band management tool for Dell devices is called iDRAC. It works fine for most cases, I suppose. You can configure iDRAC and other BIOS settings over ssh allowing you to script out your work which is much faster than doing things in the GUI. The only problem is all of the R630s were set up with static IPs of 192.168.0.120 by default, which makes connecting to 12 of them over ssh problematic. An easy way to get around this problem is by setting up a DHCP server that will assign addresses to each server, thereby letting you access each one independently.

But wait, you ask, you still need to access the GUI to change each server to use DHCP. And if you’re going to do that, why not just assign the address you want while you’re in the GUI. The answer is that getting into the GUI requires you to reboot the server which in turn means that you have to wait for the plodding iDRAC software which seems to be built on Windows (!@?$@#$) to come up before you can make any changes. I’m wanting fast here, so I’m still going to script it out. The easiest way I’ve found to do this is to connect your computer to an R630, set an IP of 192.168.0.121 (or really anything on that same /24 subnet should work, but why get too crazy), and run a quick racadm command to setup DHCP.

Physical Connection

First, connect your computer to an R630 via ethernet. Then set your IP address on that interface to 192.168.0.121 (any IP address in that /24 subnet should work; actually any address at all might…). Next, confirm you can connect to the R630 by pinging 192.168.0.120. If that works, move on to ssh.

SSH

Because we’re going to be connecting to multiple servers with the same address, we want to ignore ssh-key checking and keep ssh from adding the server and its public key to our known hosts file. To do this, use the following ssh options:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=dev/null -o LogLevel=QUIET

This will let it just connect to the server, security be damned! If you have any man-in-the-middle attacks between you and a server you’re directly connected to and sitting in front of, ssh key checking is the least of your worries.

To keep from having to enter a password for each connection made to iDRAC, use sshpass. For a Mac, install it using Homebrew. If you’re not using Homebrew, start using it. sshpass is considered by many to be bad from a security perspective, so it’s not included with Homebrew by default. But you can install it from Sourceforge (yes, I know, I know. But it is legit) using the the command:

brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

After doing that, just add sshpass -p <your password> in front of the ssh command, and you will be that much closer to automation heaven.

sshpass -p password ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=dev/null -o LogLevel=QUIET

Now we’re cooking with gas!

Scripts

We’re going to attempt to make our scripts modular, that way base ssh and racadm logic that isn’t going to change will stay in one shell script, and the actual racadm commands that will vary depending on what operation we want will stay in their own scripts. So our directory structure will look something like this:

idrac.sh
idrac_config.sh
idrac_config_bigswitch.sh
idrac_config_nostorage.sh
idrac_enable_pxe.sh
idrac_get_ips.sh

That top idrac.sh file is the main one, which all other scripts will call. And it looks like this

#!/bin/bash

USER=root
pass=$1
server=$2
shift
shift
cmd=$*

echo "$cmd"
sshpass -p "${pass}" ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=dev/null -o LogLevel=QUIET ${USER}@"${server}" racadm "${cmd}"

You can obviously modify the variable values to store or pass in values to your liking. In my example, I’m always using the root user, and rather than storing the password in a file, I’ll pass it in at runtime. This means that our other scripts will only need to call this script and supply the iDRAC password, which server to run this on (its IP address) and the actual racadm command.

Moving on, we can now create our actual iDRAC scripts. Here’s an example of one of my base idrac configuration scripts

#!/bin/bash

if [ -z "$1" ]
then
    echo "Please specify the name of the log file and servers file"
    exit 1
elif [ -f "$1"_$(basename "$0" .sh).log ]
then
    echo "Filename already exists. Please choose a different one."
    exit 1
else
    site=$1
fi

if [ -z "$2" ]
then
    echo "Please specify the password used to login to iDRAC"
    exit 1
else
    pass=$2
fi

RACADM="./idrac.sh"
LOGFILE="$1"_$(basename "$0").log


for server in $(cat "$site".txt)
do
    echo "Starting iDRAC config for ${server}" >> "$LOGFILE"
    # disable default password warning
    $RACADM "$pass" "$server" "set idrac.tuning.DefaultCredentialWarning 0" >> "$LOGFILE"
    # get idrac, BIOS and lifecycle controller version
    $RACADM "$pass" "$server" "getversion" >> "$LOGFILE"
    # get service tag
    $RACADM "$pass" "$server" "getsvctag" >> "$LOGFILE"
    # get mac address of idrac port
    $RACADM "$pass" "$server" "get iDRAC.NIC.MACAddress" >> "$LOGFILE"
    # create virtual drive for OS install
    $RACADM "$pass" "$server" "storage createvd:RAID.Integrated.1-1 -rl r1 -pdkey:Disk.Bay.0:Enclosure.Internal.0-1:RAID.Integrated.1-1,Disk.Bay.1:Enclosure.Internal.0-1:RAID.Integrated.1-1 -name LINUX_OS" >> "$LOGFILE"
    # create virtual drive for images
    $RACADM "$pass" "$server" "storage createvd:RAID.Integrated.1-1 -rl r1 -pdkey:Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1,Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1 -name IMAGES" >> "$LOGFILE"
    # actually create it here
    $RACADM "$pass" "$server" "jobqueue create RAID.Integrated.1-1 -s TIME_NOW --realtime" >> "$LOGFILE"
    # enable ipv6
    $RACADM "$pass" "$server" "set iDRAC.IPv6.Enable 1" >> "$LOGFILE"
    # set idrac port to LOM 3
    $RACADM "$pass" "$server" "set iDRAC.NIC.Selection 4" >> "$LOGFILE"
    # disable hot spare for PSUs
    $RACADM "$pass" "$server" "set System.Power.Hotspare.Enable 0" >> "$LOGFILE"
    # set redundancy policy to something
    $RACADM "$pass" "$server" "set System.Power.RedundancyPolicy 1" >> "$LOGFILE"
    echo " " >> "$LOGFILE"
done