In part 1, I showed how to automate iDRAC setup using bash scripts. Using those scripts is great and definitely is a huge improvement over doing it all by hand. However, there are some ways to improve the process. In my use, to configure a rack of about 22 R630s, it took around 15 minutes. This is because each racadm command is being run one at a time, one server at a time. There are some tools to get ssh running in parallel, like pssh and GNU parallel, but those are kind of clunky and not easy to use. One tool that I’ve been getting into lately is called Ansible. It’s similar to Chef and Puppet, but agentless and uses SSH for all of its communication. And critically for this purpose, it runs everything in parallel, which will speed up the rack configuration dramatically.

The Setup

First, we need to install Ansible. I’ll be running this from, where else, a Raspberry Pi. Unfortunately, the Raspbian repo doesn’t include the latest version of Ansible. As of this writing, it uses version 1.7, while the latest version is 2.3. So I have to grab Ansible’s Ubuntu repo and add it to the sources list, then add the key for it so we don’t get security warnings everytime we use apt-get

sudo bash -c 'echo "deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main" >> /etc/apt/sources.list'
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367

Next we’ll update our repo lists and install Ansible

sudo apt-get update
sudo apt-get install -y ansible
ansible --version

Make sure you get version 2.3 back, otherwise something went wrong. Now that Ansible is installed, we need to setup the directory structure for our project. Create a directory called r630, or whatever you want to call it. Now go in that directory and create a structure like this

mkdir group_vars playbooks
touch ansible.cfg ssh_config hosts

This creates a playbooks directory, where we will put all our playbooks, and a group_vars directory, which we will use to house variables about different hosts, but most importantly where we will store the iDRAC password in encrypted form. Then we’ll create an ansible.cfg file which will specify several parameters for how Ansible should run our playbooks. ssh_config will tell ssh to skip things like host checking (we’ll be running on a closed network) and other settings that will speed up ssh connections. Finally, the hosts file is one method we will use to tell Ansible what hosts to actually run on.

Now that we have a basic structure, let’s fill out our ansible.cfg file

[defaults]
host_key_checking = false
record_host_keys = false
timeout = 30
inventory = hosts
log_path = ~/playbook.log
forks = 22

Ansible always expects there to at least be a [defaults] configuration group. You can specify other groups that playbooks can run from, but we’re only going to use one for now, so defaults will work. We’ll turn off host key checking because we’ll be hitting several different racks, and due to the nature of DHCP, it will probably assign the same IP to different hosts as we move from rack to rack, so we want to just ignore host key checking. We’ll set a timeout, so that we’re not waiting forever for an unreachable host, tell playbooks to use the hosts file we created by default, where to place the log file for our playbook runs, and how many ssh sessions can run at once. I was working with 22 R630s per rack in my case, so you could raise that number if you wanted, but keep any on resource usage on your Pi, and network usage in general.

Next, let’s fill out ssh_config

Host *
    GSSAPIAuthentication no
    VerifyHostKeyDNS no
    HashKnownHosts no
    TCPKeepAlive yes
    ServerAliveInterval 60

I’m not going to spend much time talking about these, but basically this is telling ssh itself not to check for host keys, to skip hitting DNS for hostnames, and other tweaks to speed up the connection process

Now, let’s fill out the hosts file. Really, this file is Ansible’s inventory, but many folks will call this file hosts or some variation thereof, probably because it’s similar to the /etc/hosts file on Linux systems. Though, rather than mapping hostnames to IP addresses, this maps hosts (be them IPs or names) to groups. This allows playbooks to only run on specific groups of hosts. You could have a webserver group, a database group, and a proxy group. You’d probably want to install some similar software on the webserver and proxy groups, so a playbook could install something like nginx on both of them, while a different playbook could install mysql on the database group. A plabyook can run against one group, some groups, all groups, or even just individual hosts that are outside of groups. Here’s a basic we can use for iDRAC

[base]
192.168.1.100
192.168.1.101
192.168.1.102
192.168.1.103
192.168.1.104
192.168.1.105
192.168.1.106

[bigswitch]
192.168.1.109
192.168.1.110

[local]
127.0.0.1

We have our settings configured, now let’s actually work on a playbook. Here’s my iDRAC config for any base system

--- # Base iDRAC config for most r630s
- gather_facts: false
  name: base idrac config
  hosts: idrac
  user: root
  tasks:
    - name: disable default password warning
      raw: racadm set idrac.tuning.DefaultCredentialWarning 0

    - name: get service tag
      raw: racadm getsvctag
      register: result
    - debug:
        var: result

    - name: enable ipv6
      raw: racadm set iDRAC.IPv6.Enable 1
      register: result

    - name: set NIC to LOM3
      raw: racadm set iDRAC.NIC.Selection 4
      register: result

    - name: disable hot spare for PSUs
      raw: racadm set System.Power.Hotspare.Enable 0
      register: result

    - name: set power policy
      raw: racadm set System.Power.RedundancyPolicy 1
      register: result

    - name: get bios version
      raw: racadm getversion -f bios
      register: result
    - debug:
        var: result

    - name: get idrac version
      raw: racadm getversion -f idrac
      register: result

    - name: create linux_os vdisk
      raw: racadm storage createvd:RAID.Integrated.1-1 -rl r1 -pdkey:Disk.Bay.0:Enclosure.Internal.0-1:RAID.Integrated.1-1,Disk.Bay.1:Enclosure.Internal.0-1:RAID.Integrated.1-1 -name LINUX_OS
      register: result
    - name: create images vdisk
      raw: racadm storage createvd:RAID.Integrated.1-1 -rl r1 -pdkey:Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1,Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1 -name IMAGES
      register: result
    - name: apply new disks
      raw: racadm jobqueue create RAID.Integrated.1-1 -s TIME_NOW --realtime
      register: result
    - debug:
        var: result

I won’t spend time talking about the various iDRAC commands, but just note you just list out the actual racadm commands that you normally would via ssh or on the console. Unfortunately, there’s not an iDRAC Ansible module at this time, but using the raw module to send raw ssh commands still works fine. Another thing to note is that I’m registering the results of some of the commmands as variables, then printing them out on the screen (as well as via the playbook log). register is what stores the results of an action to a variable, and debug is what spits it out on screen. I’m working on writing the results to a log file with the blockinfile module, but haven’t finished it yet; expect another post when that’s done.

One last thing is we need to setup our encrypted group variables. We already created a group_vars directory, so now we’ll create a vars and a vault directory.

cd group_vars
touch vars vault

The way this will work, is we will use the ansible_ssh_pass builtin variable to tell Ansible the password to use when logging in to iDRAC. We’ll store this variable in the vars file. But rather than actually specifying the password there in an unencrypted file, it will reference vault_ansible_ssh_pass which actually be in the encrypted vault file.

$ cat vars
---
ansible_ssh_pass = "{{ vault_ansible_ssh_pass }}"

$ ansible-vault view vault
Vault password:
vault_ansible_ssh_pass: calvin

This setup allows you to store all your variables in the vars file, while keeping the actual contents of sensitive variables safe in the encrypted vault file. To create the vault file, use

ansible-vault create vault

Now, tying all this together, there is one command to run the playbook

ansible-playbook playbooks/base_config.yml --ask-vault-pass

And that’s it. What used to take around 10-15 minutes using bash scripts, now takes 1 and a half minutes using Ansible.

More Reading

• Ansible Vault - Digital Ocean: How to Use Vault to Protect Sensitive Ansible Data on Ubuntu 16.04