Install all firmware updates on Dell PowerEdge servers with Ansible
Contents
This feature is only available with iDRAC Enterprise License.
The supported servers are :
- 12G and 13G PowerEdge Servers: iDRAC 7 and 8 with Firmware version 2.50.50.50 and above
- 14G PowerEdge Servers: iDRAC 9 with Firmware version 3.18.18.18 and above
While looking once more for the most easy way to update Dell servers firmwares, I stepped on this awesome video showing someone from Dell use Ansible to update at once all the firmwares of a server :
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="allowfullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/oagvLq6am3s?autoplay=0&controls=1&end=0&loop=0&mute=0&start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"
></iframe>
</div>
So, after some research on how to do it, here are all the steps needed to achieve this using only the CLI from a Linux Out-of-Band machine.
Overview
In this guide, we’ll install Dell EMC Repository Manager to create and maintain a repository containing the updates files for our servers and generate a Catalog.xml file that contains the relation informations. Then, we’ll export these updates to a NFS share, and use Ansible + Dell EMC OpenManage Ansible Modules (which uses Dell EMC OpenManage Python SDK internally) to communicate with the iDRACs and trigger the updates that will be download from the NFS share.
Since iDRACs will generate Server Configuration Profile files to the NFS share as the nobody user, Ansible must be run as root because it’ll need the permission to edit these files.
Prerequisites
Dell EMC Repository Manager
We need to download Dell Repository Manager from Dell website and install it. It can be found as a .BIN file and installed just by executing it. It also works on Ubuntu/Debian, even if not stated by Dell.
chmod +x DRMInstaller_3.1.0.468.bin
./DRMInstaller_3.1.0.468.bin
The GUI didn’t worked right away for me, as I got the following error while trying to launch it using drm command :
Graphics Device initialization failed for : es2, sw
Error initializing QuantumRenderer: no suitable pipeline found
java.lang.RuntimeException: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
at com.sun.javafx.tk.quantum.QuantumRenderer.getInstance(QuantumRenderer.java:280)
at com.sun.javafx.tk.quantum.QuantumToolkit.init(QuantumToolkit.java:221)
at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:248)
at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:209)
at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:695)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$154(LauncherImpl.java:182)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:94)
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:124)
... 1 more
No toolkit found
I needed to install the package libswt-gtk-3-java to fix this issue.
But in this configuration, we try to do everything from the CLI, so that doesn’t really matter.
Optional
Plugins can be installed in order to create bootable ISO including the firmwares, concatenated BIN files, etc… We can list the available plugins with the following command :
drm --list=plugin
Listing Plugins...
Plugin Current Version Latest Version
-------------- -------------- --------------
Dell EMC System Update(BIN) 1.5.3 1.5.3
Dell EMC Bootable ISO Plug-in 902.1 902.1
Dell EMC Server Update Utility x64 Plug-in 782 782
Dell EMC System Update(EXE) 1.5.3 1.5.3
To install a plugin, we can use the following command : drm --update --plugin="Dell EMC System Update(EXE)".
Dell EMC OpenManage Python SDK
Ansible modules require that Dell EMC OpenManage Python SDK is installed on the system before being able to get installed. So here are the few steps to do it :
git clone https://github.com/dell/omsdk.git
sudo apt-get install python-pip
cd omsdk
sudo -H pip install -r requirements-python2x.txt # use requirements-python3x.txt if using Python 3
sh build.sh 1.2 345
cd dist
sudo -H pip install omsdk-1.2.345-py2.py3-none-any.whl
Please refer to the project GitHub page to install the latest version
Ansible modules
We need to install the Dell-EMC-Ansible-Modules-for-iDRAC for our playbook to work :
git clone https://github.com/dell/Dell-EMC-Ansible-Modules-for-iDRAC.git
cd Dell-EMC-Ansible-Modules-for-iDRAC
sudo -H python install.py
NFS server
A NFS share will be used by the iDRAC to access the latest firmware versions and install them onto the machine. It’ll also be used by the iDRACs to create a Server Configuration Profile file and by Ansible to modify these files.
A external NFS volume can be used but for the previously stated reasons it’ll need to be acessible from the iDRACs and mounted locally on the server that runs Ansible.
For creating an NFS share, we can proceed like this :
This is not a complete secure configuration of NFS server, please us it for POC only
sudo apt-get install rpcbind nfs-kernel-server
Then we create a folder for the NFS share :
mkdir -p /opt/dell/dellemcrepositorymanager/export
chmod 777 /opt/dell/dellemcrepositorymanager/export
Then, modify the /etc/exports file to add the share (adapt the IP to the one from your network) :
/opt/dell/dellemcrepositorymanager/export 10.220.0.0/25(rw,fsid=0,insecure,no_subtree_check,async)
The service needs to be rebooted for the share to become available :
service nfs-kernel-server restart
We can check access from the iDRAC using the Test network connection button in Maintenance, System Update, Manual Update after changing the Location Type to Network Share and entered the infos about the IP address of the NFS server and the Share Name.
Export of the DUPs to NFS share
Once Dell Repository Manager is installed, we must configure it with a repository and export the Dell EMC Update Packages to the NFS share. This can be done using the following commands.
First, let’s create the repository. Be aware that if you don’t filter on the servers, hundred of gigabytes will be downloaded !
$ drm --create --repository=Dell --inputplatformlist="R640,R740xd"
Creating repository Dell
Job Submitted for repository creation
We can check the status with the following command :
drm --repository=Dell --details
Listing version information for: Dell
Version Size Date Created
------- ---- ------------
1.00 5.63 GB 11/23/18 12:24 A.M
The firmwares will be downloaded in /var/dell/drm/store/
We can check the progress using the following command :
drm --list=job
Listing Jobs...
Name Status Job Type Next Execution date/time Last Execution date/time
---- ------ -------- ------------------------ ------------------------
Download_11/23/2018_11:04 SUCCESS DOWNLOAD COMPONENTS NA 11/23/18 11:11 A.M
Plugin Download- Dell EMC Server Update Utility x64 Plug-in SUCCESS PLUGIN_UPDATE NA 11/23/18 11:12 A.M
Plugin Download- Dell EMC Bootable ISO Plug-in SUCCESS PLUGIN_UPDATE NA 11/23/18 11:27 A.M
Plugin Download- Dell EMC System Update(BIN) SUCCESS PLUGIN_UPDATE NA 11/23/18 11:28 A.M
Plugin Download- Dell EMC System Update(EXE) SUCCESS PLUGIN_UPDATE NA 11/23/18 11:28 A.M
Once everything is downloaded, let’s export the firmware binaries and the Catalog.xml file. The Catalog.xml files contains informations about the firmwares, and is used by the iDRAC to determine the updates to install.
drm --deployment-type=export --repository=Dell:1.00 --location=/opt/dell/dellemcrepositorymanager/export
Job Submitted for export
The location can be an external NFS, in this case, we need to mount it locally of specify it with the --location command.
Once again, let’s check the progress using the following command :
drm --list=job
Listing Jobs...
Name Status Job Type Next Execution date/time Last Execution date/time
---- ------ -------- ------------------------ ------------------------
Download_11/23/2018_11:04 SUCCESS DOWNLOAD COMPONENTS NA 11/23/18 11:11 A.M
Plugin Download- Dell EMC Server Update Utility x64 Plug-in SUCCESS PLUGIN_UPDATE NA 11/23/18 11:12 A.M
Plugin Download- Dell EMC Bootable ISO Plug-in SUCCESS PLUGIN_UPDATE NA 11/23/18 11:27 A.M
Plugin Download- Dell EMC System Update(BIN) SUCCESS PLUGIN_UPDATE NA 11/23/18 11:28 A.M
Plugin Download- Dell EMC System Update(EXE) SUCCESS PLUGIN_UPDATE NA 11/23/18 11:28 A.M
Export_11/23/2018_12:42 RUNNING SHARE NA NA
Let’s wait for it to have the SUCCESS status before running Ansible.
Ansible playbook
The Ansible playbook should look like this :
---
- hosts: hosts
connection: local
name: Update Firmware Inventory
gather_facts: False
tasks:
- name: Update Firmware Inventory
dellemc_install_firmware:
idrac_ip: "{{ inveotry_hostname }}"
idrac_user: "ansible"
idrac_pwd: "*******"
share_name: "10.220.0.10:/opt/dell/dellemcrepositorymanager/export"
share_mnt: "/opt/dell/dellemcrepositorymanager/export"
catalog_file_name: "Dell_1.00_Catalog.xml"
reboot: True
job_wait: True
tags :
- installfirmware
The share_mnt option can be either the local NFS export directory on the local server, or the local mount of the external NFS volume used. As a Server Configuration Profile file a generated by the server and copied to the NFS share, it must be accessible locally.
Other options can be provided as specified in the related documentation
Let’s create an inventory file also :
# inventory file
---
[hosts]
ipmi-01.example.com
ipmi-02.example.com
Running Ansible !
Once the export is successfully completed, we can run the playbook !
Please be careful that it’ll reboot the host to apply the update if needed as we specified
reboot: Truein the playbook
sudo -H ansible-playbook dellemc_install_firmware.yml -i inventory -t installfirmware -vvvvv
It’ll take a really long time to load and apply the updates, and it can be up to 1 hour. We also need to wait as long as it takes before the updates appears in the iDRAC logs and Job queue.
After this long wait, here is what we can see :
changed: [ipmi-01.example.com] => {
"changed": true,
"invocation": {
"module_args": {
"catalog_file_name": "Dell_1.00_Catalog.xml",
"idrac_ip": "ipmi-01.example.com",
"idrac_port": 443,
"idrac_pwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"idrac_user": "root",
"job_wait": true,
"reboot": true,
"share_mnt": "/opt/dell/dellemcrepositorymanager/export",
"share_name": "10.220.0.10:/opt/dell/dellemcrepositorymanager/export",
"share_pwd": null,
"share_user": null
}
},
"msg": {
"@odata.context": "/redfish/v1/$metadata#DellJob.DellJob",
"@odata.id": "/redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_430165520584",
"@odata.type": "#DellJob.v1_0_0.DellJob",
"CompletionTime": "2018-11-23T18:25:58",
"Description": "Job Instance",
"EndTime": null,
"Id": "JID_430165520584",
"JobState": "Completed",
"JobType": "ImportConfiguration",
"Message": "Successfully imported and applied Server Configuration Profile.",
"MessageArgs": [],
"MessageId": "SYS053",
"Name": "Import Configuration",
"PercentComplete": 100,
"StartTime": "TIME_NOW",
"Status": "Success",
"TargetSettingsURI": null,
"retval": true
}
}
Author VaLouille
LastMod 2018-11-23