One of the computing industry’s dirtiest little secrets is that the total time spent nurturing a system is largely comprised of the time it costs to maintain it vs. the time it takes to deploy it. Managing change, delivering updates, rolling back due to unforeseen circumstances or scripts gone astray — all are just examples of challenges that DevOps simplifies. And thanks to the sanity affirmed by software engineering professionals like Jim Bird, maintenance and devops are inseparable.
With this blog entry I thought it’d be useful to take a hands-on look at the notion of delivering an update bundle, comprised of 200 software packages, on a Solaris 11 system that was initially installed in October 2010. Back then, nearly 3 and a half years ago, there was a version of Solaris that contained preview technologies that were made available prior to the generally available release of Solaris 11 — it was called Solaris 11 Express.
So the system at hand has actually been going through quite a few updates over the last 3.5 years – with the first installation taking place in Oct 2010, with various updates having been applied since then.
isaac@HPensolaris:~$ beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- 151a-july11 - - 12.49M static 2011-07-18 12:54 151a-july11-1 - - 15.34M static 2011-09-19 15:12 FCS - - 79.24M static 2013-01-19 15:46 S11.1 - - 16.37M static 2013-01-20 04:25 S11.1clone NR / 23.68G static 2013-01-20 22:36 S11EwithSRU11 - - 60.93M static 2011-10-10 09:18 S11EwithSRU11-1 - - 44.22M static 2013-01-19 14:57 UpgradedToFCS - - 53.88M static 2013-01-19 20:07 before_168 - - 87.91M static 2011-07-02 18:46 snv_150 - - 49.48M static 2010-10-17 11:05 snv_150-bkup - - 6.85M static 2011-03-17 00:58 snv_150-bkup-1 - - 356.53M static 2011-07-02 18:47 snv_151a_ga - - 10.59M static 2011-03-17 11:57
Notice the right-most column containing the creation date; these are boot environments that are introduced with Solaris 11. Boot environments are implemented on top of ZFS snapshots, and – because of engineering with IPS (image packaging system) they provide immensely useful benefits that simplify change management, offer ability to have reversibility, compliance validation, risk assessment, and more.
The second column (from left) contains mostly dashes, except for “NR” on the 5th line in the output. This means that the corresponding boot environment is currently active (“N”), and will be the one selected after a system reboots (“R”). (The “R” assignments can arbitrarily be changed, naturally).
So what one can glean from the output above is that the currently active boot environment is actually NOT the one that the system was initially deployed with, and that from its name, it suggests that its a Oracle Solaris 11.1 release.
isaac@HPensolaris:~$ cat /etc/release Oracle Solaris 11.1 X86 Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved. Assembled 19 September 2012
So let’s go ahead and attempt to update the system
isaac@HPensolaris:~$ pkg update pkg update: Insufficient access to complete the requested operation. Please try the operation again as a privileged user.
Oh right, can’t apply system changes by default without appropriate privileges.
isaac@HPensolaris:~$ su Password: isaac@HPensolaris:~# pkg update pkg update: Certificate '/var/pkg/ssl/c892869bfaaf335a2399fcd813b6bbef9126cbc4' for publisher 'solaris' needed to access 'https://pkg.oracle.com/solaris/support/', has expired. Please install a valid certificate. isaac@HPensolaris:~# cd /var/pkg/ssl isaac@HPensolaris:/var/pkg/ssl# ls -la *.pem -rw-r--r-- 1 isaac staff 737 Jan 19 2013 Oracle_Solaris_11_Support.certificate.pem -rw-r--r-- 1 isaac staff 887 Jan 19 2013 Oracle_Solaris_11_Support.key.pem
Ah, that means that the certificates that pull updates from pkg.oracle.com have expired because they were good for 1 year – and we’re now in February 2014.
Of course, the publisher is set and can be verified via pkg(1):
isaac@HPensolaris:/var/pkg/ssl# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris origin online F https://pkg.oracle.com/solaris/support/
With this info at hand, going over to pkg-register.oracle.com to login with my Oracle SSO and grabbing updated certificate and key associated with the Oracle Solaris 11 Support Repository, placing them in /var/pkg/ssl and running:
isaac@HPensolaris:~# pkg set-publisher -k /var/pkg/ssl/Oracle_Solaris_11_Support.key.pem -c /var/pkg/ssl/Oracle_Solaris_11_Support.certificate.pem -G '*' -g https://pkg.oracle.com/solaris/support/ solaris
Now, re-attempting the update process:
isaac@HPensolaris:~# pkg update Packages to install: 4 Packages to update: 198 Mediators to change: 1 Create boot environment: Yes Create backup boot environment: No pkg: The following packages require their licenses to be accepted before they can be installed or updated: ---------------------------------------- Package: pkg://firstname.lastname@example.org,5.11:20131230T162629Z License: LICENSE License requires acceptance. To indicate that you agree to and accept the terms of the licenses of the packages listed above, use the --accept option. To display all of the related licenses, use the --licenses option.
Cool – making progress. This shows us that there are 198 packages with available updates, and 4 new packages that will be installed. All of the changes will be recorded into a newly created boot environment, leaving the current boot environment unmodified. This means we can deliver the updates while knowing that we can boot back to the unmodified boot environment, in cases of failures validating the updated system. Naturally, all of this would be completely automated, but going through this process interactively hopefully indicates the types of gotchas that may arise, whilst demonstrating the goodness of what’s available in the OS already. Rant: I wish that more OSes would have this kind of an approach to delivering updates, including Apple’s iOS. I really do miss some of the intricacies of iOS 6 on some of my devices, and having no way to fall back onto iOS 6, even periodically, puzzles me – while reminding me of the adoption issues Microsoft experienced when users attempted to run Windows NT 3.5 on Intel’s 486-based systems, whilst most of the development and user-acceptance testing was done on Pentium/586-based systems. Newer systems run newer code faster then slower systems, d’uh. But we all live and learn from each other’s mistakes, hopefully – and that’s one of the fun parts of life. Moving on …
So if we’re scripting the updating process – the lesson here is to include “–accept”, like so:
isaac@HPensolaris:~# pkg update --accept
but to first and foremost, verify the certificates/keys are good, and try to do so at least a month prior to their expiration. Who knows what technical issues might arise getting them renewed, and you don’t want to wait until the last minute to initiate the renewal process. Start 11 months after they’ve been issued, automate the process, and go from there. Here’s the correct, expected output.
isaac@HPensolaris:~# pkg update --accept Packages to install: 4 Packages to update: 198 Mediators to change: 1 Create boot environment: Yes Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 202/202 15923/15923 441.2/441.2 717k/s PHASE ITEMS Removing old actions 1268/1268 Installing new actions 2185/2185 Updating modified actions 17628/17628 Updating package state database Done Updating package cache 198/198 Updating image state Done Creating fast lookup database Done Reading search index Done Building new search index 833/833 A clone of S11.1clone exists and has been updated and activated. On the next boot the Boot Environment S11.1clone-1 will be mounted on '/'. Reboot when ready to switch to this updated BE. --------------------------------------------------------------------------- NOTE: Please review release notes posted at: http://www.oracle.com/pls/topic/lookup?ctx=E26502&id=SERNS ---------------------------------------------------------------------------
Paying close attention to the comment about S11.1clone-1
We can actually see a new boot environment having been created:
isaac@HPensolaris:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- 151a-july11 - - 12.49M static 2011-07-18 12:54 151a-july11-1 - - 15.34M static 2011-09-19 15:12 FCS - - 79.24M static 2013-01-19 15:46 S11.1 - - 16.37M static 2013-01-20 04:25 S11.1clone N / 378.5K static 2013-01-20 22:36 S11.1clone-1 R - 25.70G static 2014-02-16 10:09 S11EwithSRU11 - - 60.93M static 2011-10-10 09:18 S11EwithSRU11-1 - - 44.22M static 2013-01-19 14:57 UpgradedToFCS - - 53.88M static 2013-01-19 20:07 before_168 - - 87.91M static 2011-07-02 18:46 snv_150 - - 49.48M static 2010-10-17 11:05 snv_150-bkup - - 6.85M static 2011-03-17 00:58 snv_150-bkup-1 - - 356.53M static 2011-07-02 18:47 snv_151a_ga - - 10.59M static 2011-03-17 11:57
…and we can follow that the next time the system reboots, it’ll be booted into the new boot environment named S11.1clone-1
This is just one example of the nice capabilities, but how do we actually measure the impact and usefulness of this to large enterprise companies? We’ve gone out and talked to customers about what these features mean to them. Based on the types of tasks they/you (and we) typically do, as contrasted to the time it takes to accomplish these tasks in Solaris 10, below is a single chart that illustrates the enormous amount of savings in terms of costs, effort and time that Oracle Solaris 11 has made possible.
Multiply these tasks by the quantity of tasks (as illustrated by some customers in Financial Services industry), by the time it takes to deliver updates to a single system, by the quantity of systems in your “cloud”, by the time gained by staff who can now focus on more critical aspects of business engineering – and you’re likely to ask yourself an important question.
And that is: aren’t these the types of financial and business benefits that DevOps is designed to bring out ?