DevOps system updates simplified with Solaris 11

One of the computing industry’s dirtiest little secrets is that the total time spent nurturing a system is largely comprised of the time it costs to maintain it vs. the time it takes to deploy it.  Managing change, delivering updates, rolling back due to unforeseen circumstances or scripts gone astray — all are just examples of challenges that DevOps simplifies.  And thanks to the sanity affirmed by software engineering professionals like Jim Bird, maintenance and devops are inseparable.

With this blog entry I thought it’d be useful to take a hands-on look at the notion of delivering an update bundle, comprised of 200  software packages, on a Solaris 11 system that was initially installed in October 2010.  Back then, nearly 3 and a half years ago, there was a version of Solaris  that contained preview technologies that were made available prior to the generally available release of Solaris 11 — it was called Solaris 11 Express.

So the system at hand has actually been going through quite a few updates over the last 3.5 years – with the first installation taking place in Oct 2010, with various updates having been applied since then.

isaac@HPensolaris:~$ beadm list
BE              Active Mountpoint Space   Policy Created
--              ------ ---------- -----   ------ -------
151a-july11     -      -          12.49M  static 2011-07-18 12:54
151a-july11-1   -      -          15.34M  static 2011-09-19 15:12
FCS             -      -          79.24M  static 2013-01-19 15:46
S11.1           -      -          16.37M  static 2013-01-20 04:25
S11.1clone      NR     /          23.68G  static 2013-01-20 22:36
S11EwithSRU11   -      -          60.93M  static 2011-10-10 09:18
S11EwithSRU11-1 -      -          44.22M  static 2013-01-19 14:57
UpgradedToFCS   -      -          53.88M  static 2013-01-19 20:07
before_168      -      -          87.91M  static 2011-07-02 18:46
snv_150         -      -          49.48M  static 2010-10-17 11:05
snv_150-bkup    -      -          6.85M   static 2011-03-17 00:58
snv_150-bkup-1  -      -          356.53M static 2011-07-02 18:47
snv_151a_ga     -      -          10.59M  static 2011-03-17 11:57

Notice the right-most column containing the creation date; these are boot environments that are introduced with Solaris 11.  Boot environments are implemented on top of ZFS snapshots, and – because of engineering with IPS (image packaging system) they provide immensely useful benefits that simplify change management, offer ability to have reversibility, compliance validation, risk assessment, and more.

The second column (from left) contains mostly dashes, except for “NR” on the 5th line in the output. This means that the corresponding boot environment is currently active (“N”), and will be the one selected after a system reboots (“R”). (The “R” assignments can arbitrarily be changed, naturally).

So what one can glean from the output above is that the currently active boot environment is actually NOT the one that the system was initially deployed with, and that from its name, it suggests that its a Oracle Solaris 11.1 release.

isaac@HPensolaris:~$ cat /etc/release
                            Oracle Solaris 11.1 X86
 Copyright (c) 1983, 2012, Oracle and/or its affiliates.  All rights reserved.
                          Assembled 19 September 2012

So let’s go ahead and attempt to update the system

isaac@HPensolaris:~$ pkg update

pkg update: Insufficient access to complete the requested operation.
Please try the operation again as a privileged user.

Oh right, can’t apply system changes by default without appropriate privileges.

isaac@HPensolaris:~$ su
Password:
isaac@HPensolaris:~# pkg update

pkg update: Certificate '/var/pkg/ssl/c892869bfaaf335a2399fcd813b6bbef9126cbc4' for publisher 'solaris' needed to access 'https://pkg.oracle.com/solaris/support/', has expired.  Please install a valid certificate.
isaac@HPensolaris:~# cd /var/pkg/ssl
isaac@HPensolaris:/var/pkg/ssl# ls -la *.pem
-rw-r--r--   1 isaac    staff        737 Jan 19  2013 Oracle_Solaris_11_Support.certificate.pem
-rw-r--r--   1 isaac    staff        887 Jan 19  2013 Oracle_Solaris_11_Support.key.pem

Ah, that means that the certificates that pull updates from pkg.oracle.com have expired because they were good for 1 year – and we’re now in February 2014.

Of course, the publisher is set and can be verified via pkg(1):

isaac@HPensolaris:/var/pkg/ssl# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F https://pkg.oracle.com/solaris/support/

With this info at hand, going over to pkg-register.oracle.com to login with my Oracle SSO and grabbing updated certificate and key associated with the Oracle Solaris 11 Support Repository, placing them in /var/pkg/ssl and running:

isaac@HPensolaris:~# pkg set-publisher -k /var/pkg/ssl/Oracle_Solaris_11_Support.key.pem  -c /var/pkg/ssl/Oracle_Solaris_11_Support.certificate.pem  -G '*' -g https://pkg.oracle.com/solaris/support/ solaris

Now, re-attempting the update process:

isaac@HPensolaris:~# pkg update
          Packages to install:   4
           Packages to update: 198
          Mediators to change:   1
      Create boot environment: Yes
Create backup boot environment:  No

pkg: The following packages require their licenses to be accepted before they can be installed or updated:
----------------------------------------
Package: pkg://solaris/runtime/java/jre-6@1.6.0.71.12,5.11:20131230T162629Z

License: LICENSE
 License requires acceptance.

To indicate that you agree to and accept the terms of the licenses of the packages listed above, use the --accept option.  To display all of the related licenses, use the --licenses option.

Cool – making progress. This shows us that there are 198 packages with available updates, and 4 new packages that will be installed.  All of the changes will be recorded into a newly created boot environment, leaving the current boot environment unmodified. This means we can deliver the updates while knowing that we can boot back to the unmodified boot environment, in cases of failures validating the updated system. Naturally, all of this would be completely automated, but going through this process interactively hopefully indicates the types of gotchas that may arise, whilst demonstrating the goodness of what’s available in the OS already.  Rant: I wish that more OSes would have this kind of an approach to delivering updates, including Apple’s iOS. I really do miss some of the intricacies of  iOS 6 on some of my devices, and having no way to fall back onto iOS 6, even periodically, puzzles me – while reminding me of the adoption issues Microsoft experienced when users attempted to run  Windows NT 3.5 on Intel’s 486-based systems, whilst most of the development and user-acceptance testing was done on Pentium/586-based systems. Newer systems run  newer code faster then slower systems, d’uh. But we all live and learn from each other’s mistakes, hopefully – and that’s one of the fun parts of life.  Moving on …

So if we’re scripting the updating process – the lesson here is to include “–accept”, like so:

isaac@HPensolaris:~# pkg update --accept

but to first and foremost, verify the certificates/keys are good, and try to do so at least a month prior to their expiration. Who knows what technical issues might arise getting them renewed, and you don’t want to wait until the last minute to initiate the renewal process. Start 11 months after they’ve been issued, automate the process, and go from there.  Here’s the correct, expected output.

isaac@HPensolaris:~# pkg update --accept
          Packages to install:   4
           Packages to update: 198
          Mediators to change:   1
      Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            202/202   15923/15923  441.2/441.2  717k/s

PHASE                                          ITEMS
Removing old actions                       1268/1268
Installing new actions                     2185/2185
Updating modified actions                17628/17628
Updating package state database                 Done
Updating package cache                       198/198
Updating image state                            Done
Creating fast lookup database                   Done
Reading search index                            Done
Building new search index                    833/833

A clone of S11.1clone exists and has been updated and activated.
On the next boot the Boot Environment S11.1clone-1 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://www.oracle.com/pls/topic/lookup?ctx=E26502&id=SERNS
---------------------------------------------------------------------------

Paying close attention to the comment about S11.1clone-1

We can actually see a new boot environment having been created:

isaac@HPensolaris:~# beadm list
BE              Active Mountpoint Space   Policy Created
--              ------ ---------- -----   ------ -------
151a-july11     -      -          12.49M  static 2011-07-18 12:54
151a-july11-1   -      -          15.34M  static 2011-09-19 15:12
FCS             -      -          79.24M  static 2013-01-19 15:46
S11.1           -      -          16.37M  static 2013-01-20 04:25
S11.1clone      N      /          378.5K  static 2013-01-20 22:36
S11.1clone-1    R      -          25.70G  static 2014-02-16 10:09
S11EwithSRU11   -      -          60.93M  static 2011-10-10 09:18
S11EwithSRU11-1 -      -          44.22M  static 2013-01-19 14:57
UpgradedToFCS   -      -          53.88M  static 2013-01-19 20:07
before_168      -      -          87.91M  static 2011-07-02 18:46
snv_150         -      -          49.48M  static 2010-10-17 11:05
snv_150-bkup    -      -          6.85M   static 2011-03-17 00:58
snv_150-bkup-1  -      -          356.53M static 2011-07-02 18:47
snv_151a_ga     -      -          10.59M  static 2011-03-17 11:57

…and we can follow that the next time the system reboots, it’ll be booted into the new boot environment named S11.1clone-1

This is just one example of the nice capabilities, but how do we actually measure the impact and usefulness of this to large enterprise companies?  We’ve gone out and talked to customers about what these features mean to them.  Based on the types of tasks they/you (and we) typically do, as contrasted to the time it takes to accomplish these tasks in Solaris 10, below is a single chart that illustrates the enormous amount of savings in terms of costs, effort and time that Oracle Solaris 11 has made possible.

benefits-updating

Multiply these tasks by the quantity of tasks (as illustrated by some customers in Financial Services industry), by the time it takes to deliver updates to a single system, by the quantity of systems in your “cloud”, by the time gained by staff who can now  focus on more critical aspects of business engineering – and you’re likely to ask yourself an important question.

And that is:  aren’t these the types of financial and business benefits that DevOps is designed to bring out ?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: