Copying an Oracle software Installation under Linux

Pull Is Easier, Push Is Better - Part 1

Copying an Oracle software installation under Linux/Unix

by Mark Bole

"Pull is Easier, Push Is Better" — what does that mean? Following is a practical illustration of this concept in the context of managing Oracle software installations.

Oracle provides a comprehensive "universal" installer (OUI) for its database products. This graphical tool has an identical appearance across all platforms, is fairly easy to use, and of course is the only method supported by Oracle for installing its products. It includes a so-called "silent install" method, where instead of interactively clicking through the installer screens and menus, you can provide a pre-determined set of responses to control installer actions. Further, the OUI normally runs a system prerequisite check, maintains an inventory of all the various required and optional components that have been installed, and keeps detailed log files for each installation.

So, why would one even think of not using it? It turns out there are some situations where you may be better off going “outside of the Oracle box” when it comes to repeat installations under Unix. This article describes the reasons why, and how to effectively use the OUI in conjunction with the standard Unix tool rsync for a more trouble-free, maintainable set of Oracle server installations.

Using the OUI, For Better and For Worse

Ralph Waldo Emerson said "A foolish consistency is the hobgoblin of little minds". He never defined what a "wise" consistency was, but it's guaranteed that inconsistency of almost any kind is the hobgoblin of computing systems. Despite the best efforts of Quality Assurance teams, the countless hours of regression testing, the small fortunes spent on multiple, redundant computing environments, how many times does a production outage occur because of one little change that someone neglected to replicate during the last rollout? How many times has a system hummed along nicely for nearly a year of 24x7 operation, only to choke and sputter when the next re-boot occurs, because a supposedly permanent configuration change was not made permanently after all? I have received enough 2 AM pager calls to know that the numbers are high—too high, considering that there is a straightforward solution available.

Here is a real-life problem I experienced a while back: several DBA's split the work of installing an Oracle upgrade on a number of servers. Everything was fine, in both test and production environments. But months later, a routine upgrade of Perl software suddenly resulted in a script that mysteriously worked on some servers but not others, even though everything was identical. Eventually the problem was found to be related to the following differences in Oracle installation files:

server 1: =========================
      9177672 Nov 17 2000 $ORACLE_HOME/lib/
          812 Oct 24 1999 $ORACLE_HOME/lib/nautab.o
server 2: =========================
      9928452 Nov 17 2000 $ORACLE_HOME/lib/
         1272 Jun  8 2000 $ORACLE_HOME/lib/nautab.o

Can you tell what happened, or how to fix it? Turns out that RADIUS authentication support, even though unused at this site, was included in the OUI session by one DBA and not the other. An obscure bug then triggered the problem with the new Perl version. It took several calendar months to finally figure it out, meanwhile many hours were wasted on implementing workarounds and other futile trouble-shooting activities.

The obvious solution would be to use a common "silent install" response file everywhere. But there are other drawbacks that won't so easily be dispensed with:

The latest version of OUI (10.1) includes brand new features to help address some of these items, such as the ability to record a response file from an OUI session, the ability to clone an ORACLE_HOME, and the ability to access the install media over an HTTP connection. But to use these new features requires a non-trivial amount of your time to review documentation, test, and maintain, once you've decided to upgrade to Oracle 10g. It's important to remember that you will always use OUI at least once, for your primary installation. What is presented here is an alternative way to duplicate and validate subsequent installations with relatively little effort and overhead.

A note about Windows: even angels fear to tread in the registry, so this article only applies to Linux and Unix systems, where it's simply a matter of getting all the files and directories in the right place, and there are no drive letters to complicate your ORACLE_HOME location.

It's all About Controlling the Source

Software developers learned decades ago that it's critical to maintain a central repository where all new and changed code is stored and versioned. No matter how many individual programmers, no matter how many individual files changed, there is one and only one place where everything officially ends up. Inconsistencies must be resolved in order to compile; the compilation then becomes a known, reproducible entity which can be tested, debugged, and eventually rolled out to production. The whole point is, once a bug is fixed or new capability implemented, it should never have to be fixed or implemented again, at least not in the same place in the code. Done properly and supported by team managers, there never is any question as to what was in any given version of the code.

Database administrators (DBA's) and their system administrator (SA) brethren must learn and adopt the same practice to avoid hobgoblins. There is a slight twist, however, in how DBA's and SA's use this technique: instead of changes coming in from many places to a single destination repository, a single source is created and maintained, and then used to replicate changes out to many targets. This source (which I will call a "push" rather than a repository) should be kept on a single administrative server, preferably dedicated to this function. This server needs very little computing power, and typically not even that much disk space, so an older system, or even a workstation-class machine, will do.

The second part of this series ("Pull is Easier, Push Is Better Part 2") discusses in more detail how such a server can handle not only generic distributions of system software, such as Oracle and Perl, but also configuration files, including those that need to be almost the same, but not quite, on each server (such as the Oracle init.ora file or the Apache httpd.conf file). But for the purpose of this topic, the requirements for this server are simple and few:

Pushing the First Time

First, decide on an ORACLE_HOME location for the version of software you will be maintaining (of course, multiple versions can be maintained as long as they are in different ORACLE_HOME's). It is critical to use the exact same ORACLE_HOME per version on every server where you run Oracle.

If necessary, use symbolic links to meet this requirement. For example, suppose you choose /opt/oracle/product/9.2.0 as the standard ORACLE_HOME. On a target server, you don't have enough room in the filesystem where /opt is, but you do have plenty of room in the filesystem mounted at /disk2. As user root on the target machine, perform these three steps to make the ORACLE_HOME directory, change its ownership, and lastly, link it symbolically to the correct top-level directory.

mkdir -p /disk2/oracle/product/9.2.0
chown -R oracle:dba /disk2/oracle
ln -s /disk2/oracle /opt

Even if you don't need to use symbolic links, make sure the ORACLE_HOME directory already exists on the target server.

Confirm that your primary, or push, server has a correctly installed (using the OUI) version of Oracle, with the components and patches you have decided to use. Also be sure that the ORACLE_HOME used for the OUI installation was indeed your standard location. You might create a fresh installation, or use an existing "known good" installation, such as production or a valid test server.

The actual push process consists mostly of running a single rsync command. I have found that rsync, like any power tool, has the capability to do a lot of harm very quickly if not used carefully. While not intended to be a tutorial, here are some points to keep in mind about rsync to stay out of trouble:

If there is a database running on the target machine, using the ORACLE_HOME you want to push to, shut it down first. Then run this command as user root on the push server (if you are using ssh as your transport for rsync, you may have to temporarily enable connections by root):

% rsync --dry-run -avH --exclude-from=ora_home_exclude --delete \
   /opt/oracle/product/9.2.0 target:/opt/oracle/product | tee rsync.out

(This assumes you have the file ora_home_exclude in place—more on this next). You will be prompted for the password, and then see output similar to the following, which is also stored in the rsync.out file you used with the tee command:

root@target's password:
building file list ... done
[many lines omitted...] 9.2.0/xdk/mesg/lsxus.msb
wrote 2068372 bytes read 238492 bytes 54279.15 bytes/sec
total size is 2218371456 speedup is 961.64

Let's briefly review the purpose of the rsync options used. --dry-run has already been discussed. -v is for verbose output, -a keeps timestamps, file ownership and permissions consistent (which is why we run this as user root instead of oracle, since some of the Oracle files have special permissions and ownership), and -H preserves hard links. --delete actually removes files from the target that don't exist in the source (normally only adds and updates are performed by rsync). As the documentation states, use with caution! But in this case, since we want consistent ORACLE_HOME's, it is appropriate.

What To Exclude From the Push

The --exclude-from option of rsync is very important. One of the most powerful features of rsync is the ability to include and exclude files and directories based on patterns. This is necessary with the ORACLE_HOME push because, despite the best intentions of Oracle's Optimal Flexible Architecture (OFA), there are still a number of server-specific files that reside within the confines of the software installation tree. In other words, ideally you should be able to completely remove the software installation and re-install without disturbing files containing server-specific configuration or logs—but such is not the case.

The main types of files we need to exclude are log files, audit files, and configuration (.ora) files. The dbs directory under ORACLE_HOME is also critical to exclude, since it contains the init.ora file(s) and the shared memory lock file(s). These are files that belong only on the machine where they were first created, not on any other machine. Here are the contents of the ora_home_exclude file used above with the --exclude-from option of rsync:

# for rsync'ing ORACLE_HOME installations

What you put in this file is determined empirically, using the rsync command itself in --dry-run mode and, optionally, the find command (some examples are given below). Following the rules in the rsync documentation, any files and directories matching these patterns will be excluded from the push. This applies both on the source and target, so even if your push does not have these files because there is no running database instance to generate them, they will still be excluded at the target.

Testing and Rectifying the Push and Target

One of the benefits of the push, which helps qualify it as the Fast-and-Hard Way, is that you can re-run it and test it quickly, usually by just recalling a command from your Unix shell history list. For example, suppose we have proceeded with the actual push by running the following command, this time without the --dry-run option that was shown above. Since in this case we are copying to a machine with no existing ORACLE_HOME, we have nothing to lose—at worst, we can always erase the target and do it again.

% rsync -avH --exclude-from=ora_home_exclude --delete \
   /opt/oracle/product/9.2.0 target:/opt/oracle/product | tee rsync.out

Since this will typically include between one and three gigabytes of files for a first-time push, it may take a while, depending on your hardware (use du -sk $ORACLE_HOME on the push server to see approximately how big the push will be). When finished, we can now see exactly what was excluded by running the following comparison (rectify) command. This is a good way to compare any two directory trees on any local or remote Unix server, not just ORACLE_HOME directory trees!

% rsync --dry-run -avH --delete \
   /opt/oracle/product/9.2.0 target:/opt/oracle/product
root@target's password: 
building file list ... done
[lines omitted...]
9.2.0/dbs/initmydb.ora -> ../../../admin/mydb/pfile/initmydb.ora
[lines omitted...]
wrote 1836163 bytes  read 404 bytes  66784.25 bytes/sec
total size is 2218960063  speedup is 1208.21

This shows that the files we expected to be excluded (log and audit files) were in fact excluded, plus we can see if any files that should have been included, were not. If you determine for your site that any specific file should be included or excluded, then add it to the ora_home_exclude and re-run rsync again.

Updating the Push and Targets

Now you can create the $ORACLE_HOME/dbs directory on the target machine, place the appropriate init.ora file, and create your database. Suppose you then try to run the Oracle Net Creation Assistant on the target machine, and get the following error:

% $ORACLE_HOME/bin/netca
  /u01/app/oracle/product/9.2.0/bin/netca: line 138:
  No such file or directory

Recall the information about rsync and the --copy-links option above. The OUI installs a Java home outside of the ORACLE_HOME directory and creates a symbolic link for it, as shown here on the push installation:

/opt/oracle/product/9.2.0/JRE -> /opt/oracle/jre/1.1.8

The update is simple, after you perform the usual --dry-run test and log the output:

rsync -avH --exclude-from=ora_home_exclude --delete \
  /opt/oracle/jre  target:/opt/oracle

Or, for another example, suppose you discover that there are incorrect permissions on the Intelligent Agent executable after a patch install:

-rwxr-xr-x 1 oracle dba 2522483 Dec 09 00:01 bin/dbsnmp*
-rwsr-s--- 1 root   dba 2472673 Apr 26 2002  bin/dbsnmp.old*

Note that the new dbsnmp executable did not pick up the same set-uid root ownership as the previous version, after a one-off patch install to fix an installation bug under Linux. You only have to fix it once, at the push location, and then, even if you forget to update all targets at that time, the push will "remember" the fix and ensure that it is propagated whenever you do subsequently perform a push.

As time goes by, suppose you do a dry run test and find that the following two files suddenly show up as wanting to be pushed (or, deleted on the target):

building file list ... done

It is straight-forward to investigate these files, determining their contents and timestamps, to see if they should be excluded or not. As a DBA or SA, this has the side benefit of being very valuable audit information, in the same vein as Tripwire—many times, I have been able to head off problems or inquire as to what my fellow admins have been doing by finding such new or changed files on either the push or the target. In this case, both files should be excluded, since the first is a result of running the Intelligent Agent, and the second is the result of running Net Creation Assistant, both of which are specific to just one server.

A supplemental approach to proactively managing the ORACLE_HOME tree is to use the find command, looking for individual files that have been modified since a certain time. In the following example, we find all the files since adding the link for the init.ora file, which is one suitable proxy for the most recent time your database configuration was updated:

% cd $ORACLE_HOME; find . -newer dbs/initmydb.ora
[lines omitted...]

These files are all candidates for being excluded in the ora_home_exclude file. However, many of them may already be included or excluded by the rsync push process, so evaluate this output in the context of a push.


Five shortcomings of the OUI process were identified. Using the standard Linux and Unix rsync command, a method for reliably cloning ORACLE_HOME's was presented, which offers improvements in all five areas. Through judicious use of the --exclude-from and --dry-run options, the power of rsync can be tamed in the service of multiple Oracle software installations.


Good Hacker Emblem