How to fix “OPatch failed with error code 41” in Exadata X4 Machine


While applying a patch through “opatch”, the prerequisite check might fail with error “CheckActiveFilesAndExecutables” failed.
Here is an example of this issue and how it can be resolved. When I tried applying the patch to my database home, the prerequisite check failed and barked out saying that some of the executables were active. It also recommended that these files needs to be modified by OPatch but are currently being used by some processes.
Symptom:
oracle@bakuexa1dbadm01:/u01/app/oracle/diag/rdbms/misdev/MISDEV1/trace$ ps -afe | grep smon
 root 1536 1 0 Nov 07 ? 1227:13 /u01/app/11.2.0.3/grid/bin/osysmond.bin
 oracle 2260 1 0 Nov 07 ? 2:06 asm_smon_+ASM1
 oracle 69450 1 0 19:26:42 ? 0:03 ora_smon_MISUAT1
 oracle 68898 1 0 19:25:52 ? 0:02 ora_smon_MISLIVE1
 oracle 69964 1 0 19:27:26 ? 0:03 ora_smon_MISTEST1
 oracle 68364 1 0 19:25:05 ? 0:03 ora_smon_MISDEV1
 oracle 13643 94752 0 09:41:51 pts/9 0:00 grep smon
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ opatch prereq CheckApplicable -ph /data/17800518/
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/11.2.0.3/dbhome_1
Central Inventory : /u01/app/oraInventory
 from : /u01/app/oracle/product/11.2.0.3/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/opatch2015-02-23_17-32-05PM_1.log
Invoking prereq "checkapplicable"
Prereq "checkApplicable" for patch 17800518 passed.
OPatch succeeded.
oracle@bakuexa1dbadm01:/u01/app/11.2.0.3/grid/bin$ 
oracle@bakuexa1dbadm01:/u01/app/11.2.0.3/grid/bin$ cd /data/
oracle@bakuexa1dbadm01:/data$ ls
17800518 MISDEV_backup MISUAT_backup
crontab_backup_script MISLIVE_backup p17800518_1120325ExadataDatabase_Solaris86-64.zip
disk_space.sh MISTEST_backup PatchSearch.xml
oracle@bakuexa1dbadm01:/data$ 
oracle@bakuexa1dbadm01:/data$ cd 17800518/
oracle@bakuexa1dbadm01:/data/17800518$ 
oracle@bakuexa1dbadm01:/data/17800518$ 
oracle@bakuexa1dbadm01:/data/17800518$ cd $ORACLE_HOME
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1$ cd OPatch/
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ pwd
/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ 
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ 
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ cd /data/
oracle@bakuexa1dbadm01:/data$ cd 17800518/
oracle@bakuexa1dbadm01:/data/17800518$ /u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch/opatch apply
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0.3/dbhome_1
Central Inventory : /u01/app/oraInventory
 from : /u01/app/oracle/product/11.2.0.3/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/17800518_Feb_23_2015_19_08_06/apply2015-02-23_19-08-06PM_1.log
Applying interim patch '17800518' to OH '/u01/app/oracle/product/11.2.0.3/dbhome_1'
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
Log file location: /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/17800518_Feb_23_2015_19_08_06/apply2015-02-23_19-08-06PM_1.log
Recommended actions: OPatch needs to modify files which are being used by some processes.
OPatch failed with error code 41
Opsss problem. :(

Fix: find out which process is currently use it, and kill it. Though the database and the listener were brought down before applying the patch, OPatch gave out this error. I tried to find out which process was using the executable “$ORACLE_HOME/bin/oracle” using the “/sbin/fuser” command.
 
oracle@bakuexa1dbadm01:/data/17800518$ /sbin/fuser /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1: 48668m 48486m 110763m 34696m
oracle@bakuexa1dbadm01:/data/17800518$ ps -ef | grep 48668
 oracle 65328 61794 0 19:14:19 pts/18 0:00 grep 48668
 oracle 48668 110736 0 18:33:22 pts/11 0:34 /u01/app/oracle/product/11.2.0.3/dbhome_1/jdk/bin/amd64/java -server -Xmx384M -
oracle@bakuexa1dbadm01:/data/17800518$ kill -9 48668
oracle@bakuexa1dbadm01:/data/17800518$ /sbin/fuser /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1: 48486m 34696m
oracle@bakuexa1dbadm01:/data/17800518$ ps -ef | grep 48486
 oracle 48486 34671 0 18:32:49 pts/7 0:37 /u01/app/oracle/product/11.2.0.3/dbhome_1/jdk/bin/amd64/java -server -Xmx384M -
 oracle 65929 61794 0 19:15:51 pts/18 0:00 grep 48486
oracle@bakuexa1dbadm01:/data/17800518$ kill -9 48486
oracle@bakuexa1dbadm01:/data/17800518$ /sbin/fuser /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1: 34696m
oracle@bakuexa1dbadm01:/data/17800518$ ps -ef | grep 34696
 oracle 65978 61794 0 19:16:10 pts/18 0:00 grep 34696
oracle@bakuexa1dbadm01:/data/17800518$ /sbin/fuser /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1:
oracle@bakuexa1dbadm01:/data/17800518$ cd $ORACLE_HOME 
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1$ cd OPatch/
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ pwd
/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ 
oracle@bakuexa1dbadm01:/u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch$ cd /data/
oracle@bakuexa1dbadm01:/data$ ls
17800518 MISDEV_backup MISUAT_backup
crontab_backup_script MISLIVE_backup p17800518_1120325ExadataDatabase_Solaris86-64.zip
disk_space.sh MISTEST_backup PatchSearch.xml
oracle@bakuexa1dbadm01:/data$ cd 17800518/
oracle@bakuexa1dbadm01:/data/17800518$ 
oracle@bakuexa1dbadm01:/data/17800518$ /u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch/opatch apply
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0.3/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.0.3/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/17800518_Feb_23_2015_19_16_51/apply2015-02-23_19-16-51PM_1.log
Applying interim patch '17800518' to OH '/u01/app/oracle/product/11.2.0.3/dbhome_1'
Verifying environment and performing prerequisite checks...
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'exa1dbadm02' 
Local node: 'exa1dbadm01'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.2.0.3/dbhome_1')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.rdbms.rsf, 11.2.0.3.0...
Patching component oracle.rdbms, 11.2.0.3.0...
Verifying the update...
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
The node 'exa1dbadm02' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'exa1dbadm02'.
(Oracle Home = '/u01/app/oracle/product/11.2.0.3/dbhome_1')
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'exa1dbadm02' 
 Apply-related files are:
 FP = "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_files.txt"
 DP = "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_dirs.txt"
 MP = "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/make_cmds.txt"
 RC = "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11.2.0.3/dbhome_1/.patch_storage/17800518_Feb_21_2015_04_34_55/rac/make_cmds.txt" with actual path.
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/network/lib; /usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk isetasmgid ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ikfod ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk irenamedg ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'exa1dbadm02': 
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 || echo REMOTE_MAKE_FAILED::>&2
The node 'exa1dbadm02' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'bakuexa1dbadm02' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/11.2.0.3/dbhome_1/network/lib; /usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk isetasmgid ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ikfod ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk irenamedg ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
cd /u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
'
Patch 17800518 successfully applied
Log file location: /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/17800518_Feb_23_2015_19_16_51/apply2015-02-23_19-16-51PM_1.log
OPatch succeeded.
And yes, opatch succeeded in applying the patch successfully to the database home. :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: