ORA-00494: enqueue [CF] held for too long (more than 900 seconds)


Problem: 
Today morning one of our development instance crashed due to following error found in alert log file.
Contents of alert log file:
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66057/AMLPROD_ora_17926_i66057.trc
Wed Apr 08 10:44:23 2015
Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)
 by killing session 1345.42449
Attempt to get Control File Enqueue by USER pid=17926 (mode=S, type=0, timeout=900) is being blocked by inst=1, pid=17923
Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs
Wed Apr 08 10:46:06 2015
minact-scn: useg scan erroring out with error e:12751
Suspending MMON action 'Block Cleanout Optim, Undo Segment Scan' for 82800 seconds
Wed Apr 08 10:46:06 2015
Sweep [inc][66057]: completed
Sweep [inc2][66057]: completed
Wed Apr 08 10:49:01 2015
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_ora_29462.trc (incident=66017):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66017/AMLPROD_ora_29462_i66017.trc
Wed Apr 08 10:49:23 2015
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_ora_17926.trc (incident=66058):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66058/AMLPROD_ora_17926_i66058.trc
Wed Apr 08 10:49:31 2015
Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=29462)
 by killing session 1345.42449
Attempt to get Control File Enqueue by USER pid=29462 (mode=S, type=0, timeout=900) is being blocked by inst=1, pid=17923
Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs
Wed Apr 08 10:49:33 2015
Sweep [inc][66058]: completed
Sweep [inc][66017]: completed
Sweep [inc2][66017]: completed
Wed Apr 08 10:49:53 2015
Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)
 by terminating the process
Attempt to get Control File Enqueue by USER pid=17926 (mode=S, type=0, timeout=300) is being blocked by inst=1, pid=17923
Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs
Wed Apr 08 10:50:33 2015
Sweep [inc2][66058]: completed
Wed Apr 08 10:54:31 2015
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_ora_29462.trc (incident=66018):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66018/AMLPROD_ora_29462_i66018.trc
Wed Apr 08 10:55:01 2015
Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=29462)
 by terminating the process
Attempt to get Control File Enqueue by USER pid=29462 (mode=S, type=0, timeout=300) is being blocked by inst=1, pid=17923
Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs
Wed Apr 08 10:56:15 2015
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_m000_5788.trc:
ORA-12751: cpu time or run time policy violation
Wed Apr 08 10:59:16 2015
Sweep [inc][66018]: completed
Sweep [inc2][66018]: completed
Wed Apr 08 11:04:53 2015
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_ora_17926.trc (incident=66059):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66059/AMLPROD_ora_17926_i66059.trc
Wed Apr 08 11:05:23 2015
Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)
 by terminating the process
Errors in file /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/trace/AMLPROD_ora_17926.trc (incident=66060):
ORA-00239: timeout waiting for control file enqueue: held by 'inst 1, osid 17923' for more than 900 seconds
Incident details in: /u01/app/oracle/diag/rdbms/amlprod/AMLPROD/incident/incdir_66060/AMLPROD_ora_17926_i66060.trc
opiodr aborting process unknown ospid (17926) as a result of ORA-603
Cause:-
The Possible reasons for this sudden instance crash are:
Database running out of memory
Due to high load,server became unresponsive
Server is running out of memory and/or swap.
High concurrency on resources, IO waits and contention
Solution:-
In our development database, we reviewed the alert log and trace files related to this ORA error and we found out that there was high load on the Linux server due to RMAN hot backup running, export of a huge schema going on simultaneously.
At the time of this instance crash, there was very high load at the server and that's why the kill blocker interface killed the background process and so the instance crashed.
In order to prevent a background blocker from being killed, you can set the following init.ora parameter to 1 (default is 3).
_kill_enqueue_blocker=1
If you want to avoid the kill of the blocker (background or non-background process) you can set _kill_controlfile_enqueue_blocker=false.
With these parameter, if the enqueue holder is a background process, then it will not be killed, therefore the instance will not crash
Note:- Don't set undocumented hidden parameters until advise by oracle support.
References:-
Database Crashes With ORA-00494 [ID 753290.1]
Ora-00494: Enqueue [Cf] Held For Too Long Causing Database To Crash [ID 1101862.1] 
PHYSICAL: ORA-00494: enqueue [CF] held for too long after Node Crash [ID 747071.1]
ORA-00494 Or ORA-600 [2103] During High Load After 10.2.0.4 Upgrade (Doc ID 779552.1)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: