ORA-00494: enqueue [CF] held for too long (more than 900 seconds)


Problem:

Today morning one of our development instance crashed due to following error found in alert log file.

Contents of alert log file:

ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66057/ANARDB_ora_17926_i66057.trc

Wed Apr 08 10:44:23 2015

Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)

 by killing session 1345.42449

Attempt to get Control File Enqueue by USER pid=17926 (mode=S, type=0, timeout=900) is being blocked by inst=1, pid=17923

Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs

Wed Apr 08 10:46:06 2015

minact-scn: useg scan erroring out with error e:12751

Suspending MMON action 'Block Cleanout Optim, Undo Segment Scan' for 82800 seconds

Wed Apr 08 10:46:06 2015

Sweep [inc][66057]: completed

Sweep [inc2][66057]: completed

Wed Apr 08 10:49:01 2015

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_ora_29462.trc (incident=66017):

ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66017/ANARDB_ora_29462_i66017.trc

Wed Apr 08 10:49:23 2015

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_ora_17926.trc (incident=66058):

ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66058/ANARDB_ora_17926_i66058.trc

Wed Apr 08 10:49:31 2015

Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=29462)

 by killing session 1345.42449

Attempt to get Control File Enqueue by USER pid=29462 (mode=S, type=0, timeout=900) is being blocked by inst=1, pid=17923

Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs

Wed Apr 08 10:49:33 2015

Sweep [inc][66058]: completed

Sweep [inc][66017]: completed

Sweep [inc2][66017]: completed

Wed Apr 08 10:49:53 2015

Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)

 by terminating the process

Attempt to get Control File Enqueue by USER pid=17926 (mode=S, type=0, timeout=300) is being blocked by inst=1, pid=17923

Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs

Wed Apr 08 10:50:33 2015

Sweep [inc2][66058]: completed

Wed Apr 08 10:54:31 2015

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_ora_29462.trc (incident=66018):

ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66018/ANARDB_ora_29462_i66018.trc

Wed Apr 08 10:55:01 2015

Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=29462)

 by terminating the process

Attempt to get Control File Enqueue by USER pid=29462 (mode=S, type=0, timeout=300) is being blocked by inst=1, pid=17923

Please check inst 1's alert log for more information on the blocker including a possible ORA-00494 and related incident logs

Wed Apr 08 10:56:15 2015

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_m000_5788.trc:

ORA-12751: cpu time or run time policy violation

Wed Apr 08 10:59:16 2015

Sweep [inc][66018]: completed

Sweep [inc2][66018]: completed

Wed Apr 08 11:04:53 2015

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_ora_17926.trc (incident=66059):

ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 17923'

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66059/ANARDB_ora_17926_i66059.trc

Wed Apr 08 11:05:23 2015

Killing enqueue blocker (pid=17923) on resource CF-00000000-00000000 by (pid=17926)

 by terminating the process

Errors in file /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/trace/ANARDB_ora_17926.trc (incident=66060):

ORA-00239: timeout waiting for control file enqueue: held by 'inst 1, osid 17923' for more than 900 seconds

Incident details in: /u01/app/oracle/diag/rdbms/ANARDB/ANARDB/incident/incdir_66060/ANARDB_ora_17926_i66060.trc

opiodr aborting process unknown ospid (17926) as a result of ORA-603

Cause:-

The Possible reasons for this sudden instance crash are:

Database running out of memory

Due to high load,server became unresponsive

Server is running out of memory and/or swap.

High concurrency on resources, IO waits and contention

Solution:-

In our development database, we reviewed the alert log and trace files related to this ORA error and we found out that there was high load on the Linux server due to RMAN hot backup running, export of a huge schema going on simultaneously.

At the time of this instance crash, there was very high load at the server and that's why the kill blocker interface killed the background process and so the instance crashed.

In order to prevent a background blocker from being killed, you can set the following init.ora parameter to 1 (default is 3).

_kill_enqueue_blocker=1

If you want to avoid the kill of the blocker (background or non-background process) you can set _kill_controlfile_enqueue_blocker=false.

With these parameter, if the enqueue holder is a background process, then it will not be killed, therefore the instance will not crash

Note:- Don't set undocumented hidden parameters until advise by oracle support.

References:-

Database Crashes With ORA-00494 [ID 753290.1]

Ora-00494: Enqueue [Cf] Held For Too Long Causing Database To Crash [ID 1101862.1]

PHYSICAL: ORA-00494: enqueue [CF] held for too long after Node Crash [ID 747071.1]

ORA-00494 Or ORA-600 [2103] During High Load After 10.2.0.4 Upgrade (Doc ID 779552.

 

1)
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: