Exadata: socket open error: Port no: 8888. Received errorno 111. Connection refused


When I checked the Exadata cell storage log Alert seen such an error : MS process is not alive. Pid is missing.

Exadata Cell Storage Alert log:

[root@bakuexa1celadm03 trace]# cat rstrc_15083_mmt.trc 
Trace file /opt/oracle/cell/log/diag/asm/cell/bakuexa1celadm03/trace/rstrc_15083_mmt.trc
ORACLE_HOME = /opt/oracle/cell
System name: Linux
Node name: bakuexa1celadm03.kfsaz.local
Release: 2.6.39-400.264.6.el6uek.x86_64
Version: #1 SMP Tue Dec 1 16:41:42 PST 2015
Machine: x86_64

*** 2016-08-19 03:18:26.442
2016-08-19 03:18:26.442010 :00000002: Trace location after redirection: /opt/oracle/cell/cellsrv/deploy/log/cellrsmmt28163.trc
2016-08-19 03:18:26.442527 :0000000D: ossrsutl_parse_args: ossrsutl_core_rs_executable /opt/oracle/cell/cellsrv/bin/cellrssrm
2016-08-19 03:18:26.442545 :0000000E: Env Vars: 
OSS_BIN: /opt/oracle/cell/cellsrv/bin
 OSS_SCRIPTS_HOME: /opt/oracle/cell/cellsrv/deploy/scripts
OSS_CONFIG_HOME /opt/oracle/cell/cellsrv/deploy/config
LOG_HOME /opt/oracle/cell/cellsrv/deploy/log.
2016-08-19 03:18:26.442557 :0000000F: ADR_BASE: /opt/oracle/cell/log
 JAVA_HOME: /usr/java/default
SRCHOME /opt/oracle/cell
OSS_HOME /opt/oracle/cell/cellsrv.
2016-08-19 03:18:26.442567 :00000010: Conf files: 
ossrs file: /opt/oracle/cell/cellsrv/deploy/config/cellinit.ora
 ossrsms file: /opt/oracle/cell/cellsrv/deploy/config/cellrsms.state
ossrsos file: /opt/oracle/cell/cellsrv/deploy/config/cellrsos.state
2016-08-19 03:18:26.442577 :00000011: Args: 
ossrs debug: 0
ossrs testing: 0
2016-08-19 03:18:26.445008 :0000002D: Service MS has status 2, enable monitoring
2016-08-19 03:18:26.445029 :0000002E: Started monitoring process /opt/oracle/cell/cellsrv/bin/cellrsmmt with pid 28163
2016-08-19 03:18:26.445366 :00000030: mon_proc_pid oldpid: 0
2016-08-19 03:18:26.448920 :00000031: mon_proc_pid newpid: 0
2016-08-19 03:18:26.448949 :00000032: MS process is not alive. Pid is missing.
2016-08-19 03:18:26.448965 :00000033: Missed a heartbeat for process MS or leaking memory, error: -75
2016-08-19 03:18:26.448976 :00000034: Service MS was not alive, try starting
2016-08-19 03:18:26.449148 :00000038: Exec new process /opt/oracle/cell/cellsrv/deploy/msdomain/bin/startWebLogic.sh
2016-08-19 03:18:26.449167 :00000039: Cmdline: /opt/oracle/cell/cellsrv/deploy/msdomain/bin/startWebLogic.sh 
2016-08-19 03:18:26.449182 :0000003A: Redirect STDOUT from process /opt/oracle/cell/cellsrv/deploy/msdomain/bin/startWebLogic.sh to MS (trace flag 3)
2016-08-19 03:18:26.449470 :0000003B: mon_proc_pid oldpid: 0
2016-08-19 03:18:26.449523 :0000003B: Trace location after redirection: /opt/oracle/cell/cellsrv/deploy/log/wls28165.trc
2016-08-19 03:18:26.452485 :0000003C: mon_proc_pid newpid: 0
2016-08-19 03:18:26.552590 :0000003D: mon_proc_pid oldpid: 0
2016-08-19 03:18:26.555905 :0000003E: mon_proc_pid newpid: 28232
2016-08-19 03:18:26.556159 :0000003F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:26.656242 :00000040: mon_proc_pid oldpid: 28232
2016-08-19 03:18:26.656401 :00000041: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:26.756491 :00000042: mon_proc_pid oldpid: 28232
2016-08-19 03:18:26.756822 :00000043: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:26.856929 :00000044: mon_proc_pid oldpid: 28232
2016-08-19 03:18:26.857329 :00000045: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:26.957435 :00000046: mon_proc_pid oldpid: 28232
2016-08-19 03:18:26.957846 :00000047: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.057902 :00000048: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.058331 :00000049: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.158441 :0000004A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.158865 :0000004B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.258978 :0000004C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.259404 :0000004D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.359514 :0000004E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.359918 :0000004F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.460038 :00000050: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.460414 :00000051: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.560515 :00000052: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.560931 :00000053: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.661036 :00000054: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.661433 :00000055: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.761538 :00000056: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.761930 :00000057: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.862035 :00000058: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.862439 :00000059: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:27.962545 :0000005A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:27.962906 :0000005B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.063010 :0000005C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.063365 :0000005D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.163468 :0000005E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.163886 :0000005F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.263988 :00000060: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.264376 :00000061: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.364486 :00000062: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.364875 :00000063: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.464981 :00000064: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.465394 :00000065: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.565505 :00000066: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.565929 :00000067: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.666042 :00000068: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.666443 :00000069: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.766551 :0000006A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.766961 :0000006B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.867067 :0000006C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.867478 :0000006D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:28.967586 :0000006E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:28.968009 :0000006F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.068117 :00000070: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.068557 :00000071: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.168668 :00000072: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.169084 :00000073: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.269193 :00000074: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.269628 :00000075: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.369740 :00000076: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.370182 :00000077: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.470291 :00000078: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.470745 :00000079: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.570858 :0000007A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.571301 :0000007B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.671422 :0000007C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.671815 :0000007D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.771932 :0000007E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.772270 :0000007F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.872369 :00000080: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.872790 :00000081: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:29.972904 :00000082: mon_proc_pid oldpid: 28232
2016-08-19 03:18:29.973314 :00000083: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.073419 :00000084: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.073834 :00000085: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.173941 :00000086: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.174348 :00000087: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.274457 :00000088: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.274873 :00000089: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.374980 :0000008A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.375444 :0000008B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.475584 :0000008C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.476008 :0000008D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.576113 :0000008E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.576490 :0000008F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.676599 :00000090: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.677000 :00000091: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.777107 :00000092: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.777588 :00000093: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.877697 :00000094: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.878097 :00000095: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:30.978204 :00000096: mon_proc_pid oldpid: 28232
2016-08-19 03:18:30.978606 :00000097: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.078710 :00000098: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.079089 :00000099: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.179196 :0000009A: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.179564 :0000009B: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.279676 :0000009C: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.279955 :0000009D: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.380056 :0000009E: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.380321 :0000009F: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.480413 :000000A0: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.480789 :000000A1: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.580883 :000000A2: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.581203 :000000A3: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.681305 :000000A4: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.681692 :000000A5: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.781794 :000000A6: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.782107 :000000A7: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.882195 :000000A8: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.882407 :000000A9: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:31.982503 :000000AA: mon_proc_pid oldpid: 28232
2016-08-19 03:18:31.982796 :000000AB: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.082924 :000000AC: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.083331 :000000AD: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.183441 :000000AE: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.183818 :000000AF: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.283927 :000000B0: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.284322 :000000B1: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.384432 :000000B2: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.384836 :000000B3: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.484947 :000000B4: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.485359 :000000B5: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.585469 :000000B6: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.585885 :000000B7: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.685993 :000000B8: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.686441 :000000B9: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.786551 :000000BA: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.786961 :000000BB: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.887072 :000000BC: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.887488 :000000BD: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:32.987596 :000000BE: mon_proc_pid oldpid: 28232
2016-08-19 03:18:32.988064 :000000BF: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.088168 :000000C0: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.088545 :000000C1: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.188652 :000000C2: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.189031 :000000C3: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.289172 :000000C4: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.289589 :000000C5: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.389694 :000000C6: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.390070 :000000C7: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.490174 :000000C8: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.490580 :000000C9: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.590687 :000000CA: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.591076 :000000CB: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.691179 :000000CC: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.691574 :000000CD: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.791690 :000000CE: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.792070 :000000CF: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.892176 :000000D0: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.892588 :000000D1: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:33.992709 :000000D2: mon_proc_pid oldpid: 28232
2016-08-19 03:18:33.993015 :000000D3: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:34.093108 :000000D4: mon_proc_pid oldpid: 28232
2016-08-19 03:18:34.093421 :000000D5: socket open error: Port no: 8888. Received errorno 111. Connection refused
2016-08-19 03:18:34.193519 :000000D6: mon_proc_pid oldpid: 28232
2016-08-19 03:18:34.193971 :000000D7: Started Service MS with pid 28232
2016-08-19 03:28:34.247566 :000000D8: Initial mem usage for ms 28232 at heartbeat 30 is 508 MB, memory usage ceil is 2040 MB
2016-08-19 05:15:15.001797 :000000D9: Next MS heartbeat in 19 seconds, since 1 of 20 already elapsed

Cheking in Cell storage status:

[root@bakuexa1celadm03 trace]# service celld status
 rsStatus: running
 msStatus: stopped
 cellsrvStatus: running

BOOOMMMM !!! What I see msStatus stopped

Solution:
Stop all the services and start the cell in order rs,ms and cellsrv service.

[root@bakuexa1celadm03 trace]# service celld stop

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.

[root@bakuexa1celadm03 trace]#cellcli -e "alter cell startup services rs"

Starting the RS services...
Getting the state of RS services... running

[root@bakuexa1celadm03 trace]#cellcli -e "alter cell startup services ms"

Starting MS services...
The STARTUP of MS services was successful.

[root@bakuexa1celadm03 trace]#cellcli -e "alter cell startup services cellsrv"

Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.

And check again:

[root@bakuexa1celadm01 ~]# service celld status
 rsStatus: running
 msStatus: running
 cellsrvStatus: running

Problem Solved !
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: