1870931 - Error RUEI40: Events stop when Oracle db service stops - Sybase CEP4.0

As events were being processed the Oracle's service was stopped, shortly after the Oracle's service was restarted, pending events to be processed were not started. The db table producer had to be restarted in order for the events to start processing.

The CEP Server & Event Insight logs shows the events occurrence, steps and date time for each action performed:

30/01/2013 14:27 - Rtds service regarding a graph have been started.
30/01/2013 14:27 - Started event processing.
30/01/2013 14:28 - Stopped Oracle's service.
30/01/2013 14:31 - Started Oracle's service.
30/01/2013 14:33 - Stopped and started DB table producer.

On monitoring the Oracle database connections it was noted that connection pool is not restored after a database down.

On starting the graph, a connection pool was created by c8server process. As there are not new incoming events, all connections are inactive and waiting state. A new JDBC connection was noticed to be opened and closed every minute (the poll period established in the DB table producer). When a new event is inserted into Oracle database, no new connections are opened.  C8server uses already established connections.

A scheduled Oracle downtime was simulated to test the impact again and when Oracle is restarted the active connections were noted exactly as seen before, the JDBC connections were also established again.
This time though the c8server connections were not reopened. The pool connection is lost. Only a few connections for c8server were found to be opened when a new event is inserted into database. Incoming event is not processed.

The only time that the connection pool is re-established again is when graph service or DB table producer was restarted.

Some errors found in the logs:

[2013/01/30 11:44:16.385433 +0100] Error C4721: Error while executing Remote Call: Database error: Unable to access database. Error Code: 1089

.......

...

[2013/01/30 11:44:14.622315 +0100] Warning C5001: Received and discarded out-of-order message. (ClockTime='1359542654622315', MaxTimestamp='1359542654609001', MessageTime='1359542654468004', QueueName='/Stream/Default/uid_12_me32b3e413c8afb7980m7d99/portIn_Event_Log')

[2013/01/30 11:44:15.656250 +0100] Warning C5001: Received and discarded out-of-order message. (ClockTime='1359542655656250', MaxTimestamp='1359542655640006', MessageTime='1359542655531001', QueueName='/Stream/Default/uid_12_me32b3e413c8afb7980m7d99/portIn_Event_Log')

[2013/01/30 11:44:16.000597 +0100] Warning C4706: Failed to rollback uncommited Database transaction(s) (if any). (Reason='Database error: Unable to access database. Error Code: 3114

 Reason: ORA-03114: no conectado a ORACLE

')

[2013/01/30 11:44:16.000597 +0100] Warning C4706: Failed to rollback uncommited Database transaction(s) (if any). (Reason='Database error: Unable to access database. Error Code: 3114

 Reason: ORA-03114: no conectado a ORACLE

')

[2013/01/30 11:44:16.000597 +0100] Warning C4706: Failed to rollback uncommited Database transaction(s) (if any). (Reason='Database error: Unable to access database. Error Code: 3114

 Reason: ORA-03114: no conectado a ORACLE

')

[2013/01/30 11:44:16.000597 +0100] Warning C4706: Failed to rollback uncommited Database transaction(s) (if any). (Reason='Database error: Unable to access database. Error Code: 3114

 Reason: ORA-03114: no conectado a ORACLE

')

[2013/01/30 11:44:16.044345 +0100] Info C4707: Closing Database connection (ConnectString='ei_user/ei_user@demosap1:1521/OR1')

[2013/01/30 11:44:16.045207 +0100] Info C4709: Database connection closed (ConnectString='ei_user/ei_user@demosap1:1521/OR1')

[2013/01/30 11:44:16.045207 +0100] Info C4708: Establishing Database connection (ConnectString='ei_user/ei_user@demosap1:1521/OR1')

[2013/01/30 11:44:16.000597 +0100] Warning C4706: Failed to rollback uncommited Database transaction(s) (if any). (Reason='Database error: Unable to access database. Error Code: 3114

 Reason: ORA-03114: no conectado a ORACLE

.....

..

[2013/01/30 11:44:16.385433 +0100] Error C4721: Error while executing Remote Call: Database error: Unable to access database. Error Code: 1089

Environment:

1. Sybase CEP4.0
Sybase CEP vc8 version - c8-server-4.0.0-winnt-x86_64_vc8-20100208.1-release.exe

[2013/01/30 11:19:06.109375 +0100] Info C1076: Sybase SySAM Message - (131228) : (Using licenses from: C:/Sybase/SybaseC8/server/SYSAM-2_0/licenses).
[2013/01/30 11:19:06.109375 +0100] Info C1076: Sybase SySAM Message - (131252) : (Checked out license for 4 CEP_CORE (2012.02150/permanent/1665 046B DC01 4BC7).).
[2013/01/30 11:19:06.109375 +0100] Info C1078: Sybase CEP Engine - retrieving the 'clustering' property from license failed. Clustering will be set to 'unlimited'.
[2013/01/30 11:19:06.109375 +0100] Status C4570: Sybase CEP Engine 4.0.0 - OEM Production License - licensed to 'SAP_EVENT_INSIGHT'
[2013/01/30 11:19:06.109375 +0100] Info C1076: Sybase SySAM Message - (131252) : (Checked out license for 1 CEP_LargeMemory

2. Event Insight version: SBOP EVENT INSIGHT 4.0

Event Insight Sybase CEP , KBA , sybase cep , event insight , BC-SYB-CEP , Sybase Aleri Streaming Platform (CEP) , BI-BIP-EVI , Event Insight , Problem