Hi Experts,
We have a load balanced distributed environment (SAP BI 4.0 SP4 Patch 7 - Windows 2008 R2 x64) with a separate web tier (tomcat) and app tier boxes (2 wide). Users can access the BusinessObjects through the load balancer URL and get redirected to either one of the web tier app tier box stacks based on the LB policy. Earlier everything was running fine, but now all of a sudden all user sessions are abrubtly being terminated when they are trying to run any report with moderate complexity. Anytime they try to run a report which runs for more than 5 - 7 minutes, the session gets terminated even without reaching the timeout value - but they same report runs perfectly fine when scheduled. This means the Business Objects services are running perfectly fine but it is an issue with the web tier.
Also, even if they are just browsing through the Public folders their session gets kicked off. It happens randomly but on a regular basis in the last couple of days.
I have attached a pic of the error i see on the report panel and the corresponding messages on tomcat logs have been included below. We also get occasional http 500 errors.
I have tried the following with no success - we still keep getting these errors
- restarting tomcat
- recreating the catalina localhost directory
- redeploying BOE on tomcat
- tried increasing connection timeout to 25 seconds on tomcat server.xml
We have allocated sufficient heap on tomcat servers and dont see any memory exceptions
I thought it was issues with our IE proxy settings and have also eliminated that issue - anything that is running within our domain is exempted from the proxy rules - so we are good on that front
Any help or suggestions would greatly be appreciated - My next step is to run reports through the rich client and see if i get the errors but most of the developers use thin client for developing/running reports.
Caused by: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:676)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:619)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 48 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:741)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:765)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:574)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
... 56 more
com.businessobjects.sdk.core.server.CommunicationException$UnexpectedServerException: java.net.SocketException: Unexpected end of file from server
at com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilde[/size]r.java:144)
at com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilder.java:101)
at com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:183)
at com.businessobjects.sdk.core.server.internal.AbstractServer.process(AbstractServer.java:133)
at com.businessobjects.sdk.core.server.internal.InstanceServer.process(InstanceServer.java:94)
at com.sap.sl.sdk.services.util.ServerRequestProcessor.processServerRequest(ServerRequestProcessor.java:49)
at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.processDataProviderCommand(DataProviderServiceImpl.java:441)
at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.processDataProviderCommand(DataProviderServiceImpl.java:453)
at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.run(DataProviderServiceImpl.java:213)
at com.sap.sl.sdk.workspace.service.WorkspaceServiceImpl.run(WorkspaceServiceImpl.java:330)
at com.sap.webi.ui.tasks.workflows.RunWorkspaceTask.doIt(RunWorkspaceTask.java:80)
at com.sap.webi.ui.tasks.workflows.RunWorkspaceTask.doIt(RunWorkspaceTask.java:16)
at com.sap.webi.toolkit.ui.tasks.WebITask$PrivateWorker.doInBackground(WebITask.java:337)
at javax.swing.SwingWorker$1.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at javax.swing.SwingWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.xml.ws.WebServiceException: java.net.SocketException: Unexpected end of file from server
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(Unknown Source)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
at $Proxy41.processRequest(Unknown Source)
at com.businessobjects.sdk.core.server.internal.http.HTTPServerImpl.doProcess(HTTPServerImpl.java:184)
at com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:171)
... 17 more
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
... 33 more
org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet equinoxbridgeservlet threw exception
java.lang.RuntimeException: org.apache.jasper.JasperException: java.lang.NullPointerException
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:254)
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.service(BundlePathAwareServiceHandler.java:197)
at com.businessobjects.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.businessobjects.pinger.TimeoutManagerFilter.doFilter(TimeoutManagerFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:421)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.eclipse.equinox.jsp.jasper.JspServlet.service(JspServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.businessobjects.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:110)
at com.businessobjects.http.servlet.internal.ServletLastFilterChainElement.service(ServletLastFilterChainElement.java:30)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:46)
at com.businessobjects.webutil.boetrustguard.BOETrustPrepareFilter.doFilter(BOETrustPrepareFilter.java:32)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.swd.shared.tracelog.TraceLogScopeFilter.doFilter(TraceLogScopeFilter.java:38)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.sdk.actionfilter.WorkflowFilter.doFilter(WorkflowFilter.java:45)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.swd.appcontext.RequestInitFilter.doFilter(RequestInitFilter.java:26)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:235)
... 20 more
Caused by: java.lang.NullPointerException
at com.businessobjects.webutil.clientaction.query.ActionQueryHelper.getObjectViewerActions(ActionQueryHelper.java:87)
at org.apache.jsp.jsp.common.appletactions_jsp._jspService(appletactions_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
... 41 more