Hi, Robin
Since the timeout issue (
QB-3627) , I have been working on the following tasks.
1. Deleted quite a few old logs on each product node. ( that's because some node has a lot of build histories )
2. Paused the Cron Schedule build set on many nodes and changed the Build Condition from "if changes found in referenced repositories" to "always build".
This is because when the build condition is set to "if changes found in referenced repositories", it seems to be making more network traffic.
(It compares the latest commit & files on remote repository of github with the latest ones on the build agent. )
There was no timeout issue on the node that did the above two tasks.
Although this is not a perfect solution, I think above things I did can reduce timeout issues at least.
And I enabled cron schedule build on only one node. the read timeout issue occurs only in the node where the cron scheduler is configured.
Please see attached screenshots and log files from build agent machine.
Take a closer look at what more we need to do. this issue bother us a lot.
The important thing is that a jetty eof exception error occurs before a timeout issue occurs. Please take a look at the log below.
jvm 2 | javax.servlet.ServletException: org.eclipse.jetty.io.EofException
jvm 2 | at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:385)
jvm 2 | at com.pmease.quickbuild.grid.GridServlet.service(GridServlet.java:36)
jvm 2 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
jvm 2 | at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1186)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
jvm 2 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
jvm 2 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
jvm 2 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
jvm 2 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
jvm 2 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
jvm 2 | at org.eclipse.jetty.server.Server.handle(Server.java:499)
jvm 2 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
jvm 2 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
jvm 2 | at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
jvm 2 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
jvm 2 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
jvm 2 | at java.lang.Thread.run(Unknown Source)
jvm 2 | Caused by: org.eclipse.jetty.io.EofException
jvm 2 | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
jvm 2 | at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
jvm 2 | at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
jvm 2 | at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)
jvm 2 | at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:724)
jvm 2 | at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
jvm 2 | at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
jvm 2 | at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:472)
jvm 2 | at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:756)
jvm 2 | at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:794)
jvm 2 | at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:155)
jvm 2 | at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148)
jvm 2 | at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:250)
jvm 2 | at com.caucho.hessian.io.Hessian2Output.flush(Hessian2Output.java:1615)
jvm 2 | at com.caucho.hessian.io.Hessian2Output.close(Hessian2Output.java:1654)
jvm 2 | at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:201)
jvm 2 | at com.caucho.hessian.server.HessianServlet.invoke(HessianServlet.java:399)
jvm 2 | at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:379)
jvm 2 | ... 21 more
jvm 2 | Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
jvm 2 | at sun.nio.ch.SocketDispatcher.write0(Native Method)
jvm 2 | at sun.nio.ch.SocketDispatcher.write(Unknown Source)
jvm 2 | at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
jvm 2 | at sun.nio.ch.IOUtil.write(Unknown Source)
jvm 2 | at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
jvm 2 | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
jvm 2 | ... 38 more
jvm 2 | 2020-10-14 21:18:42,534 WARN Could not send response error 500: javax.servlet.ServletException: javax.servlet.ServletException: org.eclipse.jetty.io.EofException
jvm 2 | 2020-10-14 21:23:43,431 WARN