| << Back to previous view |
|
[QB-2827] Builds are occasionally failing to finish...
|
|
| Status: | Closed |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | 6.1.29 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | John Szakmeister | Assigned To: | Robin Shen |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| File Attachments: |
qb-build-did-not-stop.png
|
| Description |
|
I'm not sure if it's some issue of ours--QB isn't helping me to narrow the issue down--but we appear to have some builds that have run to completion, and QB looks to be finishing everything up, but it never actually completes. What's really interesting is that a new build is allowed to start, even though QB was not finished with the previous one. We do not have concurrent builds set up, so this is kind of surprising and I'm not sure if it's part of the issue or not. I'm attaching a screen shot showing that a build is still going, but other builds have completed afterwards.
|
| Comments |
| Comment by John Szakmeister [ 09/Oct/16 09:01 PM ] |
| This shows builds having being completed despite a prior one still running (but finishing the build). |
| Comment by Robin Shen [ 10/Oct/16 12:31 AM ] |
| When this happens, are there any errors printed in server log? |
| Comment by John Szakmeister [ 10/Oct/16 12:07 PM ] |
| Not that I noticed, but the logs have rotated and I can't go back far enough to verify this. When it happens again, I'll be sure to check and save the server log. |
| Comment by John Szakmeister [ 11/Oct/16 08:14 PM ] |
|
So I managed to catch the issue again. I do see this in the log messages on the server:
2016-10-11 15:35:45,570 [pool-1-thread-1531] INFO com.pmease.quickbuild.DefaultBuildEngine - Build request has been processed. 2016-10-11 15:35:49,960 [qtp1493446822-57] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:35:50,517 [qtp1493446822-35] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:35:50,567 [qtp1493446822-35] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:01,459 [qtp1493446822-79] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:02,742 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:02,794 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:11,732 [qtp1493446822-25] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:12,903 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:12,951 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:22,481 [qtp1493446822-21] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:23,068 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:23,118 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:33,190 [qtp1493446822-61] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:33,236 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:33,281 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:44,012 [qtp1493446822-20] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:45,422 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:45,471 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:36:54,591 [qtp1493446822-64] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:36:55,588 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:36:55,632 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:05,042 [qtp1493446822-55] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:37:05,733 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:37:05,781 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:16,222 [qtp1493446822-25] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:37:17,935 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:37:17,985 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:26,621 [qtp1493446822-21] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:37:28,087 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:37:28,133 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:37,407 [qtp1493446822-77] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:37:38,227 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:37:38,285 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:49,307 [qtp1493446822-20] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:37:50,414 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:37:50,463 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:37:59,243 [qtp1493446822-27] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:00,578 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:00,627 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:38:09,973 [qtp1493446822-63] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:10,754 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:10,803 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:38:21,485 [qtp1493446822-25] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:22,947 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:22,996 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:38:30,743 [qtp1493446822-76] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:31,075 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:31,130 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:38:41,776 [qtp1493446822-51] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:43,284 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:43,315 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:38:52,790 [qtp1493446822-29] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:38:53,427 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:38:53,475 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:39:03,035 [qtp1493446822-27] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:39:03,593 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:39:03,661 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. 2016-10-11 15:39:14,748 [qtp1493446822-60] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is stopped. 2016-10-11 15:39:15,796 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Retrieving node characteristics from agent 'neovim-qb-slave-yosemite.home:8814'... 2016-10-11 15:39:15,845 [qtp1493446822-45] INFO com.pmease.quickbuild.grid.DefaultAgentManager - Build agent 'neovim-qb-slave-yosemite.home:8814' is active. I don't understand why it's bouncing so much though--that machine is a VM with plenty of hardware and memory behind it, so I find it hard to believe it's overtaxed (and I don't see that from CPU and IO meters). But perhaps that is what's leaving the build hanging? Is there something else I should check? And just an FYI: I don't have automatic updates running or anything like that. Also, the Internet connection was fine throughout (I have monitoring of that externally, so I'll get emails if the line bounces for some reason. Plus, the other nodes would have likely shown issues too (we have 3 build agents running). |
| Comment by John Szakmeister [ 11/Oct/16 08:16 PM ] |
|
I'm see this on the affected build node:
2016-10-11 13:14:56,194 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Disconnecting from server... 2016-10-11 13:14:56,240 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Waiting for finish of existing jobs... 2016-10-11 13:14:56,241 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Stopping plugins... 2016-10-11 13:14:56,246 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Stopping task scheduler... 2016-10-11 13:14:56,343 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - QuickBuild agent stopped. 2016-10-11 13:15:00,714 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Cleaning cached plugin libraries... 2016-10-11 13:15:00,764 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Cleaning temp directory... 2016-10-11 13:15:00,767 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Benchmarking... 2016-10-11 13:15:02,225 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Connecting to server 'http://neovim-qb.szakmeister.net'... 2016-10-11 13:15:02,313 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.AgentUpdater - Updating framework... 2016-10-11 13:15:02,354 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.AgentUpdater - Updating plugins... 2016-10-11 13:15:02,375 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.AgentUpdater - Updating plugins/com.pmease.quickbuild.libs... 2016-10-11 13:15:02,391 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.AgentUpdater - Updating asset... 2016-10-11 13:15:02,410 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Booting QuickBuild from '/Users/quickbuild/tmp/buildagent'... 2016-10-11 13:15:02,411 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.bootstrap.Bootstrap - Starting framework... 2016-10-11 13:15:02,441 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.equinoxadapter.EquinoxAdapter - Caching plugin libraries... 2016-10-11 13:15:04,828 [WrapperStartStopAppMain] WARN org.eclipse.jetty.util.component.AbstractLifeCycle - FAILED SocketConnector@0.0.0.0:8814: java.net.BindException: Address already in use java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) at java.net.ServerSocket.bind(ServerSocket.java:375) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:181) at org.eclipse.jetty.server.bio.SocketConnector.newServerSocket(SocketConnector.java:96) at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:85) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:156) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.Server.doStart(Server.java:288) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at com.pmease.quickbuild.Quickbuild.startJettyServer(Quickbuild.java:370) at com.pmease.quickbuild.Quickbuild.start(Quickbuild.java:249) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.pmease.quickbuild.equinoxadapter.EquinoxAdapter.startup(EquinoxAdapter.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.pmease.quickbuild.bootstrap.Bootstrap.boot(Bootstrap.java:576) at com.pmease.quickbuild.bootstrap.Bootstrap.main(Bootstrap.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:316) at java.lang.Thread.run(Thread.java:745) 2016-10-11 13:15:04,833 [WrapperStartStopAppMain] WARN org.eclipse.jetty.util.component.AbstractLifeCycle - FAILED org.eclipse.jetty.server.Server@288aa726: java.net.BindException: Address already in use java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) at java.net.ServerSocket.bind(ServerSocket.java:375) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:181) at org.eclipse.jetty.server.bio.SocketConnector.newServerSocket(SocketConnector.java:96) at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:85) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:156) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.Server.doStart(Server.java:288) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at com.pmease.quickbuild.Quickbuild.startJettyServer(Quickbuild.java:370) at com.pmease.quickbuild.Quickbuild.start(Quickbuild.java:249) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.pmease.quickbuild.equinoxadapter.EquinoxAdapter.startup(EquinoxAdapter.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.pmease.quickbuild.bootstrap.Bootstrap.boot(Bootstrap.java:576) at com.pmease.quickbuild.bootstrap.Bootstrap.main(Bootstrap.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:316) at java.lang.Thread.run(Thread.java:745) 2016-10-11 13:15:05,848 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Stopping QuickBuild agent... 2016-10-11 13:15:06,073 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Disconnecting from server... 2016-10-11 13:15:06,095 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Waiting for finish of existing jobs... 2016-10-11 13:15:06,096 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Stopping plugins... 2016-10-11 13:15:06,099 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - Stopping task scheduler... 2016-10-11 13:15:06,183 [WrapperStartStopAppMain] INFO com.pmease.quickbuild.Quickbuild - QuickBuild agent stopped. Any ideas what that's all about? |
| Comment by Robin Shen [ 12/Oct/16 12:37 AM ] |
| Looks like you have two agents on same machine with same port? Also please make sure to use fixed ip address for your build agents instead of dynamic ones allocated by DHCP |
| Comment by John Szakmeister [ 12/Oct/16 12:43 AM ] |
| I shouldn't. :-( It's started via launchd using the service wrapper you guys provided. Perhaps there's something else happening here... like something is re-launching and losing it's PID so launchd believe it needs to start the service again? |
| Comment by Robin Shen [ 12/Oct/16 01:03 AM ] |
| I can not think of any reason it is restarting repeatedly. Do you have any script re-start VM automatically as part of running a build, for instance to repopulate the VM to reset the test env? |
| Comment by John Szakmeister [ 12/Oct/16 08:06 AM ] |
| Nope, no scripts to restart anything. :-( |
| Comment by Robin Shen [ 13/Oct/16 12:37 AM ] |
| This is quite odd. Is this happening occasionally or can be reproduced reliably? |
| Comment by John Szakmeister [ 13/Oct/16 08:18 AM ] |
| Only occasionally. I've never seen the problem until I upgraded to 6.1.25. But I don't know if that's just coincidence. |
| Comment by Robin Shen [ 14/Oct/16 12:28 AM ] |
| I can not think of any reason why it is happening, other than wrong set up (two agents using same port or ip address) or agent being forcibly stopped by some other process. |
| Comment by John Szakmeister [ 14/Oct/16 10:14 AM ] |
| As I said, I installed it using the recommended way. At the moment, I believe there's some interaction between launchd and something the build agent was doing (perhaps it downloaded an update and restarted itself, but launchd wasn't happy about that). I'll keep an eye out for anything more I can figure out about it. Thanks. |
| Comment by Robin Shen [ 15/Oct/16 12:36 AM ] |
| The upgrade should happen only once after server upgrade. After upgrade, the build agent should not subject to any auto-update. So this seems quite odd... |
| Comment by Robin Shen [ 01/Dec/16 06:17 AM ] |
| Reopen it if there is more evidance |