 |
History
| Log In
|
|
|
|
[
Permlink
| « Hide
]
Robin Shen [22/Mar/15 11:24 PM]
Yes it could be that the forked process can not be terminated by QB. Can you please kill that process with OS kill utility and then check if the build can be stopped?
That's what I did and it stopped. It'd be nice if I didn't have to do that though. :-(
Which process is the build forking? And can it be killed gracefully via "kill -TERM"? If not, QB might not be able to stop it as it sends SIG_TERM to forked processes.
It was running busted, though it may have been doing it through a shell. It died with a regular TERM signal, so I'm not sure why QB had trouble stopping it. It would be nice if QB would try again at some point though. Some programs run cleanup code for SIGTERM, but it's possible they could be stuck in that cleanup. Usually, another SIGTERM will force it to exit anyways. Subversion is one such application.
Ours did not have this behavior, but it is possible that the test app in use had some bad behavior in terms of signals (though, it was not blocking SIGTERM). We use libuv in a test application, and there is a known bug for libuv and signal delivery. However, libuv also does nothing for SIGTERM so the default handler to terminate the process should still be in place. We did end up modifying the test app slightly and that helped us avoid the issue, but I still think QB should have at least tried harder here. Please check if manually stopping the build for another one or two tries works in case the timeout or first cancel does not work. Upon build timeout or first cancel, QB issues SIGTERM to all spawned process, and then it sits there waiting, and upon second cancellation (have to be issued by user manually), it will issue the "kill -9" command to forcbily kill living processes. We do not retry the kill as some cleanup steps might be running in QB (set up by QB user as a upon cancellation condition) and we leave this up to end user to retry the cancellation.
The first cancellation along with subsequent cancellation retry should be reflected in build log something like this: 08:40:44,963 INFO - Terminating launched command gracefully... 08:40:49,582 INFO - Killing process 4965... 08:40:49,582 INFO - Killing process 4964... 08:40:49,582 INFO - Killing process 4963... 08:40:49,582 INFO - Killing process 4960... 08:40:49,623 INFO - Killing process 4965... 08:41:24,404 INFO - Unable to terminate launched command gracefully, terminating forcibly instead... 08:41:24,424 INFO - Calling kill utility to forcibly kill process 4965... Unfortunately, I can't really create the issue right now. How about we close this for now. If I can provide something more concrete, I'll let you know. I will say that I did try to cancel the build several times with no effect--the build just continued to hang there. Unfortunately, it's now been a while so I don't have the logs to go back too. :-(
No problem, let's open the issue here. Please comment it if the problem happens again.
I'm seeing this in the logs:
01:58:44,044 INFO - Running step... 01:59:44,125 INFO - Terminating launched command gracefully... 03:09:33,060 INFO - Unable to terminate launched command gracefully, terminating forcibly instead... 03:10:23,912 INFO - Unable to terminate launched command gracefully, terminating forcibly instead... Nothing about process ids, and nothing about calling the kill utility. Is that expected? Seems that QB is not able to capture process id of launched processes. Can you please set up a simple step to call the sleep utility and then cancel the build to see if the log contains process ids shown previously in my log?
I tried letting a timeout lapse and manually stopping the build, and they both worked using sleep. Would turning up the logging level (to DEBUG) help uncover anything?
The various killing process IDs will be printed with INFO level, so DEBUG will not discover more things in this regard. Please help with another test:
1. Download attached "can_not_kill_by_sigterm.c", and compile with gcc to get a.out 2. Create a test.sh calling a.out #!/bin/sh /path/to/a.out 3. Add a command build step in QB to execute test.sh 4. Run the build and then stop the build manually 5. The first stop try will cause QB sending SIGTERM to a.out and process ids should be printed in build log, and the build will not exit as it can not be killed by SIGTERM 6. Now stop the build again from QB UI, now QB should call kill command to kill a.out forcibly and the build should stop after a while. Please check if above works at your side. Sorry it took so long, but I did manage to do this today. A few things came up.
First, my apologies, but I didn't make sure that the sleep step was executing on the FreeBSD node. The sleep test actually fails on the FreeBSD node: 22:45:22,841 INFO - Executing pre-execute action... 22:45:22,841 INFO - Running step... 22:45:22,850 INFO - Checking step execute condition... 22:45:22,851 INFO - Step execute condition satisfied, executing... 22:45:23,022 INFO - Executing pre-execute action... 22:45:23,023 INFO - Running step... 22:45:23,024 DEBUG - Executing command: /bin/sleep 600 22:45:23,024 DEBUG - Command working directory: /usr/home/quickbuild/buildagent/workspace/root/neovim/sleep-test 22:45:33,051 INFO - Terminating launched command gracefully... 22:46:08,935 INFO - Unable to terminate launched command gracefully, terminating forcibly instead... No PIDs of any sort. Compiling and running a.out via test.sh had similar results. So stopping a build simply executing a sleep command does not work? I do not have FreeBSD 10 at hand and will set up one to check this issue.
> So stopping a build simply executing a sleep command does not work?
That's correct. Thanks for the info, will go ahead to setup a FreeBSD 10 system to check what might be the cause.
This issue has now been fixed in 6.0.14 (due to JIRA bug, we can not update status of this issue now for some reason)
Thanks a million! This couldn't have come at a better time... my FreeBSD was getting stuck quiet frequently this week. Thank you!
It works at my side. Please create a simple configuration running a command/batch step with below command:
sleep 300 Then stop the build to see if it works. Sorry, I didn't mean to imply that it never works--it only appears to mostly work. I've definitely had builds run up against the time limits and QB has terminated them without issue.
But then, there are ones like this: http://neovim-qb.szakmeister.net/build/7676 and this: http://neovim-qb.szakmeister.net/build/7668. The first ran for 8 hours before I got a chance to take a closer look, while the second ran considerably longer (over a day) I had to log into the FreeBSD node and kill the running process for the executable (nvim, in this case). I'm not sure how it's happening. Looking at the timeout builds, there are none that are at approximately the limit (meaning, QB wasn't able to terminate them and I had to jump in and kill the process under test). Your simple test works, but I think things are failing when trying to kill a tree of processes. The OS X and Linux boxes we have configured don't appear to suffer from this issue. :-( Maybe that is because the process does not respond to SIGTERM signal. QB uses below script to loop against all process trees and send them SIGTERM:
#!/bin/sh list_descendants () { local children=$(pgrep -P "$1") for pid in $children do list_descendants "$pid" done echo "$children" } kill $(list_descendants $1) It's a good suggestion, but I use "kill <pid>" to stop the executable under test (not QuickBuild), which defaults to using SIGTERM and it exits. :-( I'll try and dig into this more, but I'm not sure where to go. What I can say is that a child process is being abandoned (because its parent dies), so "pgrep -P <pid>" no longer finds it afterwards. I wonder if there's a bit of a race between killing the children and the children starting a new process, which causes the new child to be missed? I'll try to look and see if there's a safer way to kill a tree of processes. OTOH, QB is getting stuck when this condition occurs waiting for a process that may never terminate. I have a step to kill any lingering processes, but it doesn't get to that point. Is there something that could be done there?
Seems there is no other common working approach killing the whole process tree other than using pgrep. Maybe you can add some terminating logic at the end of the main command to terminate the child processes by yourself if you know child processes being spawned.
One approach that I think would do the trick the majority of the time is to create a new session for the launched process and then use pkill to kill the session by session id. Some quick tests shows that works well, and abandoned children are still terminated. The trick here is that the launched processes needs to make a call to setsid() to set itself as a session leader. I think the only catch here is that you have to fork a new process and let the parent die so that it is able to become a session leader. I'm not sure how that affects things. I can probably work something into our testing to help make this happen for us, but that obviously won't help others who run into this issue.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Resolved
Major
Timeouts or stop not working correctly under FreeBSD 10
