| << Back to previous view |
|
[QB-2013] Step status and live log not up-to-date on some nodes.
|
|
| Status: | Open |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | 5.1.18 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | AlSt | Assigned To: | Robin Shen |
| Resolution: | Unresolved | Votes: | 0 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| File Attachments: |
build_status_running.png
builds.PNG
IMG_15042014_133417.png
log.PNG
steps.PNG
|
| Description |
|
On some nodes the step status and the live log is not up-to-date during a build. In the screenshots you can see that the build is already running a couple of minutes and no live log (also no log from master step) in which the build seems to "hang" (but does not really, it only shows it is still in master step, but the build runs as it should).
|
| Comments |
| Comment by Robin Shen [ 09/Apr/14 12:23 AM ] |
| Can you please send a backup of database to [robin AT pmease DOT com] and let me know the configuration/build to check? |
| Comment by AlSt [ 15/Apr/14 08:33 AM ] |
|
Sorry the backup is too big to send.
I think it may be related to another thing we just experienced: The build is finished but it is still shown as running in the build overview (but not in the queue as RUNNING_BUILD). |
| Comment by AlSt [ 15/Apr/14 08:34 AM ] |
| Finished build but still shown as running. Also the duration is extremely high. |
| Comment by Robin Shen [ 15/Apr/14 08:58 AM ] |
| Is it possible to reproduce this issue with a sample database? It would be very helpful for us to find out the issue. |
| Comment by AlSt [ 15/Apr/14 01:06 PM ] |
|
We experience this problem more often now and there is also something new:
At first only build 849290 (on screenshot IMG_15...) had no status (no live log, no step status and also no progress bar) and suddenly also the builds 849301 and 849324 lost the status. After the builds finished all looked fine again. I was not able to reproduce this. But it seems (but that's only a guess) that this happens more often on nodes which have higher load. Maybe there are issues with the status update mechanism when having network problems or something. |
| Comment by Robin Shen [ 16/Apr/14 01:19 AM ] |
|
Regarding this issue: "The build is finished but it is still shown as running in the build overview (but not in the queue as RUNNING_BUILD). "
Will the build eventually being displayed as finished in build overview? |
| Comment by AlSt [ 16/Apr/14 04:31 AM ] |
| After a couple of hours the build was still shown as "running". |
| Comment by Robin Shen [ 16/Apr/14 05:16 AM ] |
| We recently encounter some other issue on agents running many builds or steps, and it turns out to be that the agent is low on memory. Can you please do the same to see if the problem still occur? Agent memory can be increased by editing property "wrapper.java.maxmemory" in file "conf/wrapper.conf" under agent's installation directory. By default, it is set to 512M which is not sufficient for heavy load agents. |
| Comment by Maikel vd Hurk [ 15/May/14 10:12 PM ] |
| This looks like similar issue I have been reporting lately to you Robin, in combination with checking build condition problem. |
| Comment by Robin Shen [ 16/May/14 12:34 AM ] |
|
Yes, they should be related. So let's focus on the build non-stopping issue first. As I discussed with Lucaz previously, I need below info to analyze the build non-stopping issue:
1. Screenshot of the non-stopping build as you've shown me right now 2. Stack trace of both QB server and involved agent. 3. Memory dump of both QB server and involved agent. 4. Build log of the non-stopping build 5. System log of both QB server and involved agent (resides under <QB server/agent install dir>/logs). In case the system log is not being flushed quickly, you may need to change property "log4j.logger.com.pmease.quickbuild" to "WARN" in file "conf/log4j.properties". Please take all these info while this issue is happening, so that I can associate timing with memory/threads for that particular instance of issue. Please zip and upload the memory dump to our ftp server. |