<< Back to previous view

[QB-3987] Qucikbuild Dashboard - Major Release - spinning and not responding for 10-15 minutes
Created: 14/Jun/23  Updated: 23/Dec/23

Status: Closed
Project: QuickBuild
Component/s: None
Affects Version/s: 13.0.10
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Christopher Speier Assigned To: Robin Shen
Resolution: Fixed Votes: 0
Remaining Estimate: Unknown Time Spent: Unknown
Original Estimate: Unknown
Environment: Production

File Attachments: Text File 40397.txt     Text File output.txt     Text File queue.txt     PNG File quickbuild-major.PNG    

 Description   
version 13.0.10 - Dashboard today became unresponsive today for Major Releases - we bounced the application with no change. We restarted our VM and then restarted the application.. no change.
We have a release tomorrow night and we have started a pre-prod deployment, but have concerns for tonight daily builds.

 Comments   
Comment by Robin Shen [ 14/Jun/23 11:56 PM ]
Sorry for the inconvenience. While it is stucking, please run below to get a stack trace and send to [robin AT pmease DOT com]

/path/to/jdk/bin/jstack <QB server JVM process id>
Comment by Robin Shen [ 15/Jun/23 12:05 AM ]
Please note that QB server has two processes, the JVM process and the wrapper process.
Comment by Christopher Speier [ 15/Jun/23 12:56 AM ]
queue.txt is current process running under quickbuild
png is a picture of the dashboard
Comment by Christopher Speier [ 15/Jun/23 01:12 AM ]
from the jdk path - /opt/quickbuild/jdk1.8/bin
./jstack 2114
2114: Operation not permitted

i've asked my unix admin to perform it.
Comment by Robin Shen [ 15/Jun/23 01:16 AM ]
Please run this command as quickbuild user.
Comment by Christopher Speier [ 15/Jun/23 12:17 PM ]
[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114

2114: Unable to open socket file: target process not responding or HotSpot VM not loaded

The -F option can be used when the target process is not responding


[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114 -F

Attaching to core -F from executable 2114, please wait...

Error attaching to core file: cannot open binary file

sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:286)

        at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:673)

        at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)

        at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)

        at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)

        at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156)

        at sun.jvm.hotspot.tools.Tool.start(Tool.java:191)

        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)

        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)

        at sun.tools.jstack.JStack.main(JStack.java:106)
Comment by Robin Shen [ 15/Jun/23 12:23 PM ]
Can you please sudo as quickbuild and run this command? If not work, please restart QB server and get the stack trace while dashboard is stucking.
Comment by Robin Shen [ 15/Jun/23 12:24 PM ]
Also which JVM are you using?
Comment by Christopher Speier [ 15/Jun/23 12:33 PM ]
root and quickbuild ran against both 2114 and 1938.
also ... please restart QB server and get the stack trace while dashboard is stucking.??? we are running the jstack now while my major build dashboard is spinning.

are you suggesting just restarting the QB now?
Comment by Robin Shen [ 15/Jun/23 12:37 PM ]
From the log you posted, jstack failed to run. This seems abnormal to me. So I suggested to restart JVM and try jstack again while dashboard is stucking.
Comment by Robin Shen [ 15/Jun/23 12:40 PM ]
Also please check conf/wrapper.conf to make sure you've allocated enough memory.
Comment by Christopher Speier [ 15/Jun/23 12:48 PM ]
 ps -ef|grep quickbuild
quickbu+ 1938 1 0 Jun14 ? 00:00:50 /opt/quickbuild/quickbuild/bin/./wrapper-linux-x86-64 /opt/quickbuild/quickbuild/bin/../conf/wrapper.conf wrapper.syslog.ident=quickbuild wrapper.pidfile=/opt/quickbuild/quickbuild/bin/quickbuild.pid wrapper.daemonize=TRUE wrapper.name=quickbuild wrapper.displayname=QuickBuild Server wrapper.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.status wrapper.java.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.java.status wrapper.script.version=3.5.49
quickbu+ 2114 1938 4 Jun14 ? 00:52:19 /opt/quickbuild/jdk1.8.0_181/bin/java -Djava.awt.headless=true -Xverify:none -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED -Xms1024m -Xmx12000m -Djava.library.path=.:./lib:../plugins/com.pmease.quickbuild.libs -classpath ../plugins/com.pmease.quickbuild.bootstrap:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-core-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-slf4j-impl-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/hessian-4.0.38.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-io-1.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/slf4j-api-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-logging-1.0.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/joda-time-2.7.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-codec-1.10.jar:../plugins/com.pmease quickbuild.bootstrap/lib/jul-to-slf4j-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/wrapper.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/servlet-api-3.1.jar:../plugins/com.pmease.quickbuil .bootstrap/lib/commons-lang-2.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-api-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-1.2-api-2.17.2.jar -Dwrapper.key=NLCtPGJCjzILtiOp5BbVvkqnEFGKhqxH -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1938 -Dwrapper.version=3.5.49-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=3600 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperStartStopApp com.pmease.quickbuild.bootstrap.Bootstrap 1 start com.pmease.quickbuild.bootstrap.Bootstrap FALSE 1 stop

when i look at the wrapper in /opt/quickbuild/quickbuild/conf ..
wrapper.java.initmemory=1024
wrapper.java.maxmemory=12000

Is that enough?
Comment by Robin Shen [ 15/Jun/23 12:51 PM ]
12000 should be more than enough.
Comment by Christopher Speier [ 15/Jun/23 01:12 PM ]
quickbuild restarted.. new pids pid's 40383 and 40397 - stack attached.. major dashboard still spinning for 15 min or more.
Comment by Robin Shen [ 15/Jun/23 01:22 PM ]
Thanks for sending the stack trace. Turns out that one of the gadget in your dashboard is stucking on calling JIRA server restful api. You may open dashboard in failsafe mode by appending /failsafe to current url (for instance http://your-qb-server/dashboard/1/failsafe), then remove the jira related gadget temporarily. And then open dashboard in normal mode again. I also circled Steve in so that he can check what might wrong with the JIRA gadget.
Comment by Christopher Speier [ 15/Jun/23 01:59 PM ]
Success!!! the project removed our on-prem jira link and it fixed the issue!

Thank you!
Generated at Fri May 09 15:45:56 UTC 2025 using JIRA 189.