History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3987
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Robin Shen
Reporter: Christopher Speier
Votes: 0
Watchers: 0

If you were logged in you would be able to see more operations.

Qucikbuild Dashboard - Major Release - spinning and not responding for 10-15 minutes

Created: 14/Jun/23 05:15 PM   Updated: 23/Dec/23 01:15 AM
Component/s: None
Affects Version/s: 13.0.10
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
File Attachments: 1. Text File 40397.txt (153 kb)
2. Text File output.txt (203 kb)
3. Text File queue.txt (50 kb)

Image Attachments:

1. quickbuild-major.PNG
(92 kb)
Environment: Production

 Description  « Hide
version 13.0.10 - Dashboard today became unresponsive today for Major Releases - we bounced the application with no change. We restarted our VM and then restarted the application.. no change.
We have a release tomorrow night and we have started a pre-prod deployment, but have concerns for tonight daily builds.

 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [14/Jun/23 11:56 PM]
Sorry for the inconvenience. While it is stucking, please run below to get a stack trace and send to [robin AT pmease DOT com]

/path/to/jdk/bin/jstack <QB server JVM process id>

Robin Shen [15/Jun/23 12:05 AM]
Please note that QB server has two processes, the JVM process and the wrapper process.

Christopher Speier [15/Jun/23 12:56 AM]
queue.txt is current process running under quickbuild
png is a picture of the dashboard

Christopher Speier [15/Jun/23 01:12 AM]
from the jdk path - /opt/quickbuild/jdk1.8/bin
./jstack 2114
2114: Operation not permitted

i've asked my unix admin to perform it.

Robin Shen [15/Jun/23 01:16 AM]
Please run this command as quickbuild user.

Christopher Speier [15/Jun/23 12:17 PM]
[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114

2114: Unable to open socket file: target process not responding or HotSpot VM not loaded

The -F option can be used when the target process is not responding

[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114 -F

Attaching to core -F from executable 2114, please wait...

Error attaching to core file: cannot open binary file

sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:286)

        at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:673)

        at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)

        at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)

        at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)

        at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156)

        at sun.jvm.hotspot.tools.Tool.start(Tool.java:191)

        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)

        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)

        at sun.tools.jstack.JStack.main(JStack.java:106)

Robin Shen [15/Jun/23 12:23 PM]
Can you please sudo as quickbuild and run this command? If not work, please restart QB server and get the stack trace while dashboard is stucking.

Robin Shen [15/Jun/23 12:24 PM]
Also which JVM are you using?

Christopher Speier [15/Jun/23 12:33 PM]
root and quickbuild ran against both 2114 and 1938.
also ... please restart QB server and get the stack trace while dashboard is stucking.??? we are running the jstack now while my major build dashboard is spinning.

are you suggesting just restarting the QB now?

Robin Shen [15/Jun/23 12:37 PM]
From the log you posted, jstack failed to run. This seems abnormal to me. So I suggested to restart JVM and try jstack again while dashboard is stucking.

Robin Shen [15/Jun/23 12:40 PM]
Also please check conf/wrapper.conf to make sure you've allocated enough memory.

Christopher Speier [15/Jun/23 12:48 PM]
 ps -ef|grep quickbuild
quickbu+ 1938 1 0 Jun14 ? 00:00:50 /opt/quickbuild/quickbuild/bin/./wrapper-linux-x86-64 /opt/quickbuild/quickbuild/bin/../conf/wrapper.conf wrapper.syslog.ident=quickbuild wrapper.pidfile=/opt/quickbuild/quickbuild/bin/quickbuild.pid wrapper.daemonize=TRUE wrapper.name=quickbuild wrapper.displayname=QuickBuild Server wrapper.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.status wrapper.java.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.java.status wrapper.script.version=3.5.49
quickbu+ 2114 1938 4 Jun14 ? 00:52:19 /opt/quickbuild/jdk1.8.0_181/bin/java -Djava.awt.headless=true -Xverify:none -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED -Xms1024m -Xmx12000m -Djava.library.path=.:./lib:../plugins/com.pmease.quickbuild.libs -classpath ../plugins/com.pmease.quickbuild.bootstrap:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-core-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-slf4j-impl-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/hessian-4.0.38.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-io-1.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/slf4j-api-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-logging-1.0.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/joda-time-2.7.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-codec-1.10.jar:../plugins/com.pmease quickbuild.bootstrap/lib/jul-to-slf4j-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/wrapper.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/servlet-api-3.1.jar:../plugins/com.pmease.quickbuil .bootstrap/lib/commons-lang-2.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-api-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-1.2-api-2.17.2.jar -Dwrapper.key=NLCtPGJCjzILtiOp5BbVvkqnEFGKhqxH -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1938 -Dwrapper.version=3.5.49-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=3600 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperStartStopApp com.pmease.quickbuild.bootstrap.Bootstrap 1 start com.pmease.quickbuild.bootstrap.Bootstrap FALSE 1 stop

when i look at the wrapper in /opt/quickbuild/quickbuild/conf ..

Is that enough?

Robin Shen [15/Jun/23 12:51 PM]
12000 should be more than enough.

Christopher Speier [15/Jun/23 01:12 PM]
quickbuild restarted.. new pids pid's 40383 and 40397 - stack attached.. major dashboard still spinning for 15 min or more.

Robin Shen [15/Jun/23 01:22 PM]
Thanks for sending the stack trace. Turns out that one of the gadget in your dashboard is stucking on calling JIRA server restful api. You may open dashboard in failsafe mode by appending /failsafe to current url (for instance http://your-qb-server/dashboard/1/failsafe), then remove the jira related gadget temporarily. And then open dashboard in normal mode again. I also circled Steve in so that he can check what might wrong with the JIRA gadget.

Christopher Speier [15/Jun/23 01:59 PM]
Success!!! the project removed our on-prem jira link and it fixed the issue!

Thank you!