[#QB-3987] Qucikbuild Dashboard - Major Release - spinning and not responding for 10-15 minutes

QuickBuild

Qucikbuild Dashboard - Major Release - spinning and not responding for 10-15 minutes

Created: 14/Jun/23 05:15 PM Updated: 23/Dec/23 01:15 AM

Component/s:

None

Affects Version/s:

13.0.10

Fix Version/s:

None

Original Estimate:

Unknown

Remaining Estimate:

Unknown

Time Spent:

Unknown

File Attachments:

40397.txt (153 kb)
2.

output.txt (203 kb)
3.

queue.txt (50 kb)

Image Attachments:

1. quickbuild-major.PNG
(92 kb)

Environment:

Production

Description

« Hide

version 13.0.10 - Dashboard today became unresponsive today for Major Releases - we bounced the application with no change. We restarted our VM and then restarted the application.. no change.
We have a release tomorrow night and we have started a pre-prod deployment, but have concerns for tonight daily builds.

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Robin Shen [14/Jun/23 11:56 PM]

Sorry for the inconvenience. While it is stucking, please run below to get a stack trace and send to [robin AT pmease DOT com]

/path/to/jdk/bin/jstack <QB server JVM process id>

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:05 AM]

Please note that QB server has two processes, the JVM process and the wrapper process.

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 12:56 AM]

queue.txt is current process running under quickbuild
png is a picture of the dashboard

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 01:12 AM]

from the jdk path - /opt/quickbuild/jdk1.8/bin
./jstack 2114
2114: Operation not permitted

i've asked my unix admin to perform it.

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 01:16 AM]

Please run this command as quickbuild user.

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 12:17 PM]

[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114

2114: Unable to open socket file: target process not responding or HotSpot VM not loaded

The -F option can be used when the target process is not responding

[7:16 AM] Christopher Dalhoe

[root@appqbuildapup01 ~]# /opt/quickbuild/jdk1.8/bin/jstack 2114 -F

Attaching to core -F from executable 2114, please wait...

Error attaching to core file: cannot open binary file

sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)

        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:286)

        at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:673)

        at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)

        at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)

        at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)

        at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156)

        at sun.jvm.hotspot.tools.Tool.start(Tool.java:191)

        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)

        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)

        at sun.tools.jstack.JStack.main(JStack.java:106)

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:23 PM]

Can you please sudo as quickbuild and run this command? If not work, please restart QB server and get the stack trace while dashboard is stucking.

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:24 PM]

Also which JVM are you using?

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 12:33 PM]

root and quickbuild ran against both 2114 and 1938.
also ... please restart QB server and get the stack trace while dashboard is stucking.??? we are running the jstack now while my major build dashboard is spinning.

are you suggesting just restarting the QB now?

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:37 PM]

From the log you posted, jstack failed to run. This seems abnormal to me. So I suggested to restart JVM and try jstack again while dashboard is stucking.

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:40 PM]

Also please check conf/wrapper.conf to make sure you've allocated enough memory.

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 12:48 PM]

ps -ef|grep quickbuild
quickbu+ 1938 1 0 Jun14 ? 00:00:50 /opt/quickbuild/quickbuild/bin/./wrapper-linux-x86-64 /opt/quickbuild/quickbuild/bin/../conf/wrapper.conf wrapper.syslog.ident=quickbuild wrapper.pidfile=/opt/quickbuild/quickbuild/bin/quickbuild.pid wrapper.daemonize=TRUE wrapper.name=quickbuild wrapper.displayname=QuickBuild Server wrapper.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.status wrapper.java.statusfile=/opt/quickbuild/quickbuild/bin/quickbuild.java.status wrapper.script.version=3.5.49
quickbu+ 2114 1938 4 Jun14 ? 00:52:19 /opt/quickbuild/jdk1.8.0_181/bin/java -Djava.awt.headless=true -Xverify:none -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED -Xms1024m -Xmx12000m -Djava.library.path=.:./lib:../plugins/com.pmease.quickbuild.libs -classpath ../plugins/com.pmease.quickbuild.bootstrap:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-core-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-slf4j-impl-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/hessian-4.0.38.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-io-1.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/slf4j-api-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-logging-1.0.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/joda-time-2.7.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/commons-codec-1.10.jar:../plugins/com.pmease quickbuild.bootstrap/lib/jul-to-slf4j-1.7.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/wrapper.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/servlet-api-3.1.jar:../plugins/com.pmease.quickbuil .bootstrap/lib/commons-lang-2.4.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-api-2.17.2.jar:../plugins/com.pmease.quickbuild.bootstrap/lib/log4j-1.2-api-2.17.2.jar -Dwrapper.key=NLCtPGJCjzILtiOp5BbVvkqnEFGKhqxH -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1938 -Dwrapper.version=3.5.49-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=3600 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperStartStopApp com.pmease.quickbuild.bootstrap.Bootstrap 1 start com.pmease.quickbuild.bootstrap.Bootstrap FALSE 1 stop

when i look at the wrapper in /opt/quickbuild/quickbuild/conf ..
wrapper.java.initmemory=1024
wrapper.java.maxmemory=12000

Is that enough?

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 12:51 PM]

12000 should be more than enough.

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 01:12 PM]

quickbuild restarted.. new pids pid's 40383 and 40397 - stack attached.. major dashboard still spinning for 15 min or more.

[ Permlink | « Hide ]

Robin Shen [15/Jun/23 01:22 PM]

Thanks for sending the stack trace. Turns out that one of the gadget in your dashboard is stucking on calling JIRA server restful api. You may open dashboard in failsafe mode by appending /failsafe to current url (for instance http://your-qb-server/dashboard/1/failsafe), then remove the jira related gadget temporarily. And then open dashboard in normal mode again. I also circled Steve in so that he can check what might wrong with the JIRA gadget.

[ Permlink | « Hide ]

Christopher Speier [15/Jun/23 01:59 PM]

Success!!! the project removed our on-prem jira link and it fixed the issue!

Thank you!