[#QB-3299] Automated Database Bacup Causes issues with QB Ui

QuickBuild

Automated Database Bacup Causes issues with QB Ui

Created: 23/Nov/18 08:02 PM Updated: 19/Jan/19 08:25 AM

Component/s:

None

Affects Version/s:

7.0.16

Fix Version/s:

None

Original Estimate:	Unknown	Remaining Estimate:	Unknown	Time Spent:	Unknown
File Attachments:	1. console.log.zip (1.02 Mb)
Environment:	« Hide Operating System Linux 4.4.114-94.14-default, amd64 OS User Name root JVM Java HotSpot(TM) 64-Bit Server VM 1.7.0_79, Oracle Corporation QuickBuild Version 7.0.16 - Fri May 12 09:57:15 EDT 2017 Total Memory 1.98 GB Used Memory 1.02 GB Configurations 7629 Show » Operating System Linux 4.4.114-94.14-default, amd64 OS User Name root JVM Java HotSpot(TM) 64-Bit Server VM 1.7.0_79, Oracle Corporation QuickBuild Version 7.0.16 - Fri May 12 09:57:15 EDT 2017 Total Memory 1.98 GB Used Memory 1.02 GB Configurations 7629

Description

« Hide

Support,
On the morning of 11/22 at 1 am, the Quickbuild UI became non responsive. You could hit the sign in page but any attempt to login was met with a spinning browser. A look in the log files for the time showed only that the Automated DB backup had kicked off. In addition there were a number of messages indicating that jobs scheduled for 1 am we put in a "sleep" state. As we were unable to get to the UI even after waiting over an hour, we rebooted the system around 2 am. The UI came back at that time. We then turned off the auto backup function in the UI and this morning there was no issue. Can we get an RCA about what happened with our server during this time? Is there some tuning that needs to be done to allow the auto backup to work smoothly?

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Todd Scholl [23/Nov/18 08:15 PM]

I have attached a copy of the console for the time in question. Let me know if you need other data.

[ Permlink | « Hide ]

Robin Shen [23/Nov/18 10:44 PM]

Looks like there are too many audit log entries in your database. Please remain auto-backup off and configure less log entry preserve time in system setting. After one night, the audit log entry will be cleaned up and then enable auto-backup. Also if UI blocks next time, please run below command from QB server to get a stack trace:
/path/to/jdk/bin/jstack <QB JVM process id> (not the wrapper process id)

[ Permlink | « Hide ]

Robin Shen [23/Nov/18 10:46 PM]

Also I noticed that memory of your QB server is too small to handle many database entries. Please increase it by editing property "wrapper.java.maxmemory" in "conf/wrapper.conf". I'd suggest to give it at least 6G for millions of database entries.

[ Permlink | « Hide ]

Todd Scholl [26/Nov/18 05:05 PM]

Robin,
Thank you for the info. Will work with my team on these recommendations.

We have 16 GB of memory space on this server. Is there any down side to increasing the max memory further? say 12 - 14 GB?

[ Permlink | « Hide ]

Robin Shen [26/Nov/18 11:15 PM]

Giving it too much memory will cause noticeable JVM pause when it garbage collects memory. I'd suggest 8G for maximum.