History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3299
Type: Bug Bug
Status: Closed Closed
Resolution: Won't Fix
Priority: Major Major
Assignee: Robin Shen
Reporter: Todd Scholl
Votes: 0
Watchers: 1

If you were logged in you would be able to see more operations.

Automated Database Bacup Causes issues with QB Ui

Created: 23/Nov/18 08:02 PM   Updated: 19/Jan/19 08:25 AM
Component/s: None
Affects Version/s: 7.0.16
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
File Attachments: 1. Zip Archive console.log.zip (1.02 Mb)

Operating System Linux 4.4.114-94.14-default, amd64
OS User Name root
JVM Java HotSpot(TM) 64-Bit Server VM 1.7.0_79, Oracle Corporation
QuickBuild Version 7.0.16 - Fri May 12 09:57:15 EDT 2017
Total Memory 1.98 GB
Used Memory 1.02 GB
Configurations 7629

 Description  « Hide
On the morning of 11/22 at 1 am, the Quickbuild UI became non responsive. You could hit the sign in page but any attempt to login was met with a spinning browser. A look in the log files for the time showed only that the Automated DB backup had kicked off. In addition there were a number of messages indicating that jobs scheduled for 1 am we put in a "sleep" state. As we were unable to get to the UI even after waiting over an hour, we rebooted the system around 2 am. The UI came back at that time. We then turned off the auto backup function in the UI and this morning there was no issue. Can we get an RCA about what happened with our server during this time? Is there some tuning that needs to be done to allow the auto backup to work smoothly?

 All   Comments   Work Log   Change History      Sort Order:
Todd Scholl [23/Nov/18 08:15 PM]
I have attached a copy of the console for the time in question. Let me know if you need other data.

Robin Shen [23/Nov/18 10:44 PM]
Looks like there are too many audit log entries in your database. Please remain auto-backup off and configure less log entry preserve time in system setting. After one night, the audit log entry will be cleaned up and then enable auto-backup. Also if UI blocks next time, please run below command from QB server to get a stack trace:
/path/to/jdk/bin/jstack <QB JVM process id> (not the wrapper process id)

Robin Shen [23/Nov/18 10:46 PM]
Also I noticed that memory of your QB server is too small to handle many database entries. Please increase it by editing property "wrapper.java.maxmemory" in "conf/wrapper.conf". I'd suggest to give it at least 6G for millions of database entries.

Todd Scholl [26/Nov/18 05:05 PM]
Thank you for the info. Will work with my team on these recommendations.

We have 16 GB of memory space on this server. Is there any down side to increasing the max memory further? say 12 - 14 GB?

Robin Shen [26/Nov/18 11:15 PM]
Giving it too much memory will cause noticeable JVM pause when it garbage collects memory. I'd suggest 8G for maximum.