|
|
|
[
Permlink
| « Hide
]
Robin Shen [18/Jan/21 01:03 AM]
Increasing database connections may not solve the issue completely, but should be able to reduce the possibility of the issue. Especially when the site is busy.
We made the change to our server yesterday. Is there any reasons that you think the configuration of the database connections might cause the issue? Thanks.
Thanks, Robin. We are going to increase it to 50 as your advice, since there are many builds to run per day..
Please increase it to 50 if you have many builds to run per day.
We are using the PostgreSQL database. The configuration of database connections is 25 as in attached file. Thanks
How many database connections you've configured in conf/hibernate.properties?
The server was down again on 12/24/2020 due to that Java heap was out of memory. Unfortunately the system admin was not available to take the memory dump at that time. The issue have not occurred again since that day. I looked into the logs, it seems that number of build servers had network hiccups or down at the middle of the builds, and the server tried to connect to them. That might cause the deadlock as in the log file, and the deadlock consumed the memory and brought the server down. What do you think?
We still keep an eye on the server and build servers and will take memory dump as your advice if the issue occurs again. Thanks, ptinh In the meantime, please get a stack trace of the server when it happens:
/path/to/jdk/bin/stack <QB server JVM process id> It occurred again today, but forgot to take the memory dump prior to rebooting the server. We will do that in next one.
Thanks, Robin When OutOfMemory error occurs, please get me a memory dump by running below command as the same user running QB server process:
/path/to/jdk/bin/jmap -dump:format=b,file=/path/to/dump <QB server JVM process id> Then please put the memory dump to dropbox and send me the link |