[#QB-3617] There are two types error ( Read timed out and Error writing to server errors) that occur in many nodes of our quick build.

QuickBuild

There are two types error ( Read timed out and Error writing to server errors) that occur in many nodes of our quick build.

Created: 12/Sep/20 07:26 AM Updated: 18/Feb/21 08:23 AM

Component/s:

None

Affects Version/s:

9.0.22

Fix Version/s:

None

Original Estimate:

Unknown

Remaining Estimate:

Unknown

Time Spent:

Unknown

File Attachments:

None

Image Attachments:

1. Error Writting to Server.png (131 kb)	2. KOR-MACHINE05.png (65 kb)	3. linux machine error - timeout expired 300000.png (248 kb)
4. qbuild error.png (167 kb)	5. qbuild error1.png (42 kb)

Description

« Hide

We found a serious problem last year, and we solved it by upgrading the qbuild server from 8.0.8 to 9.0.22.
( Issue number : [~~QB-3433~~] This timeout error occurs frequently these days: TimeoutException: Idle timeout expired: 300000/300000 ms

Recently, we're facing same serious problem. There are two types error ( Read timed out and Error writing to server errors) that occur in many nodes of our quick build.
We are still having a timeout issue between our build agents and the ci server. I have attached screenshots for your understanding.
Please see this issue as soon as possible. this is a big issue for us now.

- qbuild log
"caused by: Error writing to server"
"caused by: Read timed out"

- qbuild agent
TimeoutException: Idle timeout expired: 300001/300000 ms

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Robin Shen [14/Sep/20 11:40 PM]

Please upgrade QB server and agents experiencing timeout issue to below build to see if it helps:

https://build.pmease.com/build/5203

[ Permlink | « Hide ]

Robin Shen [12/Sep/20 11:12 AM]

We will increase network timeout to see if it helps.

[ Permlink | « Hide ]

Yoongeon Lee [12/Sep/20 10:55 AM]

Hi Robin

our top engineer mentioned that there are a backup or VM snapshot rotation or similar long-lasting, high network demand transaction on our ci server.
If the issue is caused by heavy downloads or backups, how should we fix the issue? For reference, the issue occurred two days ago, and we didn't make any change on server at that time.
Please check if there were similar issues on other sites.

[ Permlink | « Hide ]

Yoongeon Lee [12/Sep/20 08:21 AM]

Thanks for the quick response, Robin

Builds happen multiple times on many nodes per day.
Regarding large artifacts, I need to check with our admin engineer. and then let you know.

[ Permlink | « Hide ]

Robin Shen [12/Sep/20 08:06 AM]

Also please check if you have builds publishing/downloading very large artifacts.

[ Permlink | « Hide ]

Robin Shen [12/Sep/20 08:00 AM]

The timeout indicates that server might be loaded. Is this happening when there are many builds running?