History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3399
Type: Bug Bug
Status: Open Open
Priority: Minor Minor
Assignee: Robin Shen
Reporter: Todd Scholl
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Parallel Steps causing a double trigger of script on node

Created: 05/Jun/19 02:16 PM   Updated: 14/Jun/19 02:23 AM
Component/s: None
Affects Version/s: 8.0.32
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
File Attachments: 1. Text File full-log(1).txt (359 kb)
2. Text File full-log.txt (379 kb)
3. Text File full-log_build_id_19747626.txt (359 kb)
4. Text File full-log_build_id_19781143.txt (379 kb)
5. Text File full-log_build_id_19781143_show_steps.txt (763 kb)

Environment:
System Date and Time 2019-06-05 09:49:12
Operating System Linux 4.4.114-94.14-default, amd64
JVM Java HotSpot(TM) 64-Bit Server VM 1.8.0_131, Oracle Corporation
QuickBuild Version 8.0.32 - Thu Jan 03 21:16:14 EST 2019
Total Heap Memory 5.33 GB
Used Heap Memory 696.15 MB


 Description  « Hide
Support,
We have had an issue with a job we set up to trigger a script across 8 servers using a parallel step sequence.Some time the job runs fine but, about twice a week, 2 servers in the parallel steps will fail out because the script was run twice. We are mystified as to the cause since on other days, the job runs fine.

What we know:
1. The steps trigger a ksh script that creates a log with a time stamp as its first step.
2. We know from that time stamp that the 2nd fire is about 1 - 4 seconds after the first
3. There are no repeat parameters or retry parameters setup for this job.
4. When you set the job up as sequential that the issue does not occur
5. There is no second steps running in the build log, it simply fires the script off a second time with the exact same arguments.

I will attach the build log for the 2 failed jobs we have. Let me know what else you need

 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [05/Jun/19 11:46 PM]
Thanks reporting. Will investigate the issue.

Robin Shen [07/Jun/19 03:06 AM]
Can you please send backup of your database (can be taken from QB admin page) and let me know the build and step to check?

Robin Shen [07/Jun/19 03:06 AM]
Backup can be sent to [robin AT pmease DOT com]

Todd Scholl [07/Jun/19 02:53 PM]
Builds to look at:
https://automation.pfsweb.com:8810/build/19747626
https://automation.pfsweb.com:8810/build/19781143

The job is one parallel composition with multiple steps all doing the same thing on different boxes.

Robin Shen [07/Jun/19 02:57 PM]
As a vendor outside of your company, I can not access your internal server.

Robin Shen [10/Jun/19 08:19 AM]
Thanks for sending the backup. I restored it successfully now. For the failed build log, please let me know the corresponding build id, also please check the option "show steps" when download the build log.

Todd Scholl [12/Jun/19 02:28 PM]
Build logs re attached with build ids referenced

Todd Scholl [12/Jun/19 02:30 PM]
Second build has rolled out of history (More then 30days) but I have re downloaded the build log for 19781143 with show steps

Robin Shen [14/Jun/19 02:23 AM]
Thanks for sending the log. Checked carefully with code but still can not find the reason. Is it possible that you upgrade to QB9 which has additional debug logging for this issue? It can be downloaded from:
https://build.pmease.com/build/4898

After upgrading to this version, please edit general setting of the configuration in question to use debug logging, then reproduce the issue to send build log (still with "show steps" option ticked).