History | Log In     View a printable version of the current page.  
   HOME       BROWSE PROJECT       FIND ISSUES   
    QUICK SEARCH:  
Issue Details (XML | Word | Printable)

Key: QB-2852
Type: Bug Bug
Status: Open Open
Priority: Major Major
Assignee: Robin Shen
Reporter: Tomasz Sadowski
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

'could not deserialize' during Perforce checkout

Created: 29/Nov/16 02:14 PM   Updated: 15/Dec/16 12:15 AM
Component/s: None
Affects Version/s: 5.1.16
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
Environment: Ubuntu 14.04.2 LTS, 131GB RAM, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz


 Description  « Hide
We have 'could not deserialize' issue during Perforce checkout. This issue happens from time to time but introduces noise in our system which our customers are complaining about.

Please let us know what could we do to get rid of such issues.

Below is stack trace with this error:

org.hibernate.type.SerializationException: could not deserialize
        at org.hibernate.util.SerializationHelper.doDeserialize(SerializationHelper.java:253)
        at org.hibernate.util.SerializationHelper.deserialize(SerializationHelper.java:284)
        at com.pmease.quickbuild.persistence.AgentQueryCarrier.invoke(AgentQueryCarrier.java:59)
        at com.sun.proxy.$Proxy22.uniqueResult(Unknown Source)
        at com.pmease.quickbuild.entitymanager.impl.DefaultBuildManager.getPrevious(DefaultBuildManager.java:128)
        at com.pmease.quickbuild.persistence.SessionInterceptor.invoke(SessionInterceptor.java:49)
        at com.pmease.quickbuild.model.Build.getPrevious(Build.java:869)
        at com.pmease.quickbuild.model.Build.getChangeBase(Build.java:1284)
        at com.pmease.quickbuild.repositorysupport.Repository.getChanges(Repository.java:393)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.CGLIB$getChanges$78(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37$$FastClassByCGLIB$$d63b8bfc.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.getChanges(<generated>)
        at com.pmease.quickbuild.repositorysupport.Repository.getChanges(Repository.java:367)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.CGLIB$getChanges$79(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37$$FastClassByCGLIB$$d63b8bfc.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.getChanges(<generated>)
        at com.pmease.quickbuild.repositorysupport.Repository.checkout(Repository.java:211)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.CGLIB$checkout$86(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37$$FastClassByCGLIB$$d63b8bfc.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$26b7fb37.checkout(<generated>)
        at com.pmease.quickbuild.repositorysupport.CheckoutStep.run(CheckoutStep.java:70)
        at com.pmease.quickbuild.repositorysupport.CheckoutStep$$EnhancerByCGLIB$$ca33999c.CGLIB$run$0(<generated>)
        at com.pmease.quickbuild.repositorysupport.CheckoutStep$$EnhancerByCGLIB$$ca33999c$$FastClassByCGLIB$$2da190dc.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.repositorysupport.CheckoutStep$$EnhancerByCGLIB$$ca33999c.run(<generated>)
        at com.pmease.quickbuild.stepsupport.Step.execute(Step.java:539)
        at com.pmease.quickbuild.stepsupport.StepExecutionJob.executeStepAwareJob(StepExecutionJob.java:31)
        at com.pmease.quickbuild.stepsupport.StepAwareJob.executeBuildAwareJob(StepAwareJob.java:47)
        at com.pmease.quickbuild.BuildAwareJob.execute(BuildAwareJob.java:61)
        at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:71)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
    Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readUTF(Unknown Source)
        at java.io.ObjectInputStream.readString(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at java.util.ArrayList.readObject(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2821.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.defaultReadObject(Unknown Source)
        at com.pmease.quickbuild.model.Build.readObject(Build.java:1338)
        at sun.reflect.GeneratedMethodAccessor2893.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at org.hibernate.util.SerializationHelper.doDeserialize(SerializationHelper.java:247)
        ... 41 more

 All   Comments   Work Log   Change History      Sort Order:
[ Permlink | « Hide ]
Robin Shen [30/Nov/16 12:30 PM]
Is this happening for a particular configuration, or happening randomly across different configurations?

[ Permlink | « Hide ]
Tomasz Sadowski [30/Nov/16 01:38 PM]
This happens across different configuration. I could not find any pattern for this type of issues.

[ Permlink | « Hide ]
Robin Shen [30/Nov/16 02:23 PM]
These intermittent odd issues (and other issues you reported recently) sounds quite like intermittent networking issue. Do you have agents far away from QB server connecting via possibly unstable networking? If so, QB can not do much except to retry as below:
http://forum.pmease.com/viewtopic.php?f=5&t=4033

[ Permlink | « Hide ]
Tomasz Sadowski [12/Dec/16 11:02 AM]
Hi Robin,

Thanks for sending the link however this is about adding retry option to groovy and this issue happened during checkout of repository which is build-in step. I am not aware of any option to wrap such step with groovy.

Thanks, Tomasz

[ Permlink | « Hide ]
Robin Shen [12/Dec/16 11:12 PM]
This is not specific to groovy scripting step. You may put that into post build script (advanced setting of configuration) to check if there is certain errors happened for any built-in step, and then re-run the build if desired.

[ Permlink | « Hide ]
Tomasz Sadowski [14/Dec/16 10:06 AM]
Unfortunately this would involve creation of the new build and our assumption is one build per CI run. I meant more about solution for retries which is part of QB when there is i.e. network issue then to retry for a few times. Is there possibly something similar in newer version of QB? This cause us issues which come back to us from time to time. Appreciate your help and suggestions which help us our system robust and highly available. I issued similar ticket earlier but unfortunately I do not have solution which would help us - when there is network issue we can only create new build (manually or automatically) and we lose our machine hours as instead of retrying single step which failed we would need to resubmit whole build and some of the steps would be repeated.

Regards,
Tomasz

[ Permlink | « Hide ]
Robin Shen [14/Dec/16 11:27 PM]
Or you can try below approach for a certain step:
http://forum.pmease.com/viewtopic.php?f=5&t=4022

We do plan to deliver step retrying, but do not have a schedule yet.

[ Permlink | « Hide ]
Robin Shen [15/Dec/16 12:15 AM]
Also instead of retrying at different places, I'd suggest to eliminate the issue either by reducing server load either by splitting it or schedule builds to run at different time frames.
Also worth to mention that we have several customers experiencing odd issues, and it turns out to be anti-virus software in the middle to disturb the communication. Once rules are configured to by pass QB traffic, everything works.


Powered by Atlassian JIRA™ the Professional Issue Tracker. (Standard Edition, Version: 3.7.4-#189) - Bug/feature request - Contact Administrators