Random errors on creating order through API

[kqian]

When creating order by calling the API, occasionaly we get errors with error message saying something related to "disposed business object" or "executing business rule". Below is some examples of error message we have seen. Re-run the request later, it may get through fine.

Error: Trying to get data from a disposed business object!
[Error executing business rule 'SoftPro.BusinessRules.Core.PayeeAggregator'
[Error executing business rule 'SoftPro.BusinessRules.Core.FindProrationCharge'

So it looks like the problem is not due to the prder data itself, it may be caused by some corrupted state in SoftPro, the error message may not be accurate about the error.
Another thing I found is that when this type of error ocurrs, the orderNumber we use for the call got saved into PendingOrder table and locked (I have to delete it from the table to re-run the request).

Code: Select all

            using (IOrder order = ot.CreateOrder())
            {
                try
                {
                }
                finally
                {
                    // This MUST be called to release any locks on the order.
                    ot.CloseOrder(order);
                }
            }

For each API call, we have the try/finally block to explicitly close the roder instance, I don't know why the order number still get locked in "PendingOrder" table. In normal situation when errors occur, our logging code will be called to record the error and the OrderNumber will not be locked. But in this type of error, I don't get logging message and the order number is locked.

So my questions are:

1) What happened that caused this type of errors, what can I do cleanly handle the errors?
2) Even though the order has been closed in the Finally block, why the state cannot be cleanly cleared? Why the order number still get locked?

Thanks.

Kevin

[Randy Mellow]

Hello Kevin.

The default behavior of Select is to keep a record of the newly-created order number in the PendingOrder table until the order is saved (or successfully exited/closed) for the fist time. Currently, there is no automated "clean up" of these pending records and, if an error occurs during the creation (or save/closing) of an order the record will remain in the pending table which will prevent that number from being used again.

I would need more information to speculate about what may be causing the errors you are seeing. For example, are the errors happening during the save of the order or during the setting of order data or during the creation itself. We have been investigating issues involving failures during the opening of an order and these typically involve a corruption of data that causes the business engine to get "confused". This usually results business rules failing to run. But we haven't seen these during the creation of a new order.

It is possible you've uncovered a bug in the API. The "disposed" message usually indicates a problem with resource/reference management, the type of management prevalent in the API. But, in order to find out for sure we'd have to be able to reproduce the problem in our environment. Do you have a code sample that can reproduce the problem?

[kqian]

Thanks for the help.
I think we can rule out the data problem. If the data was wrong, it would be handled in the normal error handling path, we have seen that with no problem. Also re-process the order will succeed. So like you said it should be happening either at ot.createOrder(...) or ot.SaveOrder(...).

But saving the order is inside the Try/Catch/Final block and we have logging in the catch, if anything wrong there we should have the info logged, but we did not see anything in logging. So most likely it is at creating the order instance. Unforunately I do not have a isolated repro sample for this issue. I will try to see if possible to get one.

Thanks.

Kevin

[Hadi Chemaly]

Kevin, what kind of application are you creating? That is, is it a Web service, a WCF service, other?

[kqian]

I got more detail as below. Stack trace indicates it crashed at the point of ot.GetOrder(). We are calling SoftPro API at a WCF Service with Per-Call concurrency, so multiple requests can be handled at the same time. Is SoftPro API thread-safe? Could this be related to concurrent calls to SoftPro API (may even for the same order number)?

7/28/2011 5:07:59 PM
Error ocurred in Test - SoftProSelect

[Error executing business rule 'SoftPro.BusinessRules.Core.CountersignaturePolicy'
Inputs:
1. Countersignature 0116fd1f-52b9-e011-b92b-00155d005d05
2.
3.
] in [SoftPro.BusinessRules.Base]

Incoming XML:
Date: 7/28/2011 5:07:59 PM
Message: SoftProSelectOrderHelper:GetOrderData
Machine: TEST-WCF
User: automation
Current Process: RCOLegal.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null Entry Assembly: Unknown
Error: Error executing business rule 'SoftPro.BusinessRules.Core.CountersignaturePolicy'
Inputs:
1. Countersignature 0116fd1f-52b9-e011-b92b-00155d005d05
2.
3.

Source: SoftPro.BusinessRules.Base
Method: Execute
Type: SoftPro.BusinessRules.Base.BusinessRuleBase

Stack:
at SoftPro.BusinessRules.Base.BusinessRuleBase.Execute(Object[] inputs, ExecutionInfo ruleInfo)
at SoftPro.BusinessObjects.Base.Engine.GetRuleDataValue(IBusinessObject businessObject, String ruleInstanceName, Object potentialNewValue)
at SoftPro.BusinessObjects.Base.BusinessObjectBase.FulfillDataItemNormally(Dependency depItem, DataValue sourceItem)
at SoftPro.BusinessObjects.Base.BusinessObjectBase.GetDataValue(String dataItemName)
at SoftPro.BusinessObjects.Base.BusinessObjectBase.GetDataItem(String dataItemName)
at SoftPro.BusinessObjects.Base.Engine.ReleaseCalculatedChildRequests()
at SoftPro.BusinessObjects.Core.Order.Load(Guid primaryID, ContextRoot rootContext)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetBusinessObject(String contextName, Guid primaryKey, ContextRoot rootContext, Boolean allowDisambiguation, SqlDataReader openReader, Boolean allowLoad, Boolean loadingAsCopy, Boolean acquiringRoot, Guid idForNewObject)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetRootObject(String contextName, Guid primaryKey, ContextRoot rootContext, Boolean loadingAsCopy)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.AcquireRootReference(String contextName, Guid primaryKey, Boolean loadAsCopy)
at SoftPro.OrderTracking.Client.OrderTracking.OpenOrderObject(Guid id)
at SoftPro.OrderTracking.Client.OrderTracking.OpenOrder(Guid id, Boolean readOnly, Boolean isTemplate)
at SoftPro.OrderTracking.Client.OrderTracking.GetOrderOrTemplate(OrderQuerySpec spec, Boolean readOnly, Boolean isTemplate)
at SoftPro.OrderTracking.Client.OrderTracking.GetOrder(OrderQuerySpec spec, Boolean readOnly)
at RCOLegal.SoftProSelect.Services.Helpers.SoftProSelectOrderHelper.GetOrderData(String orderNumber) in C:\RCOLegal\Title\Development\SoftProSelect\SoftProSelect.Services\Helpers\SoftProSelectOrderHelper.cs:line 68

Error: Exception has been thrown by the target of an invocation.
Source: mscorlib
Method: _InvokeMethodFast
Type: System.RuntimeMethodHandle

Stack:
at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at SoftPro.BusinessRules.Base.BusinessRuleBase.Execute(Object[] inputs, ExecutionInfo ruleInfo)

Error: Trying to get data from a disposed business object!
Object name: 'LoanPolicy'.
Source: SoftPro.BusinessObjects.Base
Method: GetDataValue
Type: SoftPro.BusinessObjects.Base.BusinessObjectBase

Stack:
at SoftPro.BusinessObjects.Base.BusinessObjectBase.GetDataValue(String dataItemName)
at SoftPro.BusinessObjects.Base.Engine.GetDependencyValues(DataItemReference dataItem, IBusinessObject start, List`1& dependencies, Int32& maxChildrenFound)
at SoftPro.BusinessObjects.Base.InputParameterHandle`1.LookupValue()
at SoftPro.BusinessRules.Core.CountersignaturePolicy.ExecuteImpl(InputParameterHandle`1 sig, InputParameterHandle`1 policies, InputParameterHandle`1 policySigs)

[Melissa McBerkowitz]

The current Select API is not thread-safe. If you're using the API from a WCF hosted service, you will need to include the following attributes on your serivce class.

[ServiceBehavior(CurrencyMode = CurrencyMode.Single, InstanceContextMode =InstanceContextMode.Single)]

[roteague]

This is not a data issue, we have stuggled with the issue on/off for a couple of years. I would suggest that if you need to create orders like this, put your order data into a queue of some type (MSMQ) from your WCF and process them separately, one at a time.

[kqian]

Ok, I will change it to Single mode. Hopefully that will solve the problem. Thanks!

[kqian]

More questions about how to implement our wcf service regarding calling SoftPro API.

Even though our WCF service is Per-Call based, our current execution setup is actually a sequential behavior that a single WCF client sequentially calling our WCF service. For each request, our WCF service will create new instance of SelectServer and OrderTracking to serve that request.

1) Is the creation of different instances of SelectServer/OrderTracking objects for each request causing the problem and why? Is there a way to cleanly dispose those objects?

2) If we have to change the WCF service to singleton, does that mean we should create only one instance of the SelectServer object and OrderTracking object for entire lifetime of the service, and re-use it to serve all requests?

Thanks all for the help.

Kevin

[kqian]

After changing our WCF service to Single instance mode, it's getting better. It works fine most of the time. But I still got one failed yesterday (see the exception detail below). Does anyone know the root cause of this issue that can help us resolve the issue?

Stack Trace:
at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
at SoftPro.BusinessObjects.Base.BusinessObjectBase.PopulateBaseDataIntoLastSeen()
at SoftPro.BusinessObjects.Base.BusinessObjectBase.CreateNew(ContextRoot rootContext)
at SoftPro.BusinessObjects.Core.OrderContact.CreateNew(ContextRoot rootContext)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetBusinessObject(String contextName, Guid primaryKey, ContextRoot rootContext, Boolean allowDisambiguation, SqlDataReader openReader, Boolean allowLoad, Boolean loadingAsCopy, Boolean acquiringRoot, Guid idForNewObject)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetBusinessObject(String contextName, Guid primaryKey, ContextRoot rootContext)
at SoftPro.BusinessObjects.Core.OrderContact.GetNewContact(ContactType contactType, ContextRoot root)
at SoftPro.BusinessObjects.Core.OrderContact.GetNewContact(Int16 contactTypeID, ContextRoot root)
at SoftPro.BusinessObjects.Core.Order.CreateNew(ContextRoot rootContext)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetBusinessObject(String contextName, Guid primaryKey, ContextRoot rootContext, Boolean allowDisambiguation, SqlDataReader openReader, Boolean allowLoad, Boolean loadingAsCopy, Boolean acquiringRoot, Guid idForNewObject)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.GetRootObject(String contextName, Guid primaryKey, ContextRoot rootContext, Boolean loadingAsCopy)
at SoftPro.BusinessObjects.Base.BusinessObjectCache.AcquireRootReference(String contextName, Guid primaryKey, Boolean loadAsCopy)
at SoftPro.OrderTracking.Client.OrderTracking.CreateOrderObject(String orderNumber, String baseOrderNumber, String prefix, String suffix, Boolean isTemplate, Session session, Guid owningProfileID)
at SoftPro.OrderTracking.Client.OrderTracking.CreateOrderOrTemplate(OrderCreationSpec spec, Session session, Boolean isTemplate, Guid owningProfileID)
at SoftPro.OrderTracking.Client.OrderTracking.CreateOrderOrTemplate(OrderCreationSpec spec, Session session, Boolean isTemplate)
at SoftPro.OrderTracking.Client.OrderTracking.CreateOrder(OrderCreationSpec spec)
at RCOLegal.SoftProSelect.Services.Helpers.SoftProSelectOrderHelper.CreateSoftProOrder(NwTitleOrder softProOrder) in C:\RCOLegal\Title\Development\SoftProSelect\SoftProSelect.Services\Helpers\SoftProSelectOrderHelper.cs:line 223


Inner Exception:
[Object reference not set to an instance of an object.] in [SoftPro.BusinessObjects.Core]

Stack Trace:
at SoftPro.BusinessObjects.Core.OrderContact.get_DivertProceedsTo()