Event Type: Error
Event Source: SoftPro Select Server (SOFTPRO_TEST)
Event Category: None
Event ID: 0
Date: 10/6/2010
Time: 2:17:41 PM
User: N/A
Computer: SOFTPRO
Description:
General notification failure on '2327'. The operation is not valid for the state of the transaction.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Any light you can shed on this issue would be appreciated. I notice that events are filling up in the EventNotification table and are not being processed.
Currently, all event notification entries stay in the table and are retried, in order, each time the dispatcher executes. Are your subscriptions primarily email or service destination based? If the notifications are primarily interacting with services, are you seeing many timeout errors or other communication errors in the log? I see that you are on a test machine - if you have a lot of "dead" addresses in that table that point to services that are no longer active, you may be running into a condition where the failures are being dealt with at a slower pace than the events coming in. For example, say you have 100 records in the queuing table when our dispatcher starts up. If the first 50 fail due to timeout errors (often, 60 seconds each), that's almost an hour before the first successful notification kicks off. But in the meantime, many more new notifications are flowing in from the system, so you can start to run into a condition where the queue begins to back up. How large has your table become? Does it keep failing on the same records?
As an aside, the dispatcher runs on an independent thread, and it processes the table in its entirety based on the state of the table at the point it kicks off. Only one dispatcher runs at a time.
If you have any other examples of event notification error entries, include them here (in order) so we can try and figure out what has happened and where to go from here.
We are using a service based notification process. I do think there is an infastructure issue here that I'm trying to track down, and was hoping the entries may give me some clue where to look. I see entries relating to time out issues, but it isn't consistent. What I'm most concerned about is that at the moment I see almost 200 notifications waiting, but none of them seem to be processing.
A few other details that might help us pinpoint this:
If its not already running, the dispatcher kicks off in response to a new notification being added to the queue. So if there are records in the queue, but no event activity, those records won't be processed until the next notification request arrives.
I'll just mention that if this is a test environment, records can be simply wiped from the queue table without any problem on our end. Technically, that's true for a live environment as well, but you have to consider the business effect of those notifications disappearing.
A change was made in Boylan (2.6) SP1 that corrected an issue with WCF service notifications that would cause the communications channel to not clean up adequately. This could cause us to overrun a particular service's max concurrent channel count at which point the service would begin to deny notifications until it was restarted.
SP1 is 2.6.40705.30. Besides the event notification improvement I mentioned, there were a number of other resolved issues throughout the application, as well as a noteworthy performance enhancement to Select's report generating system. If you are having any issues at all with service notification timeout errors, I would strongly recommend the upgrade to help with your situation.
As for kicking off the dispatcher, note that it does not run in response to a database change (e.g. a trigger), rather, it is launched by the Select Server as event notifications are requested.
BTW, there is one other issue, perhaps they are related. We are finding that the Select Services instances are shutting themselves off. I can't find anything in the event log about it, but on occasion we find the service offline.