Views:

Symptoms

Microsoft System Center Service Manager may exhibit performance issues or tasks may fail to complete. Primary symptoms of this issue may include:

• Subscription-based and scheduled workflows may be noticeably delayed or appear not to run as expected.

• CMDB updates and related processing may appear slow.

• Large imports from connectors or third-party tools appear to take too long or do not complete (runtime time-outs).

• Grooming tasks do not complete on time (transaction time-outs).

• Tasks involving database lookups take longer than expected.

 

Cause

In System Center Service Manager (SCSM) the Entity Change Log (ECL) tracks all changes to all CMDB entities. Related entity changes, or changes made to other entities and their properties are tracked in the Related Entity Change Log (RECL). These logs fundamentally track all the changes and as such have a significant I/O activity rate. As they are critical to the system, any negative performance related to these tables can impact overall performance.

The primary function of these tables is to allow asynchronous workflow processing to be tracked. Entries in the table are reviewed to determine if any workflows are subscribed on the associated change, if so the matching workflows are dispatched. If a large volume of changes occur and cause a large number of workflows to run this can impact available server resources. Additionally, these workflows may cause additional changes, which add to the ECL and could, through relationships, affect more objects and workflows (echo effect).

The log entries in the table are processed in a sequential manner, where a high watermark is used to identify the next change to be reviewed. A grooming job runs nightly to remove previously processed entries. If this grooming job fails or if the volume of the changes is greater than it can process within its parameters, then a backlog will build up. This artificially bloats the database and can create a negative impact on overall performance. This may require manual intervention to bring the Entity Change Log back down to a sustainable size, Microsoft should be consulted on the approach.

Provance has a number of workflows which are subscribed to changes related to our Hardware Catalog Items and Hardware Assets additionally workflows monitor changes in Microsoft’s Windows Computers and Deployed Computers CI’s. Provance has noticed that the following items may increase the amount of data in the ECL and negatively impacting performance as a result:

1. Having too many subscribed workflows.

2. Large volume of changes made to the Windows/Deployed Computer CIs by the Active Directory (AD) connector and the Systems Center Configuration Manager (SCCM) connector.

3. Large volume data imports through SCSM CSV Import, PowerShell or the Provance Data Management Pack.

Resolution

To reduce the size of your ECL table and enable grooming to keep the ECL table size under control, you should review the following list to determine whether they apply in your environment:

• Beware of scheduling conflicts where connectors and high volume changes (imports etc.) conflict with each other and/or the ECL grooming job.

• When performing data imports using tools like the Provance Data Management Pack, consider breaking them down into smaller amounts that can be managed more easily by SCSM. Provance recommends limiting the imports to 5,000 objects at a time taking into account the number of object classes being create/updated.

• Analyze the workflow subscriptions and look for opportunities to reduce their number or complexity. Conduct A-B testing with and without all custom subscriptions to identify any negative impact to performance.

IMPORTANT NOTE: Microsoft has implemented an Exclusion List in SCSM 2012. Microsoft or other technical resources may recommend excluding all relationships with MaxCardinality = 1. While this may be acceptable for a native Microsoft installation, adding Provance relationships to the exclusion list will have an impact on Provance functionality; it may also have an impact on your workflows.

Additional Information

http://blogs.technet.com/b/servicemanager/archive/2009/09/18/data-retention-policies-aka-groomingin-the-servicemanager-database.aspx

http://blogs.technet.com/b/servicemanager/archive/2013/01/14/troubleshooting-workflowperformance-and-delays.aspx

Applies to

Microsoft System Center Service Manager 2012

Provance IT Asset Management 2012