TeamDynamix Slowness Issue

May 10, 2022 10:14 a.m.
Outage category: 
TeamDynamix
Location: 
All users of TDNext, and possibly users of the Service Catalog
Status: 
Closed
Resolved alert: 
05/02/2022 9:50 am

TDX performance was impacted to the point of causing login timeouts. This was a vendor issue and was affecting all TeamDynamix users.

Initial symptoms: 

People contacted Support and ITSM Core that they could not access TDX.

Duration: 
05/02/2022 8:58 am - 05/02/2022 9:50 am
Impact to Mason: 

Users of TDNext could not log in due to slowness issue causing time-outs. Users of Service Catalog to request services or report issues may have experienced time outs as well.

Affected Services: 
Application Software
Other Affected Services: 
TeamDynamix application (TDNext, Service Catalog) affected.
ROOT CAUSE ANALYSIS
Cause: 

The performance issue was the result of a long running logging process which feeds application monitoring and performance information to the TeamDynamix application/web servers. The process caused CPU spikes on several of the application/web servers which slowed the response time of the TDNext application.

Resolution: 

The process was disabled, which restored performance around 9:08am Eastern on May 2nd.
Prevention: 

A long-term resolution was put in place during the maintenance window on Tuesday morning, May 3rd.

STATISTICS
Service Team: 
Vendor (TeamDynamix)