Information Technology Services

Incident Management Process

Standard Information

Responsible Office

ITS Enterprise Service Delivery (ESD) – Technology Support & Quality
Management: Support Center

Document Control Number

ITS.ESD-PRS004

Document Version Number

3.0

Last Reviewed Date

5/2/2024

Purpose

The intent of this document is to provide effective incident management within Mason’s Production ITS environment during an incident while simultaneously providing transparency to the root cause and lessons learned after the incident is resolved.

  1. University Policy Number 1305, Reporting Electronic Security Incidents Policy
  2. ITS.ITSO-PROC004, Security Incident Response Procedures
  3. ITS.ESD-PRS003, ITS Unplanned Service Disruption Communications Process
  4. DFARS Clause 252.204-7012, Safeguarding Covered Defense Information and Cyber Incident Reporting

Process Scope

Process

  1. ITS Incident Management is the process of identifying, triaging, and resolving incidents while, at the same time, communicating to the appropriate individuals and/or teams who have an interest in the status of the incident at any point in time. The process begins with the identification of an issue with any service or piece of infrastructure not functioning as expected. Typically, these issues are identified by customer inquiries, automated monitoring/alerting, or identification of an issue by an ITS resource.
  2. The ITS Incident Management process’s main goal is to resolve the issue efficiently while minimizing business impact, security impacts, risk, and cost. All ITS incidents that occur at Mason will be documented in the ITSM tool. To achieve this, the Incident Management procedure includes the following steps.

Applicability

  1. A customer contacts an ITS representative with information regarding a service that is not working as expected.
  2. An ITS representative during their work finds that a service is not functioning as expected.
  3. An ITS Monitoring system shows that a service or asset, examples are a phone, server, database, router, etc. is non-functional and a member of the support group for that service or asset needs to determine what has caused the issue.

Process Inputs and Outputs

Inputs

  1. . Contact from customer (email, phone, chat, or in person).
  2. 2. Technician sees alert in monitoring system.
  3. Automated Email generated by a system is received by the ITSM Tool.

Outputs

  1. Incident Ticket generated, manually or systematically based on input method.
  2. 2. Group assigned to work on Incident ticket.
  3. 3. SLA assigned to ticket based on Priority.

Definitions and Acronyms

  • Incident:A break/fix scenario. A service or asset is no longer functioning. Incidents could include Outage or Problem tickets, which are a higher level of Incident ticket.
  • Incident Ticket: A completed Incident form in the ITSM Tool.
  • Group: A selection of people assigned as pool of users that works on specific types of issues.
  • Service Level Agreements/SLA: Service Level Agreement. There are two forms of SLA associated with an Incident Ticket:
    • Respond By: this is the time limit based on the Priority of the Ticket that an ITS staff member should be contacting the customer to let them know the ticket has been received and a review of the issue has started.
    • Resolved By: this is the time limit based on the Priority of the Ticket by which the Incident should be fixed.

      SLAs do not include the time waiting for a customer or vendor response. The SLA Respond/Resolve times are explained in Appendix A: Ticket Priority Matrix
  • Dispatcher: The ITS staff assigned to review the ticket queue regularly during the workday and assign tickets to other members of the group or to another group if the ticket was assigned improperly.
  • Impact: This represents how many people are affected by the Incident. There are 7 levels:
    • Affects only ITS Service Operations.
    • Affects a single individual.
    • Affects a single building.
    • Affects a single department.
    • Affects multiple buildings.
    • Affects multiple departments.
    • Affects the entire University
  • Urgency: This represents how the customer is affected by the Incident. There are four (4) levels:
    • Work Stopped – No Workaround
    • Work Stopped – Workaround
    • Work Impaired
    • Work not Impaired
  • Priority: This is based on the Impact and Urgency according to the table in Appendix A: Ticket Priority Matrix.
  • ITSM Tool: The current ITSM Tool used by ITS and other Departments/Units at Mason is TeamDynamix. Service Owner: The ITS Director is responsible for a specific service. Incident Manager: The ITS staff chosen by the Service Owner to be the main point of contact during the Incident, Outage, and Problem. The Incident Manager is responsible for all outgoing communication on an Incident ticket regardless of level of ticket (Incident, Outage, or Problem).
  • Outage: An issue has occurred to affect a service. The service could be partially impacted, service is available but is slower than normal, or completely unavailable.
  • Problem: An issue has occurred to affect a service. The service could be partially impacted, service is available but is slower than normal, or completely unavailable, but there is a workaround that can be put in place to mitigate the outage in a temporary fashion.
  • USDC-L Listserv: Unexpected Service Disruption Communications list. This is a list of all managers and above level staff within ITS.
  • ITS Alert: This is a function completed by the ITS Support Center and the Facilities & Infrastructure Operations group. They receive request for alerts to be sent to the ITS Alerts Listserv during outage or Problem situation.

High-Level Process/Step

Formal Incident ticket created

  1. Customers contact ITS. This could be through various methods such as Phone, Email, Chat, In Person, or the TDX Client Portal Self-Servicefunction.
  2. ITS Staff member notices an issue due to monitoring service or while performing a normal job function.
  3. Monitoring tools such as event monitoring platforms, sends a notification.
  • Responsible Group: Varies depending on issue and method of contact
  • Outputs: Incident Ticket

Categorize and Prioritize the Incident

  1. Ticket is assigned to appropriate group.
    1. Responsible Group: Varies depending on issue and method of contact
    2. Outputs: Ticket assigned to Group
  2. ITS Resource (Dispatcher/Manager) for group performs triage on the Incident ticket to determine if the priority is correct and if this issue represents an Outage or Problem scenario.
    1. Responsible Group: Varies depending on issue and method of contact
    2. Outputs: Properly prioritized ticket.
  3. If Incident is determined to be an Outage or Problem, the appropriate Outage or Problem Management process is invoked.
    1. Responsible Group: Varies depending on issue and method of contact
    2. Outputs: Invocation of Outage or Problem Management processes.
  4. Incident Tickets are assigned to Technician
    1. Responsible Group: Member of Group
    2. Outputs: Assigned ticket
  5. Assigned Technician reviews ticket, contacts customer to state ticket has been received and review has begun
    1. Responsible Group: Assigned Technician
    2. Outputs: Ticket status “In Process”
  6. If Assigned Technician has questions for the customer or vendor, contact is made, and the ticket is placed on hold.
    1. Responsible Group: Assigned Technician
    2. Outputs: Customer/vendor contacted, Ticket placed in appropriate hold status: Wait Customer or Wait Vendor
  7. Assigned Technician works on the issue to resolve it. Contacts customer that the issue should be resolved, and customer should test.
    1. Responsible Group: Assigned Technician
    2. Outputs: Ticket in “Resolved” status.
  8. Customer responds issue is not fixed
    1. Responsible Group: Assigned Technician
    2. Outputs: Ticket moved to “Customer Responded” status. Work goes back to Step 5.

Outage Management

  1. The Incident Manager and/or the appropriate team member(s) will engage, in the timeliest manner possible, to confirm the incident being reported is accurate and do an initial assessment to determine the impact and root cause. Once understood, those responding to the incident will work to resolve it as quickly as possible. Creation of the Outage ticket is completed. This is done following the guidelines shown in the Outage Management Knowledge Article.
    1. Responsible Group: Service Owner/ Technicians
    2. Outputs: Outage process initiated.
  2. The Service Owner chooses the Incident Manager, which may be themselves or someone on one of their service groups. The Incident Manager will be separate from the team working on the issue, so they are available to speak with other groups or handle communications.
    1. Responsible Group: Service Owner
    2. Outputs: Outage Ticket assigned to Incident Manager
  3. Based on the severity and complexity of the incident and a reasonable estimate of time needed to resolve, the Service Owner and/or Incident Manager must identify their communication plan to include the frequency of their communications and the audience that needs to be included. Ideally, this communication should be done through the Incident ticket itself where the audience is contained to ITS staff and a limited number of impacted users. For broader communications, ITS Alerts and/or the USDC-L email list should be used as needed. An email can be sent to the entire Mason Community, Senior Leadership and ITS Communications and Marketing will need to be involved.
    1. Responsible Group: Incident Manager /Service Owner
    2. Outputs: Email to specific customers, USDC-L, ITS Alerts, or E-Files Alert as appropriate
  4. The Incident Manager must ensure that the Outage Ticket is kept up to date and contains a full accounting of the troubleshooting, findings and steps used to resolve the issue. If there are multiple incident tickets for a given incident, the Incident Manager will ensure all other tickets are identified as Child tickets and assigned to the Outage ticket. This is done following the guidelines shown in the Outage Management Knowledge Article. It is possible that the Outage Ticket be changed to a Problem Ticket if a workaround is in place while a permanent resolution can be identified and tracked. This is done by changing the Classification of the ticket. The Problem Management process would then be followed. Once email has been sent to either or both groups, regular updates should be sent on a schedule determined by the Incident Manager and Service Owner. Customer notifications should include when the next update will occur where reasonable
    1. Responsible Group: Outage Manager /Service Owner
    2. Outputs: Email to selected customer base
  5. In some instances, resolution of the Incident may require one or more Emergency Request for Change be created and implemented. Change Management policies and procedures should be followed as needed. Change tickets can be created following the steps provided in the Creating an Request For Change Knowledge Article.
    1. Responsible Group: Incident Manager / Technician
    2. Outputs: RFC creation
  6. If communication was sent to customers, the Incident Manager will send either a resolved/all clear type message that explains in general terms for the public what caused the issue and how it was resolved.
    1. Responsible Group: Incident Manager
    2. Outputs: Email sent to customer base
  7. Once the Incident is resolved, the customers affected by the outage will be contacted to ensure they see that the Incident is resolved.
    1. Responsible Group: Support Center
    2. Outputs: Closed Child Tickets
  8. Once all Child tickets are closed, the Outage ticket can be closed.
    1. Responsible Group: Outage Owner/ Service Owner
    2. Outputs: Closed Outage Tickets
  9. Once an Incident is resolved, the Incident Manager will be required to complete an AfterAction Review form, be available for discussions with management and/or peers to share lessons learned or speak to a broader audience to ensure transparency of what caused the incident, how it was resolved, and what measures can/will be taken to prevent it from recurring.
    1. Responsible Group: Incident Manager & Technician
    2. Outputs: After Action Report created and submitted for posting.

Problem Management

  1. Problem Management follows the same steps as Outage Management with one exception: the communication of the workaround. If a workaround is determined, this will be communicated as quickly as possible with the affected users. This is done following the guidelines in the Problem Management Knowledge Article.
    1. Responsible Group: Service Owner, Problem Manager, Technician
    2. Outputs: Problem Ticket used instead of an Outage Ticket.
  2. The Problem ticket will be closed when a permanent fix is in place.
    1. Responsible Group: Incident Manager/Service Owner
    2. Outputs: Closed Problem Ticket
  3. Once an Incident is resolved the Incident Manager will be required to complete an AfterAction Review form, be available for discussions with management and/or peers to share lessons learned or speak to a broader audience to ensure transparency of what caused the incident, how it was resolved, and what measures can/will be taken to prevent it from recurring.
    1. Responsible Group: Incident Manager & Technician
    2. Outputs: After Action Report created and submitted for posting

Artifacts

  • Outage Ticket: A higher-level Incident ticket that can be used when multiple users are affected by the same issue. This ticket can have Child tickets assigned to it, so the overall effect of the Outage can be known. Outage tickets are never assigned directly to a customer but to an Incident Manager.
  • Problem Ticket: A higher-level Incident ticket that can be used when multiple users are affected by the same issue. Problem tickets can have a workaround that is put in place to bypass the issue while a permanent fix is found. This ticket can have Child tickets assigned to it, so the overall effect of the Problem can be known. Problem tickets are never assigned directly to a customer but to an Incident Manager.
  • Incident Ticket: This ticket is used when a single user is reporting an issue.
  • After Action Report: A report explaining what the Initial Symptoms, Root Cause, Prevention, Resolution, what ITS Teams worked on the issue, what effect the issues had on customers, and what systems/services were affected by the issue.
  • Request for Change: A ticket that states what work needs to be done to implement a change in a configuration.

Timetable for Review

This process will be reviewed annually.

Approvals

David Lutz, ITS ESD Technology Support & Quality Management Director

Appendix A: Ticket Priority Matrix

IT service priority matrix showing how urgency and impact levels determine incident priority, response times, and resolution targets.
Ticket Priority Matrix (Accessible)

Corrected Color Legend

  • Emergency (red) indicates the highest-priority outages requiring immediate response and outage management.
  • High (yellow) appears only in the “Work Stopped, Workaround Available” row for Critical and Major impact incidents.
  • High (white) appears in the “Work Stopped, No Workaround” row for Major and Minor impact incidents and in the “Work Impaired” row for Critical impact incidents.
  • Standard (white) and Scheduled (white) are displayed in white cells throughout the matrix.

Urgency: Work Stopped, No Workaround

  • Critical impact: Emergency (red), Outage Management, response within 15 minutes, resolution within 4 hours, available 24×7.
  • Major impact: High (white), response within 1 hour, resolution within 8 hours, available 24×7.
  • Minor impact: High (white), response within 1 hour, resolution within 8 hours, available 24×7.
  • Small impact: Standard (white), response within 8 hours, resolution within 16 hours, available during Business Hours.

Urgency: Work Stopped, Workaround Available

  • Critical impact: High (yellow), Problem Management, response within 1 hour, resolution within 8 hours, available 24×7.
  • Major impact: High (yellow), Problem Management, response within 1 hour, resolution within 8 hours, available 24×7.
  • Minor impact: Standard (white), response within 8 hours, resolution within 16 hours, available during Business Hours.
  • Small impact: Scheduled (white), response within 8 hours, resolution within 32 hours, available during Business Hours.

Urgency: Work Impaired

  • Critical impact: High (white), response within 1 hour, resolution within 8 hours, available 24×7.
  • Major impact: Standard (white), response within 8 hours, resolution within 16 hours, available during Business Hours.
  • Minor impact: Standard (white), response within 8 hours, resolution within 16 hours, available during Business Hours.
  • Small impact: Scheduled (white), response within 8 hours, resolution within 32 hours, available during Business Hours.

Urgency: Work Not Impaired

  • Critical impact: Standard (white), response within 8 hours, resolution within 16 hours, available during Business Hours.
  • Major impact: Scheduled (white), response within 8 hours, resolution within 32 hours, available during Business Hours.
  • Minor impact: Scheduled (white), response within 8 hours, resolution within 32 hours, available during Business Hours.
  • Small impact: Scheduled (white), response within 8 hours, resolution within 32 hours, available during Business Hours.