Banner Issues Earlier Today

Outage category: 
Banner
Location: 
Access to Banner
Status: 
Closed
Resolved alert: 
10/29/2024 11:20 am

Registration for Spring 2025 term opened for students at 10:00am on 10/29/24.  There was a spike of activity at that time.  However, sessions were not being released as students completed registrations.  They continued to grow until the database would not accept anymore.

Once the maximum sessions limit was reached, no new sessions were created.  This cause other attempting to log into Patriot Web or  Banner Admin Apps to fail.  Jobs submitted through jobsub or appworx would fail because they could not connect.  API calls from Ellucian Workflow and Salesforce also could not connect

Initial symptoms: 

Performance was being monitored. We noticed the max sessions limits get hit. As sessions were ending, the server continued to have problems with CPU and memory. This caused the oracle listener to fail 4 times.

Duration: 
10/29/2024 10:00 am - 10/29/2024 11:20 am
Impact to Mason: 

Students attempting to register needed to wait. Others could not log into do work. Jobs needed to be rescheduled

Affected Services: 
Patriot Web
Other Affected Services: 
Banner Student Registration
ROOT CAUSE ANALYSIS
Cause: 

There were two primary causes of the problems.

1. The version of Student Registration (9.32.1) had a defect with was causing it to not release sessions.

2. The server was not properly configured in the new RHEL 8 version which caused lingering issues.

Resolution: 

To address the issue with the Student Registration application, Ellucian recommended that we downgrade to the previous version (9.31.1). This resolved the issues with the hanging sessions.

Prevention: 

The best way to ensure that this does not occur again is to conduct a stress-test for registration prior to the Spring and Fall registration to identify and resolve any issues prior to opening up registration to the students.

STATISTICS
Service Team: 
CCSO, Database Middleware and ERP Support, ITSO