Banner Unavailability March 7, 2021

Outage category: 
Applications
Location: 
All users of Banner/Patriotweb
Status: 
Closed
Resolved alert: 
03/07/2021 5:53 pm

All Banner connected applications were unavailable for a time. Error message would depend on the applications. Eventually all were up except Patriot Web. It was giving a “Service Temporarily Unavailable” message after CAS authentication.

Initial symptoms: 

Two problems occurred: First the Oracle upgrade of the Banner Production database failed. Once that was resolved, the upgrade was restarted and completed successfully. Applications connecting to PROD were restarted. All connected except Patriot Web (SSB8) which could not establish connections to the database.

Duration: 
03/07/2021 5:52 pm - 03/07/2021 5:53 pm
Impact to Mason: 

All Faculty, staff and students who wished to use Patriot Web, Banner Admin or other Banner connected applications.

Affected Services: 
Banner Admin, Banner Reports, Patriot Web
Other Affected Services: 
Banner Access Management, Jobsub, Appworx, Banner Workflow, Ellucian Workflow, and all other Banner connected applications.
ROOT CAUSE ANALYSIS
Cause: 

The upgrade failed because the upgrade utility could not lock some of the dictionary objects that it needed to upgrade. This did not happen in test.

We suspect that even though the database was in “upgrade” mode, some connections into the database were still active. PROD was removed from the listener to prevent new connections, restarted the database to remove any connections and was able to restart the upgrade. The restarted upgrade did complete.

After the upgrade finished, the applications were restarted. All came up and connected fine except SSB 8.  SSB 8 runs on Oracle WebTier. WebTier is essentially a “dumb” web server. It is very light weight and just displays the pages as the Pl/Sql gives it.

Resolution: 

For the first problem: Removed PROD from the oracle listener on MensaP to prevent new connections to the database. Restarted the database to kill all existing connections and restarted the upgrade:

For the Patriot Web outage: Copied listener configuration files from the 12c listener home to the 19c listener home.

Prevention: 

Corrective steps noted and will be added to future oracle upgrade plans.

STATISTICS
Service Team: 
DBA Team