The Blackboard service disruption on May 24 had a major impact on many staff and students at the university. This post outlines some of the ongoing activity in response to that incident.
Blackboard has a range of ways to continuously monitor our system. Since the incident, they have been providing us with twice-daily performance updates. These have shown that our Blackboard system has been performing extremely well, with very high responsiveness and no notable issues. This matches with our ongoing monitoring, and indicates that the additional capacity we added to our Blackboard system before the assessment period began has been effective. An investigation by Blackboard has identified the root cause of the incident as a flaw in the underlying database, which has been reported to the database supplier, Oracle. Blackboard and the Digital Education Office have put measures in place to ensure that the issue cannot recur, and Blackboard’s monitoring reports have shown that these have been successful.
All scheduled exams since last Monday morning have run successfully. The DEO is collating data on exam activity daily to check that exams are being completed and is working closely with IT Services and the Exams Office to provide timely and consistent support to students.
Once again, please accept our apologies for the distress that this incident caused.
Blackboard will be performing scheduled maintenance to increase the system’s capacity and resilience over the summer assessment period. This will mean that Blackboard will be unavailable from 1 am on Thursday 20 May for up to two hours. Please accept our apologies for any inconvenience caused.
Should any significant issue arise with an assessment system during the summer assessment period, the following steps will be followed:
- Issue reported: DEO identifies Incident Lead (usually service lead) and collates initial data to assess if this is a major incident. DEO informs colleagues, eg DEO Helpdesk and other DEO staff (via internal Teams group) plus IT Services TLC team (via Teams).
- Initial investigation: DEO Incident Lead reviews available data and consults with internal/external Technical Service Manager to complete initial assessment of problem and its impact (rapid triage).
- Major incident confirmed:
- Incident Lead informs Head of Digital Education (via Teams) and Exams Office (via Teams) and confirms major incident with IT Services to enable their Major Incident process.
- DEO posts to DEO blog using ‘service announcement’ category which then emails info to subscribers and updates feeds (eg Blackboard home page). This information is also used by IT Services to update their status page.
- Head of Digital Education updates senior team in Education and Student Experience.
- Major incident management: IT Services oversees communication to staff and students as part of its Major Incident process. The DEO and IT Services continue to update blog and status page until issue is resolved.
The DEO recommends that relevant school and faculty staff subscribe to the DEO blog and consult the IT Services status page. In the event of a major incident, the DEO blog will have live and detailed updates, while the IT Services status page will contain more high-level information. Both the blog and status page are available to students.
If a technical issue affecting a live assessment is reported directly to schools or faculties, they should check the DEO blog for updates first, as the issue may have already been reported to the DEO. If nothing has been posted on the blog, they should contact the DEO.
Significant work has been undertaken in preparation for the summer assessment period:
– by Professional Services
- Assessment systems’ capacity has been increased to accommodate the higher number of assessments.
- Questionmark has been upsized from 10th May to 15th June.
- Blackboard capacity is being increased over the assessment period.
- Clear communication routes and processes to manage major incidents have been put in place or reinforced.
- Additional “just in time” support is available in the DEO and IT Services.
- Where possible, standardised assessment areas and processes have been created to simplify support and improve the student experience, including:
- Processes which may be needed in the event of major technical issues have been cleared from a policy and data security perspective (eg, do not email assessments, wait until the system is available and submit as usual).
- improvements for students ( eg personalised exam times).
– by schools and faculties
- Every exam should have a backup plan in place before the assessment (eg, use of a late submission point for Blackboard file upload exams).
- Technical settings on exam assessments should be flexible enough to accommodate a short break in service (eg, by not setting MCQ-based exams to auto-submit).
- Relevant central school and faculty staff should subscribe to the DEO blog.
For centrally supported exams the DEO has set up assessments with backups and appropriate technical settings. For locally supported assessment, schools should ensure that this work has been done and any relevant information communicated to students.
IT Services will be completing their emergency upgrades to the University network on Monday 22 March between 5.30 pm and 9 pm. This will affect access to some of the services that the Digital Education Office manages, including Blackboard and Re/Play. Further information can be found in the IT Services status item about this work.
IT Services will be making emergency upgrades to the University network on Friday 19 March between 6 pm and 9 pm and on Saturday 20 March between 6 pm and 9 pm. This will affect access to some of the services that the Digital Education Office manages, including Blackboard and Re/Play. Further information can be found in the IT Services news item about this work.
The partial service disruption that was affecting the Blackboard app has been resolved. From around 11.30 am yesterday, you may have experienced an error when attempting to log in to the app.
The root cause of the issue has been identified as an emergency security fix that was applied to one of the University’s authentication servers. We have now reconfigured the app so that it uses a different authentication server. However, the app will have cached your previous login method.
In order to login again via the Blackboard app, you need to uninstall and reinstall the app on your mobile devices.
If you are still unable to log in once you have uninstalled and reinstalled the app, please contact email@example.com.
Please accept our apologies for the inconvenience that this issue has caused you.
Please check the IT service status page to see how the University’s systems and services are working and for planned maintenance information. You can also check the Blackboard status page.