Mobility Route - Slow system response on select nodes – Incident details

Web Portal experiencing major outage

Slow system response on select nodes

Resolved
Degraded performance
Started 8 months agoLasted 4 days

Affected

Mobility Route App

Degraded performance from 4:36 PM to 8:28 PM, Operational from 8:28 PM to 5:25 PM

Web Portal

Degraded performance from 4:36 PM to 8:28 PM, Operational from 8:28 PM to 5:25 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Our mitigation of the issues facing some select customers has been implemented and we continue to monitor the situation. Extensive efforts are underway to address the underlying issue. At this time, we actually suspect the issue has been due to an unforeseen situation caused by a particular user or set of users within the system, causing an overload of stalled requests that were draining resources on the particular node.

    We are also working to implement additional user rate limits, as some users are also elevating the degree of the issue when the system is experiencing this issue, making matters worse.

    We suggest that users not engage in excessive refreshing of pages and submitting search requests, and wait a couple of minutes before doing so again if any slowdowns are experienced.

    Our engineers are engaging in an extensive investigation and increasing logging to further identify issues and ensure Mobility Route is performing at the level of expectations we and you have.

    We will continue to monitor this situation carefully.

  • Identified
    Identified

    Our team has identified the cause of the issue and are actively working to address this issue.

  • Investigating
    Investigating

    We are investigating reports of slow system response on select nodes for web portal users. We request that users experiencing slowdowns not repeatedly and excessively refresh and query the system, as this may also be adding to the current issues and load for select users and accounts.

    If you are not part of the accounts/users experiencing issues, you can ignore this incident report.