Back to Blog
July 17, 2020

Outage Response: Building a More Resilient Virtual World

Kaitlyn Olsson
bottom
Ready to try Virbela?
Virbela empowers you with the virtual spaces and social connectivity your organization needs to thrive remotely.
Book a Tour

How We Responded and What Steps We're Taking

Thanks again for your patience and understanding of Wednesday's outage. The performance and reliability of your virtual world is our top priority as we grow to meet the incredible demand we're experiencing. In an effort to provide complete transparency, I've asked my team to write this detailed incident summary that speaks to what happened, what we learned, and the steps we are taking to greatly improve the resilience of our platform.

---

Incident Response: What Happened, How We Responded and the Steps We're Taking

Early Wednesday morning, VirBELA Open and Private Campuses were unreachable for many of you for approximately 4 hours. First, we’d like to apologize to all of our customers and users that were impacted. We also want to provide an explanation of what happened and the steps we’re taking to mitigate similar issues in the future.

On July 16th at approximately 5 AM PDT, our database provider, IBM Compose, experienced an unplanned total outage in their US East server cluster. This outage caused our user authentication system to go down, which is the root cause of the login issue our users experienced.

One of the main reasons we chose Compose was their robust backup and recovery features. Unfortunately the outage also caused the Compose web portal that gives us access to our backup and recovery tools to go down as well. This prevented us from recovering in minutes by switching to a different server cluster.

Although our servers were still down, as soon as the Compose web portal was operational we were able to retrieve our backup and run our recovery protocol to shift the authentication system to another server cluster. However, because our backups are scheduled on a daily basis we lost about 20 hours worth of user registrations and admin setting changes.

We decided that getting the servers up immediately outweighed the potential data losses, and our engineering team was able to stand up a new environment in a matter of minutes. We were able to fully recover all the data that was initially lost, and are currently working with individual customers to provide missing data.

We learned two main lessons dealing with this issue: our backups need to be housed using a separate vendor to prevent a single point of failure, and our customer communication needs to happen much more rapidly.

Steps we’re taking to prevent and mitigate this issue in the future

  • We’re evaluating our relationship with this partner
  • Our engineering team has started building a backup process using another vendor
  • We’re investing in additional application monitoring solutions to enhance our visibility into 3rd party services
  • We are implementing a new process to notify customers more quickly in the event of another outage
  • VirBELA teams are additionally evaluating all systems and processes for redundancy and recovery

Again, we apologize to all of our users for this outage. We take the performance and reliability of VirBELA and the worlds we build very seriously.

Please arrange for a call with our leadership team if you have any questions or concerns that we can address with you or your teams.

Back to Blog
eXp World

More from Virbela

Feb 5, 2021
Usher in a Culture of Innovation with These 5 Tips from Duncan Wardle
Read More
Jul 17, 2019
Introducing the Large Auditorium
Read More
Apr 29, 2020
Virtual Reality Comes to VirBELA
Read More

Are You Ready?

Experience the world of Virbela for yourself.