Jump to content

[2019-12-07] Server failure incident and rollback


Guest

Recommended Posts

Greetings Everyone,

At around 13:41 2019-12-07 the servers stopped working and was not possible to login and players in-game was stuck, the following is report on what happened and the revert back.

Around a week ago we did few changes on our backup system to reduce a case where the server would freeze for seconds twice a day, however the backup script had a little mistake caused it to backup a lot of unnecessary information and also not delete it as soon as the backups was moved to our backup storage, due to this mistake at around 13:01 the server had it storage fully used by the garbage left on our game server and our database server which is hosted on the same machine was not able to commit the changes to disk, in which after the ram was fully filled with uncommited changes also caused the database server to no longer process the game server requests to add more data and caused it to freeze.

Due to the previous errors also the future backups was not being collected and last viable backup was on 12:31 7th December which is around 30 minutes of when the incident happened and 1 hour from when the servers fully froze and stopped working, we had to revert to this backup to ensure no data corruptions happened during the time the server was going wiggly wiggly.

We sincerely apologize for this and we're now taking further actions to ensure this incident won't happen again.

With this said, enjoy double experience and double drop rate for the coming 24 hours.

Link to comment
Share on other sites

  • 2 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...