Survival Problems

Those of you playing today might be confused why our Survival world wasn’t available for a few hours. So here’s what happened:

Here at MCAU we want to make sure we don’t lose our world data for any reason. So we make backups. Turns out we had made enough backups to fill up the server’s hard drive completely, and that’s what happened last night. The server started crashing because there was no room to save the world any more, so we deleted some older backups to make room.

Unfortunately, during the time when the server was trying to save with no space, it started saving some chunks over the top of other ones, corrupting the world. The server kept crashing because of the corruption, so today we tried to fix it.

This didn’t work so well the first time because the backup I tried to use was the one that the server was trying to make when it ran out of space. So most of that backup never got saved… There were a lot of upset people who found their houses were gone! Luckily we make backups every 2 hours, so I have now used the next-earlier one from about 8pm last night, which wasn’t affected.

In addition to this I’ve “rolled forward” the server from that point onwards, using our LogBlock system. The same way we can roll back griefing, we can roll forward from our backups, so no progress should have been lost.

Sorry for all the inconvenience, and I’m going to make sure this doesn’t happen again: Next time the server starts running out of space it’s going to automatically send a few emails out to make sure we know about it and can make some room before things get messy.

P.S. Unfortunately some people’s /home settings were lost, they weren’t backed up with the world. Not much way we can get those back, sorry.

Thanks for your patience, and thank you for flying MCAU!

Thanks, Amazon!

Today I’ve finished moving everything from Amazon’s Singapore cloud to their Sydney cloud. We’ve still got the same DDOS protection as we’ve previously had, but it’s now a lot closer to home. You’ll notice the change mainly because the latency has dropped significantly (by about 50msec) Yay!