- Mark as unread from here
- Posted: 2 months ago
-
Post #1
Hi Dominating 12 users,
We recently had a brief outage on the website, and we just wanted to update you on what happened, what we're doing to fix things, and where we’re at with the site.
What happened
Dominating 12 went down at 20:40 UTC (server time) on June 18. We detected the issue, but it wasn’t until 4:40 UTC the next day, June 19, that we were able to fully address it. The server was rebooted, bringing things back to normal. Later that morning, we identified the root cause and applied fixes, so we don’t expect this to happen again.
How it happened
A few days before the incident, during regular maintenance, we noticed that our disk usage was getting close to full. As more games are played, and since we store the history of every game, our disk space gradually fills up. To help with this, we started running a process that automatically deletes game logs from games that finished more than a year ago. This process was running overnight, but we didn’t realize that it was also affecting the server's database logs. As the database operations were logged, the log files grew too quickly, and the server eventually ran out of space. This is what we call “a certified whoopsie”…
What we’re doing now
Going forward, we’ll be clearing the database tables more regularly. This avoids weird spikes like the one that caused this outage. However, we’re also looking into how to better manage disk space overall. We may need to explore options for expanding storage, which could affect the site’s finances, or we might start being more aggressive with deleting older data (such as entire old games, not just their logs).
We know this six-hour downtime caused missed turns and cancelled games. Hoodlum has been working hard to resolve the issues we’ve identified, but there might be a few we missed. If you need help with your game, feel free to reach out to a moderator, and we’ll do our best to find the fairest solution.
The reality of maintaining D12
Dominating 12 is run entirely by volunteers. This includes everything you see (moderation, map-making, etc.) and everything behind the scenes (server management, software maintenance, etc.). While we’ve had several programmers contribute over the years, the day-to-day upkeep has mostly been in the hands of just two people for the last decade. With jobs, hobbies, and life in general, time and energy can be scarce. So, we have to be strategic about what we tackle.
What does this mean for you? We’re committed to giving you the best Dominating 12 experience possible, and we’re not going anywhere. But without more funding or more volunteers stepping up, the site will continue to rely on the goodwill of the few people who keep it running.
How you can help
First and foremost, you can support the site financially by taking out a premium subscription. We’ve never raised the price because we want it to stay accessible, and every little bit helps. Even just a handful of premium subscribers can make a difference in covering small infrastructure costs. If you already have a premium subscription and want to do more, you can buy additional subscriptions and then contact an administrator (find them on the staff page) to transfer it to a different account.
We’re always looking for volunteers. We do ask that you’ve been an active member of the community and are in good standing, but if you’re interested in helping us keep this the best Risk community out there, get in touch with an administrator. You can also dive right in by sharing feedback on new maps, or even creating your own. If you have technical skills and want to contribute to the code, reach out to the admins or programmers (also on the staff page) and let us know about your experience and what you’d like to work on.
And lastly: be kind. As we mentioned, the site runs on volunteers. It’s easy to get discouraged when dealing with negativity, which is all too common online these days. A simple “Thanks Hoodlum, helps a lot!” every now and then can go a long way in showing your appreciation to the people working behind the scenes.
I know this was a bit of a long post, but with things being relatively quiet lately, we thought it was a good time to check in and give an update. See you all out there on the battlefield!
On behalf of the team,
Cireon
We recently had a brief outage on the website, and we just wanted to update you on what happened, what we're doing to fix things, and where we’re at with the site.
What happened
Dominating 12 went down at 20:40 UTC (server time) on June 18. We detected the issue, but it wasn’t until 4:40 UTC the next day, June 19, that we were able to fully address it. The server was rebooted, bringing things back to normal. Later that morning, we identified the root cause and applied fixes, so we don’t expect this to happen again.
How it happened
A few days before the incident, during regular maintenance, we noticed that our disk usage was getting close to full. As more games are played, and since we store the history of every game, our disk space gradually fills up. To help with this, we started running a process that automatically deletes game logs from games that finished more than a year ago. This process was running overnight, but we didn’t realize that it was also affecting the server's database logs. As the database operations were logged, the log files grew too quickly, and the server eventually ran out of space. This is what we call “a certified whoopsie”…
What we’re doing now
Going forward, we’ll be clearing the database tables more regularly. This avoids weird spikes like the one that caused this outage. However, we’re also looking into how to better manage disk space overall. We may need to explore options for expanding storage, which could affect the site’s finances, or we might start being more aggressive with deleting older data (such as entire old games, not just their logs).
We know this six-hour downtime caused missed turns and cancelled games. Hoodlum has been working hard to resolve the issues we’ve identified, but there might be a few we missed. If you need help with your game, feel free to reach out to a moderator, and we’ll do our best to find the fairest solution.
The reality of maintaining D12
Dominating 12 is run entirely by volunteers. This includes everything you see (moderation, map-making, etc.) and everything behind the scenes (server management, software maintenance, etc.). While we’ve had several programmers contribute over the years, the day-to-day upkeep has mostly been in the hands of just two people for the last decade. With jobs, hobbies, and life in general, time and energy can be scarce. So, we have to be strategic about what we tackle.
What does this mean for you? We’re committed to giving you the best Dominating 12 experience possible, and we’re not going anywhere. But without more funding or more volunteers stepping up, the site will continue to rely on the goodwill of the few people who keep it running.
How you can help
First and foremost, you can support the site financially by taking out a premium subscription. We’ve never raised the price because we want it to stay accessible, and every little bit helps. Even just a handful of premium subscribers can make a difference in covering small infrastructure costs. If you already have a premium subscription and want to do more, you can buy additional subscriptions and then contact an administrator (find them on the staff page) to transfer it to a different account.
We’re always looking for volunteers. We do ask that you’ve been an active member of the community and are in good standing, but if you’re interested in helping us keep this the best Risk community out there, get in touch with an administrator. You can also dive right in by sharing feedback on new maps, or even creating your own. If you have technical skills and want to contribute to the code, reach out to the admins or programmers (also on the staff page) and let us know about your experience and what you’d like to work on.
And lastly: be kind. As we mentioned, the site runs on volunteers. It’s easy to get discouraged when dealing with negativity, which is all too common online these days. A simple “Thanks Hoodlum, helps a lot!” every now and then can go a long way in showing your appreciation to the people working behind the scenes.
I know this was a bit of a long post, but with things being relatively quiet lately, we thought it was a good time to check in and give an update. See you all out there on the battlefield!
On behalf of the team,
Cireon
“This is how humans are: We question all our beliefs, except for the ones that we really believe in, and those we never think to question.”
- Speaker for the Dead, O.S. Card
- Speaker for the Dead, O.S. Card
