Hi everyone,
As you've surely noticed, recently there's been a lot of forum downtime and server woes. It's been a challenging and frustrating experience, as site traffic hasn't really gone up in the last year or two, so in a way this is sort of out of nowhere. The other challenge is that the $ we get from ads isn't going up, and the servers already cost a lot, so it's not easy to tack on another $x per month for an upgrade that will help (like more ram, more disks, etc), or worse, to upgrade to another server that has a bigger capacity, which jumps a huge amount in monthly price.
We've done the following today to try to help with the strain:
-Vaccuumed out the postgres database (the main site database); it was pretty messy so this probably helped a little bit
-Moved mysql/the forum to the other server, which previously just stored the activity feed (as it's quite a beast/needs a lot of resources). We think this will help because mysql is a resource hog of its own, but probably not enough to make a huge difference.
-Upgraded both servers to the latest and greatest packages/software while we were in there
-Reindexed everything in the activity feed database (mongo) and updated mongo to 2.x, which we're hoping helps with performance on that end
-Tested the RAID array to make sure it's not degrading (it's not)
-We also plan on moving the svn repository into a private github server, which will make a super minimal improvement maybe because the repository won't live on our servers, but yeah.
The server is faring better right now, but for all I know it's because the site was down for awhile. We're going to look at the load over the next few days and see if it's getting better.
But, we think the server is CPU-bound at the moment, and upgrading cores is going to be really challenging. It'll require physically changing servers again, and the next server level up jumps a huge amount in price. Because there's not consistent increases in monthly ad revenue coming in, having a higher monthly bill is unsustainable (and while donation spurts do help, they also generally aren't consistent and can't be counted on for ensuring monthly bills can be paid)
So we're left in a bit of a bind. The option we're considering is moving PHP/apache onto the other server since it has spare CPU power, and that might hold off the problem for awhile. But the issue remains that the site is just too big for the servers. I wish it was because of more traffic (which would equal more ad money to help pay for it), but that's not the case.
I'll keep you posted on improvements. I do also highly suggest you follow the site's facebook page as it gets updated very quickly and often when these things happen - I don't use the site's twitter page much because I get very minimal responses/activity there, so I try to focus on the FB page instead.
So what can you do to help?
I pretty frequently receive offers for cheaper servers - seems like everyone has a buddy who can get deals, I guess that's the nature of the industry :) while it's really appreciated, AP needs consistency and reliability more than anything, including very reliable SLAs/big support teams in case there's issues. For this reason we need to stick with a big/well known company, and we're quite in the loop on what the major companies cost - right now we're in the best place we could be.
Same deal with optimizing - we're working on it, so while many of you would like to help with troubleshooting/optimizing, trust that we're already on top of that with the very nuanced system we have :)
What you can do is:
-If the server is laggy, please hold off trying to connect to it so things can clear out
-Please, please disable adblock on Anime-Planet. This is the #1 and most important thing you can do to help out the site. It's our sole way of being able to pay for stuff like this, so not enabling adblock directly hurts the site.
-And, if you're in the USA especially, make sure to click on ads that interest you to check out the products.
-There's also donations, which go straight into site costs and help out in times like these
Thanks for your enduring patience and support of AP,
~sothis
As you've surely noticed, recently there's been a lot of forum downtime and server woes. It's been a challenging and frustrating experience, as site traffic hasn't really gone up in the last year or two, so in a way this is sort of out of nowhere. The other challenge is that the $ we get from ads isn't going up, and the servers already cost a lot, so it's not easy to tack on another $x per month for an upgrade that will help (like more ram, more disks, etc), or worse, to upgrade to another server that has a bigger capacity, which jumps a huge amount in monthly price.
We've done the following today to try to help with the strain:
-Vaccuumed out the postgres database (the main site database); it was pretty messy so this probably helped a little bit
-Moved mysql/the forum to the other server, which previously just stored the activity feed (as it's quite a beast/needs a lot of resources). We think this will help because mysql is a resource hog of its own, but probably not enough to make a huge difference.
-Upgraded both servers to the latest and greatest packages/software while we were in there
-Reindexed everything in the activity feed database (mongo) and updated mongo to 2.x, which we're hoping helps with performance on that end
-Tested the RAID array to make sure it's not degrading (it's not)
-We also plan on moving the svn repository into a private github server, which will make a super minimal improvement maybe because the repository won't live on our servers, but yeah.
The server is faring better right now, but for all I know it's because the site was down for awhile. We're going to look at the load over the next few days and see if it's getting better.
But, we think the server is CPU-bound at the moment, and upgrading cores is going to be really challenging. It'll require physically changing servers again, and the next server level up jumps a huge amount in price. Because there's not consistent increases in monthly ad revenue coming in, having a higher monthly bill is unsustainable (and while donation spurts do help, they also generally aren't consistent and can't be counted on for ensuring monthly bills can be paid)
So we're left in a bit of a bind. The option we're considering is moving PHP/apache onto the other server since it has spare CPU power, and that might hold off the problem for awhile. But the issue remains that the site is just too big for the servers. I wish it was because of more traffic (which would equal more ad money to help pay for it), but that's not the case.
I'll keep you posted on improvements. I do also highly suggest you follow the site's facebook page as it gets updated very quickly and often when these things happen - I don't use the site's twitter page much because I get very minimal responses/activity there, so I try to focus on the FB page instead.
So what can you do to help?
I pretty frequently receive offers for cheaper servers - seems like everyone has a buddy who can get deals, I guess that's the nature of the industry :) while it's really appreciated, AP needs consistency and reliability more than anything, including very reliable SLAs/big support teams in case there's issues. For this reason we need to stick with a big/well known company, and we're quite in the loop on what the major companies cost - right now we're in the best place we could be.
Same deal with optimizing - we're working on it, so while many of you would like to help with troubleshooting/optimizing, trust that we're already on top of that with the very nuanced system we have :)
What you can do is:
-If the server is laggy, please hold off trying to connect to it so things can clear out
-Please, please disable adblock on Anime-Planet. This is the #1 and most important thing you can do to help out the site. It's our sole way of being able to pay for stuff like this, so not enabling adblock directly hurts the site.
-And, if you're in the USA especially, make sure to click on ads that interest you to check out the products.
-There's also donations, which go straight into site costs and help out in times like these
Thanks for your enduring patience and support of AP,
~sothis
![url]](http://i.imgur.com/YVrI6gK.png[/url])
![url]](http://i.imgur.com/JAkhHDw.png[/url])