|07-13-2012, 10:09 PM||#1|
Join Date: May 2002
Location: Seattle, Washington
Recent feed issues, upcoming scheduled maintenance/downtime, and how you can help
Today, there was an issue where the feed showed an error on the profiles. Just wanted to give you a heads up about what happened, and tell you about some upcoming downtime tomorrow to maintain the servers.
Basically, the hard drive the feed's database resides on had some issues with bad data (as it's a year old drive, is heavily used, and has started to fail), and the drive needs to be replaced. It took awhile, but we were able to fix the bad sectors and stabilize the drive. But, it does need to be replaced. This issue ONLY AFFECTED THE SITE FEED, which is stored on a separate type of database on a separate server. Your list data, all site content (reviews, screenshots, everything) are on the main server, and are not affected by this.
Meanwhile, there's some things we need to do on the main server as well - one of the drives of the RAID array needs to be replaced, and a few other things tinkered with to keep them going. With such a heavily trafficed server, that comes with the territory.
We've decided we'll do these all at once, since each thing would require the server to be down. We're planning it for tomorrow at some point, and I'll post updates warning people ahead of time. Updating the feed server may take extra time to get up and running after the hard drive switch, but we'll update the site code to not output errors as a temporary measure, once the server is back up, if needed.
HUGE thanks to holms and dmitrijus who assisted for hours today with troubleshooting what could be wrong with the server. They deserve a lot of kudos.
Finally, while I really, really dislike bringing up donations, several people in irc today asked about it, so I'll mention it. If you feel like supporting the site, here's the link to donate
Right now times are pretty tough - the site keeps growing, traffic-wise, which means more resources (bigger and better servers, more support, etc) are needed, but the revenue coming in from ads is not growing, and it's low to begin with. The site is only pulling in a little more money than the servers and other related bills cost (I don't pay myself a dime, never have, all money goes straight into the site).
Meanwhile, because of this crash, I just had to pay 10gen $2,500 USD for the next year, for support on the activity feed database - it's NoSQL, which is still a newish technology and there aren't really any reputable companies out there (or highly skilled server admins with that specialty). There wasn't really another choice for the feed, as all site activity is logged and to a huge degree (it's a 20 gig size for the last 8 months). The main developer I had for years is gone now, and I'm not an expert on the subject. We need ongoing support to update safely, optimize so those cursortimeout errors stop, etc. So $200/month sucks, but is pretty necessary to keep the site going.
And because of that extra $200/month, we're now basically breaking even. That's really scary, given there's no money to do ANYTHING - nothing for emergencies that pop up, I can't hire anyone to help with some of the really complicated aspects of the site, and I can't move forward with some of the bigger projects I've wanted to do (like getting a professional redesign for cetain pages, etc). Unlike some of our competitors, we aren't corporate owned and don't have huge sponsors that can pay for multiple salaries - it's just me, and i work for free.
So, again, if you love Anime-Planet and want to see it able to continue, please consider donating
**note: I realize I still haven't inputted badges for a few months for donations - I apologize, it's due to a lack of time. ;_; with no developers now except me, I've found it increasingly difficult to get everything done, by myself, that's needed to support a site of 400,000 people a month. Donations help, among other things, to ensure i can actually get more people onboard to help out.
Thanks everyone for your patience during this time, and I'll update more once it gets closer to the scheduled downtime. And thanks for using A-P