QCDN APR 7 Downtime explained - QCDN likely to remain slow a bit longer

S

Sculli

Guest
Alright, so I did submit a support request to Liquidweb based on QCDN sluggishness as reported in this thread QCDN slow? on 2006-04-07 02:13:59 (this is Eastern time). On 2006-04-07 09:36:28 Liquidweb support responded with:

Paul Morley said:
I have restarted your server and another on this server as they were both
using alot of resources and causing the load on the server to get pretty high.
This is what was causing the slowness you were seeing.

---------
Paul Morley
System Administrator
Liquidweb Inc.
Click to expand...

The problem this caused is that named, the DNS server, did experience some problem during the restart. It's not quite clear when exactly it stopped working completely, but it is save to assume that anything after 2006-04-07 09:36:28 (Eastern) was running with major difficulties.

Liquidweb has a warning mechanism in place that checks for non-functioning DNS servers and alerts the support staff that there is a problem. However, that system is essentially disabled by a server restart as a restart doesn't count as an alarm. It is unclear why the system didn't catch the downed DNS server for hours after the restart though.

I have worked with LW to create a cron job that runs on the QCDN server will check the DNS server every minute and restart it should it be down for some reason.

That's all I have for now, will post to the iPod thread and start it up in a few minutes.
 
Current QCDN sluggishness is likely caused by the server using disk swap space when the memory usage exceeds the available physical memory.

QCDN performance will remain on the sluggish side for some more time as Liquidweb and I work on removing the root cause of QCDN using disk swap space when we run out of physical memory.
 
well, it is running much faster right now atleast. so whatever you've done, you've done something correctly ^^. Unlike SE -.- DAMN POL - 1356
 
I find it ironic that this happened on the day you mentioned in your newsletter about us "closing down"
 
lol internet
 
Just to mention this one more time; I am aware of QCDN being slower than it has been in the past few months. The reason for that is that QCDN is continiously exceeding the 256MB guaranteed memory allocation. The account is burstable to 1024MB memory, but that hingest on no one else using that physical memory.

In the past bursting above 256MB physical is what happened. Now the physical server QCDN is on doesn't always have physical memory available to burst above 256MB and thus disk swap space is being used which results in slower pages loads, and sometimes in "page cannot be displayed" or "cannot find server" messages.

I am aware of this issue, Liquidweb staff is aware of this issue, we are trying to optimize things to lower memory usage, primarily mysql memory loads. Please bear with us as we are trying to fix this without to move up to a hosting plan that allocates more resources to QCDN.

I will keep you guys updated.
 
Site always appears slow to me, vBulletin uses stupid amounts of JavaScript. Apart from that, there doesn't seem to be any problem now...
 
To a degree it's not a matter of funding at this point. I can move to a VPS that provides 512MB of guaranteed physical memory for $40 more per month, and I will do that when necessary, it's currently a matter of figuring out whether it is necessary to more of not.

Basically we are working on finding out whether the mysql processes that are running on the server are all supposed to be there, and whether they would be fine if they were set to use less memory than they are currently using.

This will require some tinkering and trial and error over the next week or so. If optimizations are not going to lower the memory usage below 256MB for most of the time, then I will move QCDN up to the next best VPS which guarantees 512MB of physical memory.

The downtime required to move should be minimal as it's all happening within the same company.
 
As of APR 9 02:22AM PDT the server QCDN is on is experiencing technical difficulties. Liquidweb staff has been made aware of it and is working on resolving the issue.
 
LW staff rebooted the physical server QCDN is located on earlier today because pretty much all customers who are located on it had put in support calls because of sluggishness.

During that restart the QCDN DNS server yet again failed to start on it's own, and for whatever reason yet again LWs alert system didn't catch the failure to restart.

Also, because of the sudden reboot of the physical machine some of the mysql tables we are using were corrupted. I did repair those tables and it seems to be working alright now.

QCDN will likely remain slow as we are trying to figure out which account the problem child is. QCDN has been with LW since DEC 03 2005 and it worked flawlessly till MAR 27 2006. It is possible that QCDN's DB size has grown so large that it is causing problems. Unfortunatley it is difficult to pinpoint the issue since I as customer do not have access to how much memory exactly QCDN is using over time, and LW does have some sort of record but it is inconclusive.

For the time being the deal is that once things slow down again, I will call LW support so that they can look at it the very moment it happens and go from there.
 
Saw this last night:

"vBulletin Message

APR 9 - 01:56 PDT Bringing QCDN down for another attempt at mySQL optimization. Chances are this will take a while, so don't wait up, go get some sleep. ;) -Sculli"

Aww. Sculli <3's us.
 
There has been some new development and I will likely update the PHP version QCDN is using in the coming days as well as uninstall Zend and install ACP instead (for those to whom this makes any sense).

MySQL memory usage remains an issue as it persistently takes up 700-ish MB of RAM. That is fine as long as no one else on the server is using all of their memory (which is often the case), but as soon as other VPS clients use their memory, QCDN is pushed into swap space and things slow down.

Regardless of the PHP optimization I will probably have to move QCDN to a hosting plan that guarantees 1024MB of RAM (rather than the current 256MB) if the MySQL optimizations do not yield the desired results.
 
Downtime rescheduled for Saturday, 04/15, 12AM PDT.
I am still working right now, had to take a bunch of work home to get it done for tomorrow morning, no way I can fit in QCDN downtime tonight.
 
Alright, after I didn't get around to actually doing the maintenance last week, I really will do it tonight. See: DOWNTIME SAT 04/21 12AM

I will try and make it as short as possible considering that people will be online all night playing ToAU. I won't mess with the PHP version, I will do that later sometime. I will just update the MySQL and Apache configs. This will still require about an hour of downtime as I need to make sure I have a up-to-the-minute backup of the database before I fiddle with any configs.
 
You must log in or register to reply here.
Back
Top