S
Sculli
Guest
A mysql update got stuck earlier today which lead to degraded performance and ultimately about 30 minutes of QCDN downtime.
Should all be fixed now.
Should all be fixed now.
QCDN mysql update caused temporary downtime
- Joined
- Aug 9, 2005
- Messages
- 7,485
- Age
- 38
- Location
- Winnipeg, Manitoba, Canada
- Nation Banner
- Main Job
- Secondary Job
- Third Job
- Server
- Quetzalcoatl
- Linkshell
- Rise
- FFXIV
- Myrrh
Hmmm. that's why i wasn't able to get on. and here i'm am hitting my computer for no reason XD
- Joined
- Oct 4, 2005
- Messages
- 65
- Location
- (Fei'Yin)
- Nation Banner
- Main Job
- Secondary Job
- Third Job
- Server
- Quetzalcoatl
- Linkshell
- Kaizen
- FFXIV
- Tsavi
MySQL I found to be the most annoying thing to upgrade on my works server. I am shocked and amazed it was this short of downtime. One time updating my works server we got stuck down for 3 days cause of the MySQL update 
Anyway, glad to hear that it is all smooth and dandy now 
Anyway, glad to hear that it is all smooth and dandy now
S
Sculli
Guest
Well, we are not out of the woods yet.
QCDN is continuing to have intermittend "Out of Memory" errors. The hosting plan comes with 256MB guaranteed and 1024 burstable memory. The usual memory utilization is between 150 and 180MB, however, twice in the last couple of days the memory usage spiked above 256MB and eventually resulted in an out-of-memory error that would show a "There is a problem with the database." error to the end-user.
I am working with the Liquidweb support staff to figure out just what exactly caused those spikes. Once we know that we can take action to prevent that from happening again.
We initially blamed an incomplete mysql update for the out-of-memory error on Sunday, but today at 9-ish AM another out-of-memory error occured and this one didn't have anything to do with mysql updates.
QCDN is continuing to have intermittend "Out of Memory" errors. The hosting plan comes with 256MB guaranteed and 1024 burstable memory. The usual memory utilization is between 150 and 180MB, however, twice in the last couple of days the memory usage spiked above 256MB and eventually resulted in an out-of-memory error that would show a "There is a problem with the database." error to the end-user.
I am working with the Liquidweb support staff to figure out just what exactly caused those spikes. Once we know that we can take action to prevent that from happening again.
We initially blamed an incomplete mysql update for the out-of-memory error on Sunday, but today at 9-ish AM another out-of-memory error occured and this one didn't have anything to do with mysql updates.
S
Sculli
Guest
Here is an update:
Out-of-memory errors continue to occur for short periods of time (usually less than a minute) during certain times of the day (usually between 12pm and 1pm Central Time).
We are still on the look-out for the root cause of this problem. The nightly backup process has been eliminated as cause as it happens ..., drumroll ...., at NIGHT! and not during the "lunch hour".
One of the measures to attempt to fix the issue is to optimize the mysql server and enable query caching. In order to do so I will have to take QCDN down for some time to ensure that no one is accessing the database while I am frequently restarting the whole thing.
If anyone has experience with optimizing mysql 4.x and wants to look over the cnf file I will be using, here it is:
Out-of-memory errors continue to occur for short periods of time (usually less than a minute) during certain times of the day (usually between 12pm and 1pm Central Time).
We are still on the look-out for the root cause of this problem. The nightly backup process has been eliminated as cause as it happens ..., drumroll ...., at NIGHT! and not during the "lunch hour".
One of the measures to attempt to fix the issue is to optimize the mysql server and enable query caching. In order to do so I will have to take QCDN down for some time to ensure that no one is accessing the database while I am frequently restarting the whole thing.
If anyone has experience with optimizing mysql 4.x and wants to look over the cnf file I will be using, here it is:
Code:
[LEFT]datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=16M
query_cache_type=1
max_connections=900
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_cache_size=24
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=256
sort_buffer_size=512k
read_buffer_size=256k
read_rnd_buffer_size=256k
max_connect_errors=10
thread_concurrency=4
myisam_sort_buffer_size=16M
server-id=1[/LEFT]