CPU load: 259.33/166.35/119.93

Portal Home > Knowledgebase > Articles Database > CPU load: 259.33/166.35/119.93

Posted by SlAiD, 07-14-2009, 04:19 AM
You think this is funny? Not at all. I'm on LON03 server and I recive yesterday from Future Hosting this advise about hight load. I got then 1 month ago, and the solution was *JUST* to remove a phpBB website with 80+ users. I do not believe that the VPS could get this kind of loads. Funny is that the VPS was "monitored" by them at the same time. They say the load spikes out for times to times in a matter of seconds. This load is taken from the Parallels Infrastructure Panel or even the Node Panel, not from the VSP itself. Even when I'm logged in. I've been a client for 6 month, and 2 of the last 3 constantly getting CPU hight usage. Strange is, that I come from a 512mb/25gb PowerVPS server to a 1GB/30GB server, with the *SAME* sites. Anyway, the prupose of this topic is to: - check if anyone had this problem (problem is: beeing sure that this load is IMPOSSIBLE to be created from yout service) - recommend a decent VPS provider besides Future Hosting and PowerVPS. Thanks in advance, SL (the 259.33/166.35/119.93 killer man... or not)
Posted by kaniini, 07-14-2009, 04:25 AM
those loads look like an i/o contention issue. have your provider use a tool like iotop to determine what vps is being abusive.
Posted by SlAiD, 07-14-2009, 04:41 AM
They told me it's mine. But I will get in touch to then asking that. SL
Posted by SlAiD, 07-14-2009, 05:27 AM
They've a reply for me: We were not able to record the logs due to very high load but it seems to be regarding CPU and from your VPS. So, basicly, this does not help anything. SL
Posted by kaniini, 07-14-2009, 05:55 AM
those loads are impossible based on CPU usage. the maximum number of processes in a waitqueue based on CPU utilization would be equivilant to the number of CPU cores your VPS can utilize. those loads *are* i/o contention, be it disk or swap, it does not matter.
Posted by futurehosting, 07-14-2009, 06:17 AM
I'll post what your ticket was updated with: It is IO related and you aren't on lon03. You are on an isolated node where your VPS caused this. Your VPS is with one other VPS on a server with 6 x 300GB 15K drives in raid 10. 28GB RAM and dual/quad core processors. [root@lon02 ~]# free -m total used free shared buffers cached Mem: 28084 25890 2194 0 502 21903 -/+ buffers/cache: 3484 24600 Swap: 16378 0 16377 When your VPS is on any node it causes issues for everybody on the node. Even on a node with two VEs, it started causing issues as you saw from yesterday. Yes, our techs were specifically monitoring your VPS; however, they can't stare at the VPS processes every minute for 24 hours a day. We have numerous tools to monitor nodes and VPSs and your VPS seems to go from low CPU usage, low loads and no IO issues to causing these types of loads and high IO wait in a matter of seconds. Feel free to update the ticket if you need assistance.
Posted by kaniini, 07-14-2009, 06:22 AM
25890MB used on an "isolated" node? something does not smell right about your post... but what would i know.
Posted by futurehosting, 07-14-2009, 06:24 AM
I don't appreciate the implication that we aren't being truthful. We use the same general configuration for all of our nodes, including spare nodes for hardware failures. In this case, that node is one of our spare nodes which we keep on hand for hardware failures. We don't keep lower quality nodes to isolate VEs, it's a waste of our resources.
Posted by kaniini, 07-14-2009, 06:30 AM
thanks for clarifying, but 25GB of RAM *used* does not indicate isolation. it just means you put him on a different node than lon03. i doubt seriously that one or two VPS are using 25GB of RAM.
Posted by futurehosting, 07-14-2009, 06:32 AM
As a sign of good faith here is the output from the license which indicates the node is licensed for 20 VEs and 2 are used. [root@lon02 ~]# vzlicview|grep ct ct_total=20 (2) architecture="x86" architecture="x86_64" product="Virtuozzo" [root@lon02 ~]# free -m total used free shared buffers cached Mem: 28084 25860 2224 0 506 21918 -/+ buffers/cache: 3434 24650 Swap: 16378 0 16377 [root@lon02 ~]#
Posted by kaniini, 07-14-2009, 06:32 AM
oh, nevermind, i'm an idiot. 24GB of that is going to the filesystem cache. well, good luck in debugging the issue, but it definitely looks like i/o contention. iotop will be useful.
Posted by bikster, 07-14-2009, 06:33 AM
These loads are nearly impossible for CPU usage. Such loads on your CPU most probably will cause the server to crash. Anyway, have you tried SSHing and checking whether there are any proccesses using abnormally high memory by typing "top"?
Posted by Layershift Damien, 07-14-2009, 10:58 AM
My first call would be MySQL slow query log (if it's not enabled - why not!). Also check for high number of temporary tables created on disk, full table scans, sorts requiring temp. tables and so forth. It's really difficult to say what legitimate processes might cause such high I/O without more information about the sites etc., but MySQL seems the most likely assuming nothing else "interesting" is around on this. Given an unmanaged service (which I presume this is), it's not exactly surprising that your provider haven't already spotted this, but it really could be something as simple as a few badly constructed MySQL queries and/or some (lack of) appropriate indexing. If not, but still caused by MySQL, you could also take a look at throwing memcached on there perhaps (together with appropriate code changes of course!). If the hardware specs/details provided by Future are accurate (and there's no reason to assume any different), there's no way you can host these sites on any VPS unless you can find and fix the problem.
Posted by bigks, 07-14-2009, 01:43 PM
This is the best response you can get and I would agree 100% to being forced to find and fix the problem before jumping host. Most host would have just killed the VPS and dumped you data in your lap so you must give futurehost props for trying to work with you.
Posted by SlAiD, 07-14-2009, 05:46 PM
Hi there. Let's clarify this: I'm not blaiming FutureHosting. It's a bad luck they didn't catch the issue. In fact, FutureHosting is spending more time analysing this than the one I'm paying. I'm 100% sure of that. And @Layershift Damien, the service is managed. Why did you think it was unmanaged? @futurehosting I think that the "implication that we aren't being truthfu" was not for me. Anyway, I got some idea and things to run, and I will do it this week to see the results. Btw, some more technical question for those who're Virtuozzo owners:: I've a (I think) not related issue, regarding bandwith not beeing update on PBA, just VZPP. Now this I/O spikes. Isn't your VZ licences cover with support and if that is the case, will or not, or does it have any cousts for you, to ask fir Paralels to take a look at this, since if this is a i/o issue from the OS, is a virtualization problem? Once again, I'm just asking. Thanks, SL
Posted by Layershift Damien, 07-15-2009, 11:53 AM
Was my impression based on the information in this thread. For a managed VPS I would've expected more information (from your posts or from Future Hosting's) about what was investigated and found/ruled out thus far. I guess this was just omitted, but it makes it somewhat difficult to offer additional suggestions (beyond already offered) so that you can get this thing solved. Support arrangements between providers and Parallels vary depending on individual contracts, so there's no proper way to generalise this. However, if there's a bug in Parallels' software then of course the provider should report it to them - I have no doubt that Future Hosting would have done that, but reporting the bug isn't the same as having it fixed and that can take quite some time for complex and/or low priority problems. As far as disk I/O is concerned that's not really an issue for Parallels to get involved with since it almost certainly isn't caused by their software, and of course it also isn't running on their hardware. The I/O issue is most probably coming from MySQL, but certainly if not by migrating your VPS around different nodes Future Hosting have effectively ruled out an issue with specific hardware / VZ kernels etc. - the fact the issues occur exactly on/because of your VPS clearly suggests something internal to that is the cause.

Add to Favourites Print this Article