MRatWork Forum by Mustafa Ramadhan

Sawo Project - Kloxo-MR Discussions => Kloxo-MR Technical Helps => Topic started by: cvrle77 on 2014-05-15, 18:28:48

Title: High CPU usage
Post by: cvrle77 on 2014-05-15, 18:28:48
After my last venture  from here http://forum.mratwork.com/kloxo-mr-technical-helps/php-broke/ , I've noticed load increase from max 0.5 to 1.5+ which got me suspended from my VPS provider today, and don't know what caused this, since I haven't changed anything in my website setup.

I've also noticed that when I run 'htop' command, I have more than 4 mysqld processes loaded (I had only 4 before)

Any recommendation for 1GB VPS+512MB SWAP on CentOS 6, hosting 2 wordpress websites?

/script/sysinfo

A. Kloxo-MR: 6.5.0.f-2014042802

B. OS: CentOS release 6.5 (Final) x86_64

C. Apps:
   1. MySQL: mysql55-5.5.37-1.ius.el6.x86_64
   2. PHP: php53u-5.3.28-3.ius.el6.x86_64
   3. Httpd: httpd-2.2.27-1.el6.x86_64
   4. Lighttpd: --uninstalled--
   5. Nginx: nginx-1.7.0-1.el6.x86_64
   6. Qmail: qmail-toaster-1.03-1.3.38.mr.el6.x86_64
      - with: courier-imap-toaster-4.1.2-1.3.16.mr.el6.x86_64
   7. Dns: djbdns-1.05-17.3.mr.el6.x86_64

D. Php-type (for Httpd/proxy): mod_php_ruid2

E. Memory:
                total       used       free     shared    buffers     cached
   Mem:           996        884        111          0         32        296
   -/+ buffers/cache:        554        441
   Swap:          511          7        504

A. Kloxo-MR: 6.5.0.f-2014020301

B. OS: CentOS release 6.5 (Final) x86_64

C. Apps:
   1. MySQL: mysql-5.5.34-1.el6.x86_64
   2. PHP: php-5.3.28-4.el6.x86_64
   3. Httpd: httpd-2.2.27-1.el6.x86_64
   4. Lighttpd: --uninstalled--
   5. Nginx: nginx-1.7.0-1.el6.x86_64
   6. Qmail: qmail-toaster-1.03-1.3.38.mr.el6.x86_64
      - with: courier-imap-toaster-4.1.2-1.3.16.mr.el6.x86_64
   7. Dns: djbdns-1.05-17.3.mr.el6.x86_64

D. Php-type (for Httpd/proxy): mod_php_ruid2

E. Memory:
                total       used       free     shared    buffers     cached
   Mem:           996        856        139          0         86        504
   -/+ buffers/cache:        265        730
   Swap:          511          0        511
Title: Re: High CPU usage
Post by: I<3Kloxo on 2014-05-15, 20:06:14
How much your VPS has core and their speed ?
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-15, 20:20:40
CPU model :    QEMU Virtual CPU version (cpu64-rhel6)
Number of cores : 4
CPU frequency : 2099.998 MHz
Total amount of ram : 996 MB
Total amount of swap : 511 MB
System uptime :   2:16,
Download speed : (75.6MB/s)
I/O speed : 396MB/s
Title: Re: High CPU usage
Post by: I<3Kloxo on 2014-05-16, 00:13:30
Hi,

I had a similar problem and I googled the internet 7 days to find solution. In the end I made it that I installed a new CentOS and new Kloxo-MR because I have not had more ideas how to fix a hight load. When I did it I saw it again, I have a big load and I got to know that is not up to me but it is up to the hosting provider (Overseeling Node).

This is my case, does not need to mean that the same with your VPS. If you can make freshly installation of Kloxo-MR it would be good because in this way you can find out that (find the problem).
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 11:38:07
Still in trouble...all I can conclude from htop is that httpd is making this load even though nginxproxy is in place
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 11:46:45
Investigate your '/etc/my.cnf'. Maybe have parameter like 'max_connections=4'.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 11:57:30
I am getting errors in wordpress after putting this, and load didn't dropped a bit.

Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:31:14
I mean if your my.cnf have 'max_connections' so disable it with '#max_connections'
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:31:53
It was disabled by default
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:36:16
Ok, inform your 'cat /etc/my.cnf'.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:38:03
cat /etc/my.cnf
##  _______________________________________________________________________
## / Rackspace MySQL 5.5 Terse Configuration File                          \
## |                                                                       |
## | This is a base configuration file containing the most frequently used |
## | settings with reasonably defined default values for configuring and   |
## | tuning MySQL. Note that these settings can likely be further tuned in |
## | order to get optimum performance from MySQL based upon the database   |
## | configuration and hardware platform.                                  |
## |                                                                       |
## | While the settings provided are likely sufficient for most            |
## | situations, an exhaustive list of settings (with descriptions) can be |
## | found at:                                                             |
## | http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html   |
## |                                                                       |
## | Take care to only add/remove/change a setting if you are comfortable  |
## | doing so! For Rackspace customers, if you have any questions or       |
## | concerns, please contact the MySQL Database Services Team. Be aware   |
## | that some work performed by this team can involve additional billable |
## \ fees.                                                                 /
##  -----------------------------------------------------------------------
##         \   ^__^
##          \  (oo)\_______
##             (__)\       )\/\
##                 ||----w |
##                 ||     ||

[mysqld]
skip-innodb
default-storage-engine=myisam
#OpenVZ AIO bug fix
innodb_use_native_aio=0

## General
datadir                         = /var/lib/mysql
tmpdir                          = /var/lib/mysqltmp
socket                          = /var/lib/mysql/mysql.sock
skip-name-resolve
sql-mode                        = NO_ENGINE_SUBSTITUTION
#event-scheduler                = 1

## Cache
thread-cache-size               = 16
table-open-cache                = 4096
table-definition-cache          = 2048
query-cache-size                = 64M
query-cache-limit               = 1M

## Per-thread Buffers
sort-buffer-size                = 1M
read-buffer-size                = 1M
read-rnd-buffer-size            = 1M
join-buffer-size                = 1M

## Temp Tables
tmp-table-size                  = 32M
max-heap-table-size             = 64M

## Networking
back-log                        = 100
#max-connections                = 4
max-connect-errors              = 10000
max-allowed-packet              = 16M
interactive-timeout             = 3600
wait-timeout                    = 600

### Storage Engines

## MyISAM
key-buffer-size                 = 64M
myisam-sort-buffer-size         = 128M

## InnoDB
#innodb-buffer-pool-size        = 128M
#innodb-log-file-size           = 100M
#innodb-log-buffer-size         = 8M
#innodb-file-per-table          = 1
#innodb-open-files              = 300

## Replication
server-id                       = 1
#log-bin                        = /var/log/mysql/bin-log
#relay-log                      = /var/log/mysql/relay-log
relay-log-space-limit           = 16G
expire-logs-days                = 7
#read-only                      = 1
#sync-binlog                    = 1
#log-slave-updates              = 1
#binlog-format                  = STATEMENT
#auto-increment-offset          = 1
#auto-increment-increment       = 2

## Logging
#log-output                      = FILE
#slow-query-log                  = 1
#slow-query-log-file             = /var/log/mysql/slow-log
#log-slow-slave-statements
#long-query-time                 = 2

[mysqld_safe]
log-error                       = /var/log/mysqld.log
open-files-limit                = 65535

[mysql]
no-auto-rehash
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:42:22
AS far as I can see, majority of load % is coming from process /usr/sbin/httpd

(http://i.imgur.com/Vip0cFJ.png)
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:43:59
Inform here 'ps aux|grep mysql' also.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:44:44
ps aux|grep mysql
root     21960  0.0  0.1 108300  1584 pts/1    S    06:38   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql    22470  1.2  3.8 535908 39672 pts/1    Sl   06:38   0:06 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root     22622  0.0  0.0 103244   840 pts/1    R+   06:47   0:00 grep mysql
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:45:15
Mod_php_ruid2 not designed for high traffic. Better php-fpm_event.
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:46:13
Your 'ps aux|grep mysql' is a normal service loaded.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:48:06
So, all I need to switch from mod to php_fpm?

(http://i.imgur.com/q3riLM2.png)
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 12:51:28
My recommendation is using php-fpm_event (for apache or -proxy). Lighttpd/Nginx/Hiawatha automatically using php-fpm.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 12:59:27
Well, I am using nginxproxy since day one

(http://i.imgur.com/JQYPw1o.png)

So, if proxies are using php-fpm_event automatically, that means I was already using it.

I don't really think my sites are high traffic ones with 30-60 users online.

And even after switching, load is still unchanged. Also, this is SSD VPS, with IO of 300-600MB/s, so that isn't an issue here...

When I mentioned mysql, all I said is that in htop, before changes, I had 4 mysqld processes visible, and after change there were 8+, I never said mysql is an issue.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 13:16:06
Now after change, I have Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

error 500
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 13:18:54
I think your issue related to http://forum.mratwork.com/kloxo-mr-releases-and-announcements/(update)-please-update-to-6-5-0-f6-5-1-b-to-201405102201405104/

You need update Kloxo-MR first.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 13:54:42
Did everything, but I see 0 improvement. Load still jumping on 1, and main processes php-fpm: pool admin are eating CPU. I have to stop httpd every time it reaches load of 1, so I don't get banned from hosting.
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 14:07:06
There something in your website application using cpu too high.

I am not sure, maybe you can limiting with 'ulimit'.

Inform here 'ulimit -a'.
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 14:29:12
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7823
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7823
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-16, 14:34:39
Read http://www.itworld.com/operating-systems/317369/setting-limits-ulimit and http://www.askapache.com/optimize/optimize-nice-ionice.html
Title: Re: High CPU usage
Post by: cvrle77 on 2014-05-16, 15:06:12
As I said in my first post

Quote
After my last venture  from here http://forum.mratwork.com/kloxo-mr-technical-helps/php-broke/ , I've noticed load increase from max 0.5 to 1.5+ which got me suspended from my VPS provider today, and don't know what caused this, since I haven't changed anything in my website setup.

I never touched anything on my VPS regarding websites (i've even disabled lots of stuff to see if plugins or theme is causing those issues), I had smooth config with kloxo mr, before I've updated everything, after that bad venture with trying to install APC.

At around 18th april there was some holiday, and I had over 70K PV and load of 0.3, I had more than 200 people  online at the same time, now when it comes close to 50 people online, load goes to 1, few people more and it goes up over 1. I have WP super cache installed, Better Wordpress Minify, static content is served all the time and I can see that in source by the date cache file is created.

I know this can work smooth, with waaaay lesser load on server, because I've had it, and lost that setup after updates...
Title: Re: High CPU usage
Post by: merbot on 2014-05-16, 17:15:34
also
check this with mysqltuner
Title: Re: High CPU usage
Post by: I<3Kloxo on 2014-05-16, 23:48:19
As I understand for web sites that have hight traffic it is better "php-fpm_event" ? Is that correct?

Edit:
I changed to php-fpm_event and I do not see any change regarding CPU.
Title: Re: High CPU usage
Post by: I<3Kloxo on 2014-05-19, 03:59:07
bump !
Title: Re: High CPU usage
Post by: MRatWork on 2014-05-19, 04:03:44
Try 'sh /script/mysql-optimize --select=repair'. Maybe something wrong your database.