MRatWork Forum by Mustafa Ramadhan

Sawo Project - Kloxo-MR Discussions => Kloxo-MR Technical Helps => Topic started by: hoangsang on 2013-01-11, 23:37:09

Title: Web Server response error or very slow
Post by: hoangsang on 2013-01-11, 23:37:09
Hi all.
My server running: Current Version:   6.5.0.candidate.2012-12-17-01
And:
Webserver: Nginx
Php Branch: php3u
Site: Forum VBB4.

Some time client access website is show error:
Quote
@charset "UTF-8";.toolsmenu{background:#e9e9e9 url(http://bachngocsach.com/forum/images...btn_middle.png (http://bachngocsach.com/forum/images...btn_middle.png)) repeat-x ;width:100%;display:block;border:1px solid #cecece;float:left;clear:both;margin:0.5em 0}.toolsmenu div{text-align:right;color:#4e4e4e;width:100%;clear:both;fl oat:left}.toolsmenu h1{font-size:14px;font-weight:bold;margin-top:0.25em;text-align:left;display:inline;float:left;margin-left:10px}.toolsmenu h1 a{color:#4e4e4e}.toolsmenu ul{padding:5px 0;float:right;margin-right:10px}.toolsmenu .nopopupgroup li{float:left;margin-left:10px}.toolsmenu .popupgroup a.popupctrl,.toolsmenu .popupgroup .popupmenu a.popupctrl,.toolsmenu .nopopupgroup li a{font-size:11px;color:#4e4e4e;display:block;_display:inl ine;float:left;clear:right;padding:0.2em 0.4em}.toolsmenu .popupgroup a.popupctrl,.toolsmenu .popupgroup .popupmenu a.popupctrl,.toolsmenu .popupgroup .popupmenu:hover a.popupctrl,.toolsmenu .popupgroup .popupmenu:hover .popupctrl a.popupctrl.active{border:0;color:#4e4e4e;padding-right:15px}.toolsmenu li{position:relative;display:block;clear:right}.to olsmenu .popupgroup ul{text-align:left;float:left}.toolsmenu .popupgroup
....................................................
Server: nginx
Date: Fri, 11 Jan 2013 17:02:05 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.18
Cache-Control: private
Pragma: private
Set-Cookie: bb_thread_lastview=d2ba7df341c311d3ccc2cceda69d124 51f37f3a4a-2-%7Bi-4075_i-1357922545_i-4073_i-1357908589_%7D; path=/forum/
Content-Encoding: gzip
Cache-Control: public

1ec8
‹??????ÿí½ko$Ç•6ø¹ø?¤J? ?ȶÈ*ïMvQh²¯vßÔdËÖ6{ YUIVª«*Ku!›– ´*0 ƒcv?b0¬eÁ??ë1l¯?~Q?Å| VŸô'úŸ?óœ™u!û"?btaUeÆåDĉs —ß¼zogïƒûׂæ ?
î?ܾ}k'(?—J?YÚ)•®î]
.............................................

Or Web Server response very slow (server uses about 15% of resources)

Please help me!

Thank you!
Title: Re: Web Server response error or very slow
Post by: Tsanten on 2013-01-12, 00:30:03
what plugin is active?
make a dd and ioping test to see the disk performance
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-12, 07:25:44
Quote from: "tsanten"
what plugin is active?
make a dd and ioping test to see the disk performance

No plugin is installed, and HDD is SAS 15k, and running raid 10.

But the problem is that someone users is error and other users is normal (load the website is very fast)
Title: Re: Web Server response error or very slow
Post by: Tsanten on 2013-01-12, 07:26:31
Same browser?
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-12, 07:31:42
Quote from: "tsanten"
Same browser?

Usually is IE, Chrome. But if change Webserver to Apache is normal no error :(.

I want Nginx :)
Title: Re: Web Server response error or very slow
Post by: Tsanten on 2013-01-12, 07:40:42
I see lots of 304 errors
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-12, 07:43:05
Quote from: "tsanten"
I see lots of 304 errors

can you help me?
Title: Re: Web Server response error or very slow
Post by: Tsanten on 2013-01-12, 17:09:32
Do you have some plugin ho interact with your theme?
For what i can see its an incompatibility problem of some plugin with webserver.
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-18, 13:51:24
if you are using php-fpm it needs to tune the config. like the max children, requests etc.
how is the sites traffic and servers hardware?
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-22, 10:37:00
Quote from: "Anik"
if you are using php-fpm it needs to tune the config. like the max children, requests etc.
how is the sites traffic and servers hardware?
monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Dec '12     40.58 GiB |  375.32 GiB |  415.90 GiB |    1.30 Mbit/s
       Jan '13     20.22 GiB |  245.32 GiB |  265.54 GiB |    1.20 Mbit/s
     ------------------------+-------------+-------------+---------------
     estimated     29.18 GiB |  354.03 GiB |  383.21 GiB |

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday    924.33 MiB |   10.97 GiB |   11.87 GiB |    1.15 Mbit/s
         today    342.13 MiB |    3.74 GiB |    4.07 GiB |  822.78 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       711 MiB |    7.78 GiB |    8.47 GiB |

And server hardware:
Chipset: 2 x Intel Quad Core E5504 Xeon CPU, 2.0Ghz
RAM: 4 x 2Gb DDR3 ECC
HDD: SAS 3.5", 15k rpm, RAID 10
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-22, 11:51:20
what is the config you are using?
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-22, 15:14:50
Quote from: "Anik"
what is the config you are using?
nginx.conf
Code: [Select]
user nginx;

worker_processes 8;
worker_rlimit_nofile 100000;

pid /var/run/nginx.pid;

events {
    worker_connections 2048;

    use epoll;
}


http {
    add_header Cache-Control public;

    server_names_hash_max_size 1024;
    server_names_hash_bucket_size 128;

    client_header_buffer_size 2k;

client_max_body_size 32M;

    include '/etc/nginx/conf.d/*.conf';
}


/etc/php-fpm.d/default.conf
Code: [Select]
[default]
listen = 127.0.0.1:50048
listen.backlog = -1
listen.allowed_clients = 127.0.0.1
user = apache
group = apache
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 2
pm.max_requests = 500
;pm.status_path = /status
;ping.path = /ping
;ping.response = pong
request_terminate_timeout = 0
request_slowlog_timeout = 0
slowlog = /var/log/php-fpm/slow.log
rlimit_files = 1024
rlimit_core = 0
;chroot =
;chdir = /var/www
catch_workers_output = yes
security.limit_extensions = .php .php3 .php4 .php5

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f webmaster@domain.com
;php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/error.log
php_admin_value[session.save_path] = /var/lib/php/session
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 64M


/home/nginx/conf/globals/php-fpm.conf
Code: [Select]
   ### MR -- just enough remove # below for enable nginx cache
    #fastcgi_cache fcache;
    fastcgi_cache_valid 200 1h;
    fastcgi_cache_valid 301 1h;
    fastcgi_cache_valid 302 1h;
    fastcgi_cache_valid any 1m;

    ### MR -- must be using nginx with including ngx_purge_cache
    ### must be using special nginx (nginx with ngx_purge_cache include
    location ~ /purge(/.*) {
        ### MR -- just enough remove # below for enable purge_cache
        #proxy_cache_purge pcache "$scheme://$host$1$request_method";
        allow 127.0.0.1;
        deny all;
    }

    ### MR -- for userdir (access with http://ip/~user
    location ~ ^/~([^/]+)/(.+.php)$ {
        if (!-f /home/$1/public_html/$2) {
            rewrite ^ 404;
        }
   
        alias '/home/$1/public_html/$2';

        fastcgi_pass 127.0.0.1:$fpmport;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
 
    }

    ### MR -- for userdir (access with http://ip/~user
    location ~ ^/~(.+?)(/.*)?$ {
        alias '/home/$1/public_html$2';

        autoindex on;
    }

    location ~ .php$ {
        include fastcgi_params;

        fastcgi_pass   127.0.0.1:$fpmport;
        fastcgi_index  index.php;

        fastcgi_param  REDIRECT_STATUS 200;
        fastcgi_split_path_info ^(.+.php)(/.+)$;

        fastcgi_param SCRIPT_FILENAME $rootdir$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_script_name;
        fastcgi_param PHP_ADMIN_VALUE 'sendmail_path=/usr/sbin/sendmail -t -i';
        fastcgi_param PHP_ADMIN_VALUE 'sendmail_from=$domain';
    }

    location / {
        limit_conn addr 25;

        ## MR - increasing upload size to 15M
        client_max_body_size 15M;
        client_body_buffer_size 128k;

        try_files $uri $uri/ /index.php;
    }

Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-22, 16:08:58
depending on your server configuration, you should consider increasing the values. specially the php-fpm one.
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-22, 22:46:57
Quote from: "Anik"
depending on your server configuration, you should consider increasing the values. specially the php-fpm one.

You can specific value examples?
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-22, 23:04:08
Quote from: "hoangsang"
Quote from: "Anik"
depending on your server configuration, you should consider increasing the values. specially the php-fpm one.

You can specific value examples?

that will depend on circumstances, but you can first increase the maxchildren to something 12/15, 5 is really not good for you.
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-22, 23:43:57
Quote from: "Anik"
Quote from: "hoangsang"
Quote from: "Anik"
depending on your server configuration, you should consider increasing the values. specially the php-fpm one.

You can specific value examples?

that will depend on circumstances, but you can first increase the maxchildren to something 12/15, 5 is really not good for you.

I'm changed to:
Code: [Select]
pm.max_children = 70
pm.start_servers = 50
pm.min_spare_servers = 40
pm.max_spare_servers = 55
pm.max_requests = 1000

But access by Chrome is very slow :(
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-22, 23:55:33
70 is a too large value. just set something between 12 to 15 depending on traffic.
what the log files is saying? specially the slowlog?
any errors, timeouts, or memory overload?
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-23, 00:01:19
Calculate by memory usage too. like if one php-fpm child process is taking 150 MB ram then 70 process will take 10,500 MB which will end up giving you 500 error and probably locking down the server. so choose wisely by using your brain. Good Luck.
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-23, 00:07:48
Quote from: "Anik"
Calculate by memory usage too. like if one php-fpm child process is taking 150 MB ram then 70 process will take 10,500 MB which will end up giving you 500 error and probably locking down the server. so choose wisely by using your brain. Good Luck.

Thanks. This my log
Code: [Select]
[22-Jan-2013 17:07:21] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 17:08:22] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 19:31:35] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 19:46:54] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 19:51:59] ERROR: unable to read what child say: Bad file descriptor (9)
[22-Jan-2013 20:46:03] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 21:24:33] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 21:47:16] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 21:59:59] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 22:07:46] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 22:07:53] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 22:29:44] ERROR: unable to read what child say: Bad file descriptor (9)
[22-Jan-2013 22:57:50] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[22-Jan-2013 23:25:17] WARNING: [pool bachngocsach] server reached pm.max_children setting (5), consider raising it
[23-Jan-2013 00:56:58] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:56:58] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:08] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:41] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 00:59:41] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 01:04:19] ERROR: unable to read what child say: Bad file descriptor (9)
[23-Jan-2013 01:04:19] ERROR: unable to read what child say: Bad file descriptor (9)


ERROR: unable to read what child say: Bad file descriptor (9)
hichic :(
Title: Re: Web Server response error or very slow
Post by: hoangsang on 2013-01-23, 00:49:10
Quote from: "Anik"
Calculate by memory usage too. like if one php-fpm child process is taking 150 MB ram then 70 process will take 10,500 MB which will end up giving you 500 error and probably locking down the server. so choose wisely by using your brain. Good Luck.

I'm changed to:
Code: [Select]
pm.max_children = 15
pm.start_servers = 14
pm.min_spare_servers = 13
pm.max_spare_servers = 15
pm.max_requests = 1000

And view log:
[23-Jan-2013 01:45:30] WARNING: [pool bachngocsach] server reached pm.max_children setting (15), consider raising it
Title: Re: Web Server response error or very slow
Post by: Tsanten on 2013-01-23, 00:54:06
You need to configure php mysql nginx for your settings this mean you have to know how much ram  and how many cpu do you have
Normally i configure mysql to take at maximum 50% (mysqltuner.pl)of the memory and let other for caching and web server.
Start from this and observe were you have to modify settings.
Title: Re: Web Server response error or very slow
Post by: Anik on 2013-01-23, 00:54:57
As i told, you should have to raise the max children. which log file it is? /var/log/php-fpm/error.log ?
looks like segfault or open files overload.
increase the connection and children.
Title: Re: Web Server response error or very slow
Post by: Spacedust on 2013-02-21, 01:39:03
Quote from: "hoangsang"
Quote from: "Anik"
Calculate by memory usage too. like if one php-fpm child process is taking 150 MB ram then 70 process will take 10,500 MB which will end up giving you 500 error and probably locking down the server. so choose wisely by using your brain. Good Luck.

I'm changed to:
Code: [Select]
pm.max_children = 15
pm.start_servers = 14
pm.min_spare_servers = 13
pm.max_spare_servers = 15
pm.max_requests = 1000

And view log:
[23-Jan-2013 01:45:30] WARNING: [pool bachngocsach] server reached pm.max_children setting (15), consider raising it

Set this to:

Code: [Select]
pm.max_children = 30
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 6
pm.max_requests = 1000