MRatWork Forum by Mustafa Ramadhan
Sawo Project - Kloxo-MR Discussions => Kloxo-MR Bugs and Requests => Topic started by: Spacedust on 2016-12-30, 21:53:51
-
It's the 5th time already today after last update.
All pages stops responding then comes 504 error and I had to restart all web server software.
sh /script/apache-optimize --high doesn't help.
I'm running on a dedicated server with 8-core Atom and 8 GB RAM.
-
Did you run 'sh /script/cleanup'?.
-
Corrupt dababase may have the same issue. Try 'sh /script/mysql-optimize --select=repair'.
-
No errors in MySQL database. Today I got 502+504 Errors. /script/cleanup also doesn't help.
-
Ok, give me ssh and Kloxo-MR panel access. I want to know what's happens.
-
Ok, give me ssh and Kloxo-MR panel access. I want to know what's happens.
See PM. Second time today :(
-
Didn't received in PM.
-
Please check again now ;)
-
Read my message.
-
Just replied. That is solved now, but https in Kloxo-MR is still not working properly with:
"Unable to establish a secure connection" errors e.g. when manually generating stats.
-
It happened again today on 6:37AM - 502 error on all websites, so still not solved.
Before it happens all pages loads indefinitely and then I had to restart php-fpm.
-
I'm testing now that setting:
emergency_restart_threshold 3
emergency_restart_interval 1m
process_control_timeout 5s
-
Now also testing this:
pm.max_children = 70
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500
-
Still crashing with 503 errors ! Please help.
-
This seems to be an issue:
2017/01/03 09:38:01 [error] 32590#32590: *65230 limiting connections by zone "conn_limit_per_ip", client: 62.210.81.x, server: xxx.pl, request: "HEAD /test.php HTTP/2.0", host: "xxx.pl"
-
This seems to be an issue:
2017/01/03 09:38:01 [error] 32590#32590: *65230 limiting connections by zone "conn_limit_per_ip", client: 62.210.81.90, server: xxx.pl, request: "HEAD /test.php HTTP/2.0", host: "xxx.pl"
This is to handle DDOS attack
-
This seems to be an issue:
2017/01/03 09:38:01 [error] 32590#32590: *65230 limiting connections by zone "conn_limit_per_ip", client: 62.210.81.x, server: xxx.pl, request: "HEAD /test.php HTTP/2.0", host: "xxx.pl"
This is to handle DDOS attack
But blocks normal connections too...
-
What's you mean 'normal connection'. They blocked external IP.
-
What's you mean 'normal connection'. They blocked external IP.
That's actually my server IP :( I use many cron jobs.
That helped a little bit, now also disabled pagespeed which is also causing stability issues.
-
What's this ?
[FastCGI: neither POLLIN nor POLLOUT...]
-
I've downgraded from PHP 7.1.0 to 7.0.14 for testing purposes. I think it was way more stable.
-
I didn't helped. I'm testing new php config now.
pm.max_children = 64
pm.start_servers = 16
pm.min_spare_servers = 16
pm.max_spare_servers = 64
pm.max_requests = 20000
-
Try using 'pm=dynamic' instead "pm=ondemand"
-
Try using 'pm=dynamic' instead "pm=ondemand"
Changed that as well.
I've also disabled this - this simply doesn't work well for high-traffic websites:
# limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
# limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=10r/s;
-
Still crashing !
[07-Jan-2017 04:11:07] ERROR: FPM initialization failed
[07-Jan-2017 04:11:07] ERROR: FPM initialization failed
[07-Jan-2017 04:11:07] ERROR: FPM initialization failed
[07-Jan-2017 11:31:21] ERROR: FPM initialization failed
[07-Jan-2017 11:31:30] ERROR: FPM initialization failed
[07-Jan-2017 12:10:03] ERROR: FPM initialization failed
[07-Jan-2017 12:10:03] ERROR: FPM initialization failed
[07-Jan-2017 12:10:03] ERROR: FPM initialization failed
[07-Jan-2017 12:10:03] ERROR: FPM initialization failed
[07-Jan-2017 13:10:03] ERROR: FPM initialization failed
[07-Jan-2017 13:10:03] ERROR: FPM initialization failed
[07-Jan-2017 13:10:04] ERROR: FPM initialization failed
[07-Jan-2017 13:10:04] ERROR: FPM initialization failed
-
Try 'pkill -9 php-fpm; sh /script/restart-web'.
-
Try 'pkill -9 php-fpm; sh /script/restart-web'.
It helps for just 20 minutes, then again pages are loading indefinitely until I restart php-fpm.
-
I've switched from php71w to php71u. Maybe we can add remi's php71 ?
-
Now testing:
pm.max_children = 2
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 2
pm.max_requests = 20000
-
Now testing:
pm.max_children = 2
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 2
pm.max_requests = 20000
Are you serious with 'pm.max_children=2'?.
-
Now testing:
pm.max_children = 2
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 2
pm.max_requests = 20000
Are you serious with 'pm.max_children=2'?.
Sorry there is 6 out there - still freezing with no errors under minimal load ;(
-
Now testing this:
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 500
-
Nothing helps - only php is freezing. Normal html pages works perfectly. How to debug this ?
-
Find out log_level and change value to 'debug'.
-
Find out log_level and change value to 'debug'.
[07-Jan-2017 16:47:02] WARNING: [pool admin] server reached pm.max_children setting (6), consider raising it
Why ???
When I got 50 in /etc/php-fpm.d/admin.conf ?
Should it be changed somewhere in /opt ?
-
Every each clients, you can change 'Number Of FastCGI Children' via 'limit' (or 'resouce plan') except for 'admin'.
-
Every each clients, you can change 'Number Of FastCGI Children' via 'limit' (or 'resouce plan') except for 'admin'.
I have no clients - only admin.
-
Every each clients, you can change 'Number Of FastCGI Children' via 'limit' (or 'resouce plan') except for 'admin'.
I have no clients - only admin.
Create new client and then 'change owner' of website and mysql from 'admin' to new client. And then, to minimize memory usage for php-fpm, change 'php-ftpm type' to 'ondemand' for 'admin'.
-
Create new client and then 'change owner' of website and mysql from 'admin' to new client. And then, to minimize memory usage for php-fpm, change 'php-ftpm type' to 'ondemand' for 'admin'.
[/quote]
It seems changing in /opt/configs/php-fpm/conf/php71m/php-fpm.d/admin.conf fixed the problem.
Such config works the best:
pm = ondemand
pm.max_children = 100
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 20000
-
Still crashing ;(
I've restored old backup from 28th of December. Maybe this will help...
It looks like these children are simply creating but not closing properly:
[08-Jan-2017 17:10:43] WARNING: [pool admin] server reached pm.max_children setting (100), consider raising it
-
We should add:
keepalive_requests 0;
to nginx config !
-
How much traffic your website till 100 php-fpm client is not enough?.
-
How much traffic your website till 100 php-fpm client is not enough?.
It's actually very small, but there are many search engine robots on my sites.
I got tons of such logs:
Sun Jan 08 18:21:57 2017 fastcgi:error [pid 28998] [mod_fastcgi.c(2241):] [FastCGI: neither POLLIN nor POLLOUT...] [world.domain.com]
-
Error of 'FastCGI: neither POLLIN nor POLLOUT...' tend to meaning 'unfinish process'. DDOS attack use this 'unfinish preocess' trick.
-
Error of 'FastCGI: neither POLLIN nor POLLOUT...' tend to meaning 'unfinish process'. DDOS attack use this 'unfinish preocess' trick.
No, even single entering phpinfo page is causing this error. Something is wrong in that mod_fastcgi package. Can you check it out with PHP 7.1 ?
-
Only found 5 times for this error in error_log file with 75KB.
-
Only found 5 times for this error in error_log file with 75KB.
So far stable ;) I'm waiting for the first crash.
-
2 crashes during the night and all pages never loading in the morning until I restart php-fpm :(
-
Almost stable now with these settings:
pm = dynamic
pm.max_children = 200
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 20
pm.max_requests = 500
pm.process_idle_timeout = 10s
-
ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'
Shows just 14M per each php-fpm process - so 200 children is just 2,8 GB RAM
-
Allright - full stable ;D