MRatWork Forum by Mustafa Ramadhan

Sawo Project - Kloxo-MR Discussions => Kloxo-MR Bugs and Requests => Topic started by: Spacedust on 2014-09-13, 15:13:43

Title: [RESOLVED] Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-13, 15:13:43: Nobody sees this ? chris ?

It all started after KloxoMR 7.0b with new configs

You just add domain:

- pages are stalled
- then pages instantly shows nginx error message
- then nothing is working at all apache+nginx died - no such server message from the browser
- then after a minute apache+nginx is restarted by watchdog and everything is working fine again

Quote
[Sat Sep 13 01:33:22 2014] [notice] FastCGI: process manager initialized (pid 31962)
[Sat Sep 13 14:06:16 2014] [notice] FastCGI: process manager initialized (pid 9202)
[Sat Sep 13 15:07:18 2014] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0)
[Sat Sep 13 15:07:18 2014] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
[Sat Sep 13 15:07:39 2014] [notice] FastCGI: process manager initialized (pid 19808)
[Sat Sep 13 15:08:16 2014] [notice] FastCGI: process manager initialized (pid 21851)
[Sat Sep 13 15:10:20 2014] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Sat Sep 13 15:10:20 2014] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
[Sat Sep 13 15:10:40 2014] [notice] FastCGI: process manager initialized (pid 28093)
[Sat Sep 13 15:11:17 2014] [notice] FastCGI: process manager initialized (pid 29858)
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-13, 16:40:12: See how it looks like !

It says no such file or directory in one place. But which file ???

(http://files.tinypic.pl/i/00575/5zo7uuxxlqkg.jpg)
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-13, 16:46:25: There are some nasty bugs - Mustafa fix this immediately !!!

This is the first one:

Quote
FCGIWrapper /home/kloxo/client/php5.fcgi .php

but Apache+nginx still dying even with only one domain.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-15, 01:21:57: Still not fixed !

Quote
[root@c44-49 /]# tail -f /var/log/httpd/error_log
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Sep 15 01:20:23 2014] [notice] Digest: done
[Mon Sep 15 01:20:23 2014] [notice] FastCGI: process manager initialized (pid 1984)
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Sep 15 01:20:23 2014] [notice] Apache/2.2.29 (Unix) DAV/2 mod_fastcgi/2.4.7 mod_rpaf/0.6 mod_ssl/2.2.29 OpenSSL/1.0.1e-fips mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Mon Sep 15 01:20:57 2014] [notice] caught SIGTERM, shutting down
[Mon Sep 15 01:20:57 2014] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Mon Sep 15 01:20:57 2014] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
^C
[root@c44-49 /]# yum upgrade
Loaded plugins: fastestmirror, priorities, replace
Loading mirror speeds from cached hostfile
* base: www.gtlib.gatech.edu
* elrepo: ftp.nluug.nl
* epel: mirror.its.dal.ca
* extras: www.gtlib.gatech.edu
* mratwork-epel: mirror.its.dal.ca
* mratwork-ius-stable: mirror.amsiohosting.net
* rpmforge: mirror.nl.leaseweb.net
* updates: www.gtlib.gatech.edu
Setting up Upgrade Process
No Packages marked for Update
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-15, 01:28:04: Happens on mod_ruid2 and suphp too.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-15, 01:31:24: Happens on Apache only configuration too.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-15, 01:33:51: Happens on nginx only configuration too:

2014/09/15 01:32:14 [crit] 12186#0: *3 open() "/home/httpd/tester.xxx.pl/stats/tester.xxx.pl-custom_log" failed (13: Permission denied) while logging request, client: 62.21.44.49, server: tester.xxx.pl, request: "GET /images/kloxo-mr.png HTTP/1.1", host: "www.tester.xxx.pl", referrer: "http://www.tester.xxx.pl/"
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-16, 22:04:07: Still not fixed even on Kloxo-MR 7.0.0.b-2014091602 with /script/cleanup. All pages are down till Kloxo executes magic commands:

21:03 Sep/16/2014: Closed Exec /usr/local/lxlabs/kloxo/cexe/closeinput '/etc/init.d/httpd graceful' >/dev/null 2>&1 &
21:03 Sep/16/2014: Closed Exec /usr/local/lxlabs/kloxo/cexe/closeinput '/etc/init.d/nginx restart' >/dev/null 2>&1 &
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-16, 22:12:48: Every delete/add domains, kloxo will be execute restart for services in queue (execute with createRestartFile).
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-16, 22:29:51: Quote from: MRatWork on 2014-09-16, 22:12:48
Every delete/add domains, kloxo will be execute restart for services in queue (execute with createRestartFile).

I know, but nginx+apache is dying before Kloxo executes /etc/init.d/nginx reload +/etc/init.d/httpd graceful

so you have about 20-60 seconds downtime on each domain added/removed...

It happens even on clean system with just 1 domain !
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-17, 00:57:26: Interesting thing - if I use Kloxo-MR 6.5.1.b-2014070704 folder from backup then it all works again, so our configs are fine, the problem is inside Kloxo code. What was changed ???
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-17, 01:34:09: Happens on Kloxo-MR 7.0.0.b-2014082703 too. I need to test older versions to see where this bug was introduced.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-17, 01:39:34: Suprise - Kloxo-MR 7.0.0.b-2014082402 works perfectly - so there must be a bug introduced from this version.

kloxomr7-7.0.0.b-2014082701.mr.noarch - is not working ok
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 12:49:11: I've tested latest configs with version Kloxo-MR 7.0.0.b-2014082402 and all works fine.

Something is inside Kloxo source code which is causing this ! I suspect httpdocs directory.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 13:13:19: The problem is inside /usr/local/lxlabs/kloxo/httpdocs/driver/web directory.

Now we have to find which file is causing this !
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-18, 13:14:38: Every process to add/delete/update will be execute 'sh /script/fixweb --domain=<domain>'.

Because at previous code, except '--target=defaults', will be delete all domains configs and it's make trouble for '--domain='. So, in latest update, this issue already fixed. Read fixweb.php.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 13:31:34: OK. I'm updating and testing.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 13:38:45: Just updated to Kloxo-MR 7.0.0.b-2014091704 and Apache+nginx is still dying so nothing was fixed.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 13:42:25: It seems /etc/init.d/nginx reload is not responding after adding domain:

Quote
[root@c44-49 ~]# /etc/init.d/nginx reload
[root@c44-49 ~]#

because it's dead (while it should be running):

Quote
[root@c44-49 ~]# ps aux | grep nginx
root 29775 0.0 0.0 103260 892 pts/1 S+ 13:42 0:00 grep nginx

so we have to restart it:

Quote
[root@c44-49 ~]# /etc/init.d/nginx restart
Zatrzymywanie nginx: [NIEUDANE]
Uruchamianie nginx: [ OK ]
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-18, 14:12:31: Need waiting 5-10 secs for restart web services.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-18, 14:27:35: I know, but it's dying completely, with old drivers everything is working great:

See the video: http://galeriaportali.pl/kloxomr.wmv
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-20, 17:59:46: Still not fixed in Kloxo-MR 7.0.0.b-2014092002 - I have to restart nginx manually, watchdog does not restart it.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-20, 18:40:01: Try copy /usr/local/lxlabs/kloxo/sbin/kloxo.php to custom.kloxo.php and modified 'timed_exec' value.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-21, 14:31:33: It does not help. Mustafa let me explain this one more time.

Old driver allowed nginx to just reload graceful after each domain was added/removed. No downtime !

New driver kills nginx so we have about 60-120 seconds downtime till it will be restarted - kloxo it trying to reload dead nginx but that is not possible !

Restore the old driver it was working much better.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-21, 15:14:30: What's nginx version better according your opinion.

Before centalt repo down, Kloxo-MR use nginx from centalt but after that use from nginx.org directly.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-21, 20:14:55: Quote from: MRatWork on 2014-09-21, 15:14:30
What's nginx version better according your opinion.

Before centalt repo down, Kloxo-MR use nginx from centalt but after that use from nginx.org directly.

nginx version does not matter, because it works fine. I'm using the latest upload with old webdrivers for Kloxo and all works fine. Kloxo is reloading nginx and gracefulling apache so there is no downtime when adding/deleting domains (nginx is displaying a copy of page from it's cache while apache is not ready).
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-22, 04:54:58: Update your Kloxo-MR 7.0.0.
Title: Re: Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-22, 16:21:38: It works again - thanks !!!
Title: Re: [RESOLVED] Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-26, 01:11:29: Still crashing when removing customers with many domains, FTP accounts etc.
Title: Re: [RESOLVED] Apache+nginx still dying after removing/adding domain
Post by: MRatWork on 2014-09-26, 02:09:35: Quote from: Spacedust on 2014-09-26, 01:11:29
Still crashing when removing customers with many domains, FTP accounts etc.
Try update.
Title: Re: [RESOLVED] Apache+nginx still dying after removing/adding domain
Post by: Spacedust on 2014-09-26, 14:09:22: I know what's the problem ! Kloxo-MR since version 7.0b does not reload php-fpm so fast after reloading gracefulling apache.

I'm on Kloxo-MR 7.0.0.b-2014092203 now