MRatWork Forum by Mustafa Ramadhan

Sawo Project - Kloxo-MR Discussions => Kloxo-MR Technical Helps => Topic started by: Anik on 2013-01-13, 00:43:58

Title: The connection was reset
Post by: Anik on 2013-01-13, 00:43:58

Sometimes (not always) when i access my site i see errors like..

Error 101 (net::ERR_CONNECTION_RESET): The connection was reset.

or

On Firefox:
Problem loading page
The connection to the server was reset while the page was loading.

on Chrome:
This webpage is not available
The connection to domain.com was interrupted.
Error 101 (net::ERR_CONNECTION_RESET): The connection was reset.

any idea what is the reason of it?

Title: Re: The connection was reset
Post by: Anik on 2013-01-13, 13:00:41

is it something due to timeout?
i never saw it in opera.

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-13, 21:36:10

It's because httpd is restarting in background. Mustafa should change restart to more graceful.

Title: Re: The connection was reset
Post by: Anik on 2013-01-13, 21:52:06

did you ever got this problem Marek? Is there any temporary fix?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-13, 22:04:25

Quote from: "Anik"

did you ever got this problem Marek? Is there any temporary fix?

I always got it.

Modify restart line in /etc/init.d/httpd to:

Code: [Select]

restart)
      $apachectl $@
        RETVAL=$?
        ;;

Title: Re: The connection was reset
Post by: Anik on 2013-01-14, 00:09:17

I'm using php-fpm with nginxproxy. will this fix apply for this setup?

and now the config is something like

Code: [Select]

#!/bin/bash
#
# httpd        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: The Apache HTTP Server is an efficient and extensible  
#       server implementing the current HTTP standards.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
# config: /etc/sysconfig/httpd
# pidfile: /var/run/httpd.pid
#
### BEGIN INIT INFO
# Provides: httpd
# Required-Start: $local_fs $remote_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Should-Start: distcache
# Short-Description: start and stop Apache HTTP Server
# Description: The Apache HTTP Server is an extensible server
#  implementing the current HTTP standards.
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server
# with the thread-based "worker" MPM; BE WARNED that some modules may not
# work correctly with a thread-based MPM; notably PHP will refuse to start.

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/sbin/apachectl
httpd=${HTTPD-/usr/sbin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd.pid}
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
RETVAL=0

# The semantics of these two functions differ from the way apachectl does
# things -- attempting to start while running is a failure, and shutdown
# when not running is also a failure.  So we just do it the way init scripts
# are expected to behave here.
start() {
        echo -n $"Starting $prog: "
        LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

# When stopping httpd a delay of >10 second is required before SIGKILLing the
# httpd parent; this gives enough time for the httpd parent to SIGKILL any
# errant children.
stop() {
echo -n $"Stopping $prog: "
killproc -p ${pidfile} -d 10 $httpd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=$?
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
    fi
    echo
}

# See how we were called.
case "$1" in
  start)
start
;;
  stop)
stop
;;
  status)
       status $httpd
RETVAL=$?
;;
  restart)
stop
start
# MR -- back to original because gracefull have a trouble with 2.2.23 version
#$apachectl configtest
#echo -e $"Gracefully restarting"
#$apachectl graceful
#status $httpd
;;
  condrestart)
if [ -f ${pidfile} ] ; then
stop
start
fi
;;
  force-reload|reload)
        reload
;;
  graceful|help|configtest|fullstatus)
$apachectl $@
RETVAL=$?
;;
  *)
echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
RETVAL=3
esac

exit $RETVAL

should i delete them all and put just this following 4 lines as you said?

Code: [Select]

restart)
      $apachectl $@
        RETVAL=$?
        ;;

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-14, 02:07:57

No just replace this:

Code: [Select]

 restart)
   stop
   start

Title: Re: The connection was reset
Post by: Anik on 2013-01-14, 13:35:06

after doing this apache failed to start and all my websites is now completely down. :(
I used this config

Code: [Select]

#!/bin/bash
#
# httpd        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: The Apache HTTP Server is an efficient and extensible  
#       server implementing the current HTTP standards.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
# config: /etc/sysconfig/httpd
# pidfile: /var/run/httpd.pid
#
### BEGIN INIT INFO
# Provides: httpd
# Required-Start: $local_fs $remote_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Should-Start: distcache
# Short-Description: start and stop Apache HTTP Server
# Description: The Apache HTTP Server is an extensible server
#  implementing the current HTTP standards.
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server
# with the thread-based "worker" MPM; BE WARNED that some modules may not
# work correctly with a thread-based MPM; notably PHP will refuse to start.

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/sbin/apachectl
httpd=${HTTPD-/usr/sbin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd.pid}
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
RETVAL=0

# The semantics of these two functions differ from the way apachectl does
# things -- attempting to start while running is a failure, and shutdown
# when not running is also a failure.  So we just do it the way init scripts
# are expected to behave here.
start() {
        echo -n $"Starting $prog: "
        LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

# When stopping httpd a delay of >10 second is required before SIGKILLing the
# httpd parent; this gives enough time for the httpd parent to SIGKILL any
# errant children.
stop() {
echo -n $"Stopping $prog: "
killproc -p ${pidfile} -d 10 $httpd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=$?
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
    fi
    echo
}

# See how we were called.
case "$1" in
  start)
start
;;
  stop)
stop
;;
  status)
       status $httpd
RETVAL=$?
;;
  restart)
        $apachectl $@
        RETVAL=$?
# MR -- back to original because gracefull have a trouble with 2.2.23 version
#$apachectl configtest
#echo -e $"Gracefully restarting"
#$apachectl graceful
#status $httpd
;;
  condrestart)
if [ -f ${pidfile} ] ; then
stop
start
fi
;;
  force-reload|reload)
        reload
;;
  graceful|help|configtest|fullstatus)
$apachectl $@
RETVAL=$?
;;
  *)
echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
RETVAL=3
esac

exit $RETVAL

is anything i missed out?

Title: Re: The connection was reset
Post by: Anik on 2013-01-14, 13:59:05

just fixed it. reason was some permission error.

Title: Re: The connection was reset
Post by: Anik on 2013-01-14, 14:09:49

but that above config didn't worked. apache isn't starting.

Title: Re: The connection was reset
Post by: Anik on 2013-01-14, 15:38:50

can you post here the exact config file which will work Marek?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-14, 16:47:33

Your are missing:

Quote

;;

before

Quote

# MR -- back to original because gracefull have a trouble with 2.2.23 version

Title: Re: The connection was reset
Post by: Anik on 2013-01-15, 15:04:39

tried that too before but failed

also there is already ;;

Code: [Select]

 restart)
        $apachectl $@
        RETVAL=$?
   # MR -- back to original because gracefull have a trouble with 2.2.23 version
   #$apachectl configtest
   #echo -e $"Gracefully restarting"
   #$apachectl graceful
   #status $httpd
   ;;

Title: Re: The connection was reset
Post by: Anik on 2013-01-15, 15:06:10

those lines are commented out so at the end of them theres already a ;;

Title: Re: The connection was reset
Post by: MRatWork on 2013-01-15, 16:20:17

Look inside '/usr/local/lxlabs/kloxo/httpdocs/htmllib/lib/lxserverlib.php' in 'function checkRestart()'

Code: [Select]

...
default:
exec_with_all_closed("/etc/init.d/$cmd restart");
...

How about if change to:

Code: [Select]

...
default:
// exec_with_all_closed("/etc/init.d/$cmd restart");

if (exec("service $cmd status | grep 'is running'")) {
exec_with_all_closed("/etc/init.d/$cmd reload");
} else {
exec_with_all_closed("/etc/init.d/$cmd restart");
}
...

It's mean if service status as 'running' just 'reload' and but if not running will be 'restart'

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-15, 16:43:53

Here's my config:

Code: [Select]

#!/bin/bash
#
# httpd        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: The Apache HTTP Server is an efficient and extensible  
#              server implementing the current HTTP standards.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
# config: /etc/sysconfig/httpd
# pidfile: /var/run/httpd.pid
#
### BEGIN INIT INFO
# Provides: httpd
# Required-Start: $local_fs $remote_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Should-Start: distcache
# Short-Description: start and stop Apache HTTP Server
# Description: The Apache HTTP Server is an extensible server
#  implementing the current HTTP standards.
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server
# with the thread-based "worker" MPM; BE WARNED that some modules may not
# work correctly with a thread-based MPM; notably PHP will refuse to start.

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/sbin/apachectl
httpd=${HTTPD-/usr/sbin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd.pid}
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
RETVAL=0

# The semantics of these two functions differ from the way apachectl does
# things -- attempting to start while running is a failure, and shutdown
# when not running is also a failure.  So we just do it the way init scripts
# are expected to behave here.
start() {
        echo -n $"Starting $prog: "
        LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

# When stopping httpd a delay of >10 second is required before SIGKILLing the
# httpd parent; this gives enough time for the httpd parent to SIGKILL any
# errant children.
stop() {
        echo -n $"Stopping $prog: "
        killproc -p ${pidfile} -d 10 $httpd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=$?
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
    fi
    echo
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status $httpd
        RETVAL=$?
        ;;
  restart)
      $apachectl $@
        RETVAL=$?
        ;;
  condrestart)
        if [ -f ${pidfile} ] ; then
                stop
                start
        fi
        ;;
  force-reload|reload)
        reload
        ;;
  graceful|help|configtest|fullstatus)
        $apachectl $@
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
        RETVAL=3
esac

exit $RETVAL


Title: Re: The connection was reset
Post by: Anik on 2013-01-15, 18:11:55

now theres 2. should i apply both fixes?  : :

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-15, 18:22:56

Quote from: "Anik"

now theres 2. should i apply both fixes?  : :

No just replace your /etc/init.d/httpd with the last one.

Title: Re: The connection was reset
Post by: Anik on 2013-01-15, 19:33:57

ok i replaced with Marek's config. but if i don't apply Mustafa's fix then he will be angry ->  :evil:
so tellme Mustafa, should i apply yours too?  :D

Title: Re: The connection was reset
Post by: Anik on 2013-01-15, 19:37:15

another point i want to mention. my php-fpm expires after some times. maybe 24 hours or less. so i have to restart it periodically.
is it related to this matter?

Title: Re: The connection was reset
Post by: MRatWork on 2013-01-15, 20:12:23

I hope you can try my modification.

I think, changing option on 'switch applications' and 'webserver configs' need restart service, but add/modified/delete domains (it's related to service configs) don't need restart service. Just enough reload.

If this modification running well, final release will be use this modification.

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-15, 23:30:15

No, no, no !

Reload won't add new domains etc.

It must be graceful !

"A reload is not sufficient as it does not reload the SSL certificates if there are changes there. Any new VirtualHosts, new domain, new subdoman, etc.. (even if they use the same certs) require a new read of the certs, and reload doesn't do this.

A restart is required... however, the graceful restart, mentioned above, can do a full restart without interruption of current connections."

Title: Re: The connection was reset
Post by: Anik on 2013-01-16, 13:21:41

So guys, there is now two modification. with which i should stick with?
Marek, did you ever got that connection reset problem in browser after applying your graceful restart fix on your own server?
Mustafa, you can try Marek's fix if he is certain that this problem won't happen again.
I am now watching the server and will update with further information, because this problem do not occurs always.

Title: Re: The connection was reset
Post by: MRatWork on 2013-01-16, 16:07:27

Quote from: "Anik"

So guys, there is now two modification. with which i should stick with?
Marek, did you ever got that connection reset problem in browser after applying your graceful restart fix on your own server?
Mustafa, you can try Marek's fix if he is certain that this problem won't happen again.
I am now watching the server and will update with further information, because this problem do not occurs always.

I know about proposal. But, I don't like this approach where with this approach to modified init config file.

As we know, some services have 'restart', 'reload' and 'graceful'. But 'graceful' not always exists.

I have a plan (I hope implementing on next release), when Kloxo 'restart' (that mean running createRestart) certain service (say it, httpd) then Kloxo process this depend on 'restart' type ( 'restart', 'reload', 'graceful' or others) form 'listing' file.

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-16, 23:37:19

Quote from: "Anik"

Marek, did you ever got that connection reset problem in browser after applying your graceful restart fix on your own server?

Almost never. Only 0,03% connections to my Varnish failed (probably due to some kernel swapping error):

Quote

client_conn            1383757        16.23 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req             3174663        37.24 Client requests received
cache_hit               365940         4.29 Cache hits
cache_hitpass               45         0.00 Cache hits for pass
cache_miss             2706253        31.74 Cache misses
backend_conn            188075         2.21 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail              2459         0.03 Backend conn. failures
backend_reuse          2807769        32.93 Backend conn. reuses
backend_toolate         100708         1.18 Backend conn. was closed
backend_recycle        2908890        34.12 Backend conn. recycles
backend_retry              635         0.01 Backend conn. retry

Title: Re: The connection was reset
Post by: Anik on 2013-01-17, 13:06:36

Quote

I have a plan (I hope implementing on next release),

So should we wait for the next release?  is upgrading from the current release will be okay? before i got some serious problem for which i need to reinstall everything, so now i fear to upgrade.  :|

Quote

backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 2459 0.03 Backend conn. failures
backend_reuse 2807769 32.93 Backend conn. reuses
backend_toolate 100708 1.18 Backend conn. was closed

where you get this stat?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-18, 01:41:16

Create varnish.php and put this inside the file:

Code: [Select]

<?php
// ==============================
// Server Uptime And Hardware Information |
// ==============================
?>

<html>
<head>
<title><?php echo $SERVER_NAME; ?> - Varnish Stat</title>
<STYLE type=text/css>
BODY { FONT-SIZE: 8pt; COLOR: black; FONT-FAMILY: Verdana,arial, helvetica, serif; margin : 0 0 0 0;}
</STYLE>
</head>
<body>
<pre>

<?php system("varnishstat -1"); ?>

</pre>


Title: Re: The connection was reset
Post by: Tsanten on 2013-01-18, 11:56:28

Or ssh varnishstat -1

Title: Re: The connection was reset
Post by: Anik on 2013-01-18, 13:44:18

Failed. PHP returned a WSOD
ssh varnishstat: command not found

should i need to install varnishstat via yum?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-18, 17:30:05

Quote from: "Anik"

Failed. PHP returned a WSOD
ssh varnishstat: command not found

should i need to install varnishstat via yum?

Yes

Title: Re: The connection was reset
Post by: Anik on 2013-01-19, 13:25:10

Update about connection reset problem.

Just saw it once in chrome. Should i now apply Mustafa's fix too with Marek's config?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-19, 22:55:53

Quote from: "Anik"

Update about connection reset problem.

Just saw it once in chrome. Should i now apply Mustafa's fix too with Marek's config?

My kernel 3.0.57 is still crashing with httpd and sometimes curl.

I will update to Kloxo-MR and this won't fix my issues I will update to 3.0.59.

Title: Re: The connection was reset
Post by: Anik on 2013-01-20, 00:01:01

so what do you recommend?

Title: Re: The connection was reset
Post by: Spacedust on 2013-01-20, 00:25:20

Quote from: "Anik"

so what do you recommend?

Test my fix

Title: Re: The connection was reset
Post by: Anik on 2013-01-20, 00:47:53

yes i am using your fix Marek (your httpd config file) but it recenly gave 2 connection reset error.

Title: Re: The connection was reset
Post by: Anik on 2013-01-24, 21:19:14

seen 4 more errors.