[URGENT] WARN: admin@xxx.pl: chdir(/home/lxadmin/mail/domains/xxx.pl

[Spacedust]

Replaced all hardware and after 9 minutes the problem exists again ;(

[MRatWork]

What's you mean 'replaced all hardware'?.

[Spacedust]

What's you mean 'replaced all hardware'?.

CPU, motherboard, 4 disks etc.

[Spacedust]

Something new:

nginx: page allocation failure. order:1, mode:0x20
Pid: 2292, comm: nginx Not tainted 2.6.32-504.30.3.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff811345aa>] ? __alloc_pages_nodemask+0x74a/0x8d0
 [<ffffffff811737a2>] ? kmem_getpages+0x62/0x170
 [<ffffffff811743ba>] ? fallback_alloc+0x1ba/0x270
 [<ffffffff81173e0f>] ? cache_grow+0x2cf/0x320
 [<ffffffff81174139>] ? ____cache_alloc_node+0x99/0x160
 [<ffffffff81174f43>] ? kmem_cache_alloc+0x123/0x190
 [<ffffffff8144cdd8>] ? sk_prot_alloc+0x48/0x1c0
 [<ffffffff8144e002>] ? sk_clone+0x22/0x2e0
 [<ffffffff814a2256>] ? inet_csk_clone+0x16/0xd0
 [<ffffffff814bbd83>] ? tcp_create_openreq_child+0x23/0x470
 [<ffffffff814b952d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
 [<ffffffff814bbb26>] ? tcp_check_req+0x226/0x460
 [<ffffffff814b8f6b>] ? tcp_v4_do_rcv+0x35b/0x490
 [<ffffffffa0207557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
 [<ffffffff814ba812>] ? tcp_v4_rcv+0x532/0x910
 [<ffffffff814974f0>] ? ip_local_deliver_finish+0x0/0x2d0
 [<ffffffff814975cd>] ? ip_local_deliver_finish+0xdd/0x2d0
 [<ffffffff81497858>] ? ip_local_deliver+0x98/0xa0
 [<ffffffff81496d1d>] ? ip_rcv_finish+0x12d/0x440
 [<ffffffff814972a5>] ? ip_rcv+0x275/0x350
 [<ffffffff8145ce48>] ? __netif_receive_skb+0x208/0x570
 [<ffffffff81460c08>] ? netif_receive_skb+0x58/0x60
 [<ffffffff81460d10>] ? napi_skb_finish+0x50/0x70
 [<ffffffff81462619>] ? napi_gro_receive+0x39/0x50
 [<ffffffffa0159a58>] ? e1000_receive_skb+0x58/0x90 [e1000]
 [<ffffffffa015e392>] ? e1000_clean_rx_irq+0x2b2/0x5c0 [e1000]
 [<ffffffffa015bc5f>] ? e1000_clean+0x1ef/0x990 [e1000]
 [<ffffffff81462733>] ? net_rx_action+0x103/0x2f0
 [<ffffffff8107d901>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100fbd5>] ? do_softirq+0x65/0xa0
 [<ffffffff8107d7b5>] ? irq_exit+0x85/0x90
 [<ffffffff81533c95>] ? do_IRQ+0x75/0xf0
 [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
 <EOI>  [<ffffffff81130a9f>] ? get_pageblock_flags_group+0x7f/0xa0
 [<ffffffff811348e8>] ? free_hot_cold_page+0xc8/0x220
 [<ffffffff81146e0e>] ? __dec_zone_page_state+0x2e/0x30
 [<ffffffff81134a82>] ? __pagevec_free+0x42/0x90
 [<ffffffff8113b0cc>] ? release_pages+0x21c/0x250
 [<ffffffff8113b484>] ? ____pagevec_lru_add+0x164/0x180
 [<ffffffff8113bbfb>] ? lru_add_drain_cpu+0x8b/0xc0
 [<ffffffff8113bc76>] ? __pagevec_release+0x26/0x40
 [<ffffffff8113c578>] ? truncate_inode_pages_range+0x358/0x500
 [<ffffffff811f2b61>] ? dqget+0x291/0x390
 [<ffffffff811f1eac>] ? dqput+0x5c/0x200
 [<ffffffff811f3173>] ? dquot_initialize+0xe3/0x1b0
 [<ffffffffa009de80>] ? ext4_delete_inode+0x0/0x2f0 [ext4]
 [<ffffffff8113c7b5>] ? truncate_inode_pages+0x15/0x20
 [<ffffffffa009dece>] ? ext4_delete_inode+0x4e/0x2f0 [ext4]
 [<ffffffff811ac45e>] ? generic_delete_inode+0xde/0x1d0
 [<ffffffff811ac5b5>] ? generic_drop_inode+0x65/0x80
 [<ffffffff811ab402>] ? iput+0x62/0x70
 [<ffffffff811a7ff0>] ? dentry_iput+0x90/0x100
 [<ffffffff811a8151>] ? d_kill+0x31/0x60
 [<ffffffff811a9d2c>] ? dput+0x7c/0x150
 [<ffffffff8118fcb9>] ? __fput+0x189/0x210
 [<ffffffff8118fd65>] ? fput+0x25/0x30
 [<ffffffff8118afbd>] ? filp_close+0x5d/0x90
 [<ffffffff8118b095>] ? sys_close+0xa5/0x100
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
nginx: page allocation failure. order:1, mode:0x20
Pid: 2314, comm: nginx Not tainted 2.6.32-504.30.3.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff811345aa>] ? __alloc_pages_nodemask+0x74a/0x8d0
 [<ffffffff811737a2>] ? kmem_getpages+0x62/0x170
 [<ffffffff811743ba>] ? fallback_alloc+0x1ba/0x270
 [<ffffffff81173e0f>] ? cache_grow+0x2cf/0x320
 [<ffffffff81174139>] ? ____cache_alloc_node+0x99/0x160
 [<ffffffff81174f43>] ? kmem_cache_alloc+0x123/0x190
 [<ffffffff8144cdd8>] ? sk_prot_alloc+0x48/0x1c0
 [<ffffffff8144e002>] ? sk_clone+0x22/0x2e0
 [<ffffffff814a2256>] ? inet_csk_clone+0x16/0xd0
 [<ffffffff814bbd83>] ? tcp_create_openreq_child+0x23/0x470
 [<ffffffff814b952d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
 [<ffffffff814bbb26>] ? tcp_check_req+0x226/0x460
 [<ffffffff814b8f6b>] ? tcp_v4_do_rcv+0x35b/0x490
 [<ffffffffa0207557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
 [<ffffffff814ba812>] ? tcp_v4_rcv+0x532/0x910
 [<ffffffff814974f0>] ? ip_local_deliver_finish+0x0/0x2d0
 [<ffffffff814975cd>] ? ip_local_deliver_finish+0xdd/0x2d0
 [<ffffffff81497858>] ? ip_local_deliver+0x98/0xa0
 [<ffffffff81496d1d>] ? ip_rcv_finish+0x12d/0x440
 [<ffffffff814972a5>] ? ip_rcv+0x275/0x350
 [<ffffffff8145ce48>] ? __netif_receive_skb+0x208/0x570
 [<ffffffff8145d24a>] ? process_backlog+0x9a/0x100
 [<ffffffff81462733>] ? net_rx_action+0x103/0x2f0
 [<ffffffff8107d901>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 <EOI>  [<ffffffff8100fbd5>] ? do_softirq+0x65/0xa0
 [<ffffffff8107e5fa>] ? local_bh_enable+0x9a/0xb0
 [<ffffffff81462344>] ? dev_queue_xmit+0x124/0x320
 [<ffffffff8149d6f8>] ? ip_finish_output+0x148/0x310
 [<ffffffff8149d978>] ? ip_output+0xb8/0xc0
 [<ffffffff8149cc1f>] ? __ip_local_out+0x9f/0xb0
 [<ffffffff8149cc55>] ? ip_local_out+0x25/0x30
 [<ffffffff8149d150>] ? ip_queue_xmit+0x190/0x420
 [<ffffffff814b2684>] ? tcp_transmit_skb+0x4b4/0x8b0
 [<ffffffff814b4bca>] ? tcp_write_xmit+0x1da/0xa90
 [<ffffffff814b57b0>] ? __tcp_push_pending_frames+0x30/0xe0
 [<ffffffff814a48be>] ? tcp_push+0x6e/0x90
 [<ffffffff814a58bc>] ? tcp_sendmsg+0x64c/0xa20
 [<ffffffff8144ad9b>] ? sock_aio_write+0x19b/0x1c0
 [<ffffffff8144ac00>] ? sock_aio_write+0x0/0x1c0
 [<ffffffff8118df6b>] ? do_sync_readv_writev+0xfb/0x140
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8107d42a>] ? local_bh_enable_ip+0x9a/0xb0
 [<ffffffff8152d18b>] ? _spin_unlock_bh+0x1b/0x20
 [<ffffffff8122da86>] ? security_file_permission+0x16/0x20
 [<ffffffff8118efc6>] ? do_readv_writev+0xd6/0x1f0
 [<ffffffff8118f126>] ? vfs_writev+0x46/0x60
 [<ffffffff8118f251>] ? sys_writev+0x51/0xb0
 [<ffffffff810e5c5e>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
swapper: page allocation failure. order:1, mode:0x20
Pid: 0, comm: swapper Not tainted 2.6.32-504.30.3.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff811345aa>] ? __alloc_pages_nodemask+0x74a/0x8d0
 [<ffffffff812a2798>] ? swiotlb_dma_mapping_error+0x18/0x30
 [<ffffffff811737a2>] ? kmem_getpages+0x62/0x170
 [<ffffffff811743ba>] ? fallback_alloc+0x1ba/0x270
 [<ffffffff81173e0f>] ? cache_grow+0x2cf/0x320
 [<ffffffff81174139>] ? ____cache_alloc_node+0x99/0x160
 [<ffffffff81174f43>] ? kmem_cache_alloc+0x123/0x190
 [<ffffffff8144cdd8>] ? sk_prot_alloc+0x48/0x1c0
 [<ffffffff8144e002>] ? sk_clone+0x22/0x2e0
 [<ffffffff814a2256>] ? inet_csk_clone+0x16/0xd0
 [<ffffffff814bbd83>] ? tcp_create_openreq_child+0x23/0x470
 [<ffffffff814b952d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
 [<ffffffff814bbb26>] ? tcp_check_req+0x226/0x460
 [<ffffffff814b8f6b>] ? tcp_v4_do_rcv+0x35b/0x490
 [<ffffffffa0207557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
 [<ffffffff814ba812>] ? tcp_v4_rcv+0x532/0x910
 [<ffffffff814974f0>] ? ip_local_deliver_finish+0x0/0x2d0
 [<ffffffff814975cd>] ? ip_local_deliver_finish+0xdd/0x2d0
 [<ffffffff81497858>] ? ip_local_deliver+0x98/0xa0
 [<ffffffff81496d1d>] ? ip_rcv_finish+0x12d/0x440
 [<ffffffff814972a5>] ? ip_rcv+0x275/0x350
 [<ffffffff8145ce48>] ? __netif_receive_skb+0x208/0x570
 [<ffffffff81460c08>] ? netif_receive_skb+0x58/0x60
 [<ffffffff81460d10>] ? napi_skb_finish+0x50/0x70
 [<ffffffff81462619>] ? napi_gro_receive+0x39/0x50
 [<ffffffffa0159a58>] ? e1000_receive_skb+0x58/0x90 [e1000]
 [<ffffffffa015e392>] ? e1000_clean_rx_irq+0x2b2/0x5c0 [e1000]
 [<ffffffffa015bc5f>] ? e1000_clean+0x1ef/0x990 [e1000]
 [<ffffffff810f0071>] ? cpu_quiet_msk+0xc1/0x130
 [<ffffffff810f01f4>] ? rcu_process_dyntick+0x114/0x120
 [<ffffffff81462733>] ? net_rx_action+0x103/0x2f0
 [<ffffffff8107d901>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100fbd5>] ? do_softirq+0x65/0xa0
 [<ffffffff8107d7b5>] ? irq_exit+0x85/0x90
 [<ffffffff81533c95>] ? do_IRQ+0x75/0xf0
 [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
 <EOI>  [<ffffffff812eabee>] ? intel_idle+0xde/0x170
 [<ffffffff812eabd1>] ? intel_idle+0xc1/0x170
 [<ffffffff81426207>] ? cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
 [<ffffffff8151075a>] ? rest_init+0x7a/0x80
 [<ffffffff81c29f8f>] ? start_kernel+0x424/0x430
 [<ffffffff81c2933a>] ? x86_64_start_reservations+0x125/0x129
 [<ffffffff81c29453>] ? x86_64_start_kernel+0x115/0x124
nginx: page allocation failure. order:1, mode:0x20
Pid: 2292, comm: nginx Not tainted 2.6.32-504.30.3.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff811345aa>] ? __alloc_pages_nodemask+0x74a/0x8d0
 [<ffffffff814629cf>] ? netif_rx+0xaf/0x160
 [<ffffffff811737a2>] ? kmem_getpages+0x62/0x170
 [<ffffffff811743ba>] ? fallback_alloc+0x1ba/0x270
 [<ffffffff81173e0f>] ? cache_grow+0x2cf/0x320
 [<ffffffff81174139>] ? ____cache_alloc_node+0x99/0x160
 [<ffffffff81174f43>] ? kmem_cache_alloc+0x123/0x190
 [<ffffffff8144cdd8>] ? sk_prot_alloc+0x48/0x1c0
 [<ffffffff8144e002>] ? sk_clone+0x22/0x2e0
 [<ffffffff814a2256>] ? inet_csk_clone+0x16/0xd0
 [<ffffffff814bbd83>] ? tcp_create_openreq_child+0x23/0x470
 [<ffffffff814b952d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
 [<ffffffff814bbb26>] ? tcp_check_req+0x226/0x460
 [<ffffffff814b8f6b>] ? tcp_v4_do_rcv+0x35b/0x490
 [<ffffffffa0207557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
 [<ffffffff814ba812>] ? tcp_v4_rcv+0x532/0x910
 [<ffffffff814974f0>] ? ip_local_deliver_finish+0x0/0x2d0
 [<ffffffff814975cd>] ? ip_local_deliver_finish+0xdd/0x2d0
 [<ffffffff81497858>] ? ip_local_deliver+0x98/0xa0
 [<ffffffff81496d1d>] ? ip_rcv_finish+0x12d/0x440
 [<ffffffff814972a5>] ? ip_rcv+0x275/0x350
 [<ffffffff8145ce48>] ? __netif_receive_skb+0x208/0x570
 [<ffffffff8145d24a>] ? process_backlog+0x9a/0x100
 [<ffffffff81462733>] ? net_rx_action+0x103/0x2f0
 [<ffffffff8107d901>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
 <EOI>  [<ffffffff8100fbd5>] ? do_softirq+0x65/0xa0
 [<ffffffff8107e5fa>] ? local_bh_enable+0x9a/0xb0
 [<ffffffff81462344>] ? dev_queue_xmit+0x124/0x320
 [<ffffffff8149d6f8>] ? ip_finish_output+0x148/0x310
 [<ffffffff8149d978>] ? ip_output+0xb8/0xc0
 [<ffffffff8149cc1f>] ? __ip_local_out+0x9f/0xb0
 [<ffffffff8149cc55>] ? ip_local_out+0x25/0x30
 [<ffffffff8149d150>] ? ip_queue_xmit+0x190/0x420
 [<ffffffff814b2684>] ? tcp_transmit_skb+0x4b4/0x8b0
 [<ffffffff814b4bca>] ? tcp_write_xmit+0x1da/0xa90
 [<ffffffff814b57b0>] ? __tcp_push_pending_frames+0x30/0xe0
 [<ffffffff814a48be>] ? tcp_push+0x6e/0x90
 [<ffffffff814a58bc>] ? tcp_sendmsg+0x64c/0xa20
 [<ffffffff8144ad9b>] ? sock_aio_write+0x19b/0x1c0
 [<ffffffff8144ac00>] ? sock_aio_write+0x0/0x1c0
 [<ffffffff8118df6b>] ? do_sync_readv_writev+0xfb/0x140
 [<ffffffff814474cd>] ? sock_poll+0x5d/0x140
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811d6d40>] ? ep_send_events_proc+0x0/0x110
 [<ffffffff8122da86>] ? security_file_permission+0x16/0x20
 [<ffffffff8118efc6>] ? do_readv_writev+0xd6/0x1f0
 [<ffffffff8118f126>] ? vfs_writev+0x46/0x60
 [<ffffffff8118f251>] ? sys_writev+0x51/0xb0
 [<ffffffff810e5c5e>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b

[Spacedust]

Adding this to sysctl unlocked the server for about 2 minutes then it locked again:

vm.min_free_kbytes = 512000
vm.zone_reclaim_mode = 1

[Spacedust]

That was broken hardware issue - moved all data to a new server and suddenly everything works fine.