Thanks for reply. I see nothing in top command. I attached top results for now (only apache and mysql)
What is scavenge? and how can I find out if it's running, and how to disable it?
also I see following message on my VM console (when the server crushed)
ahead+0x53/0xb2
[<ffffffff80263a8a>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff80263ad4>] .text.lock.mutex+0xf/0x14
[<ffffffff8020d6f3>] do_lookup+0xf5/0x24b
[<ffffffff8020a98f>] __link_path_walk+0x9f4/0xf39
[<ffffffff8020efd2>] link_path_walk+0x45/0xb8
[<ffffffff8020d47d>] do_path_lookup+0x294/0x311
[<ffffffff80213387>] getname+0x15b/0x1c2
[<ffffffff80224762>] __user_walk_fd+0x37/0x4c
[<ffffffff80241269>] vfs_lstat_fd+0x18/0x47
[<ffffffff8022b8a8>] sys_newlstat+0x19/0x31
[<ffffffff80260295>] tracesys+0x47/0xb6
[<ffffffff802602f9>] tracesys+0xab/0xb6
INFO: task httpd:28740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
httpd D ffffffff8022d0e2 0 28740 28940 28743 28739 (NOTLB)
ffff8800a8dcbc78 0000000000000282 ffff8800a8dcbba8 ffffffff80264929
0000000000000008 ffff88018f2e30c0 ffff8801ffe8d0c0 0000000000f43d03
ffff88018f2e32a8 0000000000000000
Call Trace:
[<ffffffff80264929>] _spin_lock_irqsave+0x9/0x14
[<ffffffff802c6e4b>] try_to_free_pages+0x1da/0x2d7
[<ffffffff80263a8a>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff80263ad4>] .text.lock.mutex+0xf/0x14
[<ffffffff8020d6f3>] do_lookup+0xf5/0x24b
[<ffffffff8020a98f>] __link_path_walk+0x9f4/0xf39
[<ffffffff8020efd2>] link_path_walk+0x45/0xb8
[<ffffffff8020d47d>] do_path_lookup+0x294/0x311
[<ffffffff80213387>] getname+0x15b/0x1c2
[<ffffffff80224762>] __user_walk_fd+0x37/0x4c
[<ffffffff802296ac>] vfs_stat_fd+0x1b/0x4a
[<ffffffff80267d7b>] do_page_fault+0xfa5/0x131b
[<ffffffff80224508>] sys_newstat+0x19/0x31
[<ffffffff80260295>] tracesys+0x47/0xb6
[<ffffffff802602f9>] tracesys+0xab/0xb6
INFO: task httpd:29259 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
httpd D ffffffff8022d0e2 0 29259 28940 29260 29258 (NOTLB)
ffff88016b765c78 0000000000000286 ffff8801ff53e488 ffffffff8804db3d
0000000000000008 ffff8801b2c51830 ffff88012c3a1040 000000000002bcd9
ffff8801b2c51a18 ffffffff80207116
Call Trace:
[<ffffffff8804db3d>] :ext3:ext3_mark_iloc_dirty+0x300/0x368
[<ffffffff80207116>] kmem_cache_free+0x84/0xd7
[<ffffffff880317c5>] :jbd:journal_stop+0x1f7/0x203
[<ffffffff88055d23>] :ext3:__ext3_journal_stop+0x1f/0x3d
[<ffffffff80263a8a>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff80263ad4>] .text.lock.mutex+0xf/0x14
[<ffffffff8020d6f3>] do_lookup+0xf5/0x24b
[<ffffffff8020a98f>] __link_path_walk+0x9f4/0xf39
[<ffffffff8020efd2>] link_path_walk+0x45/0xb8
[<ffffffff8020d47d>] do_path_lookup+0x294/0x311
[<ffffffff80213387>] getname+0x15b/0x1c2
[<ffffffff80224762>] __user_walk_fd+0x37/0x4c
[<ffffffff80241269>] vfs_lstat_fd+0x18/0x47
[<ffffffff8022b8a8>] sys_newlstat+0x19/0x31
[<ffffffff80260295>] tracesys+0x47/0xb6
[<ffffffff802602f9>] tracesys+0xab/0xb6
INFO: task qmail-rspawn:3298 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
qmail-rspawn D 0000000080000000 0 3298 3288 27474 3299 3297 (NOTLB)
ffff8801f0cbfe78 0000000000000286 ffff8801ac41b080 0000000000000000
0000000000000007 ffff8801f136f080 ffff8801abc93080 000000000c3881f4
ffff8801f136f268 ffff8801ac41b080
Call Trace:
[<ffffffff8029cc79>] attach_pid+0x7c/0xa9
[<ffffffff8028a644>] enqueue_task+0x41/0x56
[<ffffffff80262fd7>] wait_for_completion+0x7d/0xaa
[<ffffffff8028acc0>] default_wake_function+0x0/0xe
[<ffffffff80232b10>] do_fork+0x17e/0x1c1
[<ffffffff802602f9>] tracesys+0xab/0xb6
[<ffffffff80260519>] ptregscall_common+0x3d/0x64
INFO: task qmail-rspawn:3298 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
qmail-rspawn D 0000000080000000 0 3298 3288 27474 3299 3297 (NOTLB)
ffff8801f0cbfe78 0000000000000286 ffff8801ac41b080 0000000000000000
0000000000000007 ffff8801f136f080 ffff8801abc93080 000000000c3881f4
ffff8801f136f268 ffff8801ac41b080
Call Trace:
[<ffffffff8029cc79>] attach_pid+0x7c/0xa9
[<ffffffff8028a644>] enqueue_task+0x41/0x56
[<ffffffff80262fd7>] wait_for_completion+0x7d/0xaa
[<ffffffff8028acc0>] default_wake_function+0x0/0xe
[<ffffffff80232b10>] do_fork+0x17e/0x1c1
[<ffffffff802602f9>] tracesys+0xab/0xb6
[<ffffffff80260519>] ptregscall_common+0x3d/0x64
INFO: task qmail-rspawn:3298 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
qmail-rspawn D 0000000080000000 0 3298 3288 27474 3299 3297 (NOTLB)
ffff8801f0cbfe78 0000000000000286 ffff8801ac41b080 0000000000000000
0000000000000007 ffff8801f136f080 ffff8801abc93080 000000000c3881f4
ffff8801f136f268 ffff8801ac41b080
Call Trace:
[<ffffffff8029cc79>] attach_pid+0x7c/0xa9
[<ffffffff8028a644>] enqueue_task+0x41/0x56
[<ffffffff80262fd7>] wait_for_completion+0x7d/0xaa
[<ffffffff8028acc0>] default_wake_function+0x0/0xe
[<ffffffff80232b10>] do_fork+0x17e/0x1c1
[<ffffffff802602f9>] tracesys+0xab/0xb6
[<ffffffff80260519>] ptregscall_common+0x3d/0x64
httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff802c353b>] out_of_memory+0x8b/0x203
[<ffffffff8020fac9>] __alloc_pages+0x27f/0x308
[<ffffffff80213a5c>] __do_page_cache_readahead+0xc8/0x1af
[<ffffffff802142d9>] filemap_nopage+0x14c/0x360
[<ffffffff80208e91>] __handle_mm_fault+0x444/0x144f
[<ffffffff8020d5c1>] do_sync_read+0xc7/0x104
[<ffffffff80267d48>] do_page_fault+0xf72/0x131b
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff8020e537>] do_mmap_pgoff+0x3f0/0x783
[<ffffffff8020e75f>] do_mmap_pgoff+0x618/0x783
[<ffffffff8026082b>] error_exit+0x0/0x6e
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:46
cpu 1 hot: high 186, batch 31 used:30
cpu 1 cold: high 62, batch 15 used:35
cpu 2 hot: high 186, batch 31 used:31
cpu 2 cold: high 62, batch 15 used:31
cpu 3 hot: high 186, batch 31 used:31
cpu 3 cold: high 62, batch 15 used:54
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:7
cpu 0 cold: high 62, batch 15 used:21
cpu 1 hot: high 186, batch 31 used:1
cpu 1 cold: high 62, batch 15 used:54
cpu 2 hot: high 186, batch 31 used:112
cpu 2 cold: high 62, batch 15 used:53
cpu 3 hot: high 186, batch 31 used:16
cpu 3 cold: high 62, batch 15 used:23
HighMem per-cpu: empty
Free pages: 29508kB (0kB HighMem)
Active:1059630 inactive:926875 dirty:0 writeback:0 unstable:0 free:7377 slab:8897 mapped-file:560 mapped-anon:1990606 pagetables:37382
DMA free:2040kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9052kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 8064 8064
DMA32 free:21816kB min:5732kB low:7164kB high:8596kB active:2054800kB inactive:1868912kB present:4120800kB pages_scanned:6493757 all_unreclaimable? yes
lowmem_reserve[]: 0 0 4040 4040
Normal free:5652kB min:5752kB low:7188kB high:8628kB active:2183720kB inactive:1838588kB present:4136960kB pages_scanned:28478562 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB
DMA32: 10*4kB 2*8kB 10*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 5*4096kB = 21816kB
Normal: 27*4kB 1*8kB 12*16kB 3*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5652kB
HighMem: empty
6121 pagecache pages
Swap cache: add 22246870, delete 22241367, find 6645993/9065426, race 10+577
Free swap = 0kB
Total swap = 6258680kB
Out of memory: Killed process 27795, UID 48, (httpd).
INFO: task qmail-rspawn:3298 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
qmail-rspawn D 0000000080000000 0 3298 3288 27474 3299 3297 (NOTLB)
ffff8801f0cbfe78 0000000000000286 ffff8801ac41b080 0000000000000000
0000000000000007 ffff8801f136f080 ffff8801abc93080 000000000c3881f4
ffff8801f136f268 ffff8801ac41b080
Call Trace:
[<ffffffff8029cc79>] attach_pid+0x7c/0xa9
[<ffffffff8028a644>] enqueue_task+0x41/0x56
[<ffffffff80262fd7>] wait_for_completion+0x7d/0xaa
[<ffffffff8028acc0>] default_wake_function+0x0/0xe
[<ffffffff80232b10>] do_fork+0x17e/0x1c1
[<ffffffff802602f9>] tracesys+0xab/0xb6
[<ffffffff80260519>] ptregscall_common+0x3d/0x64
INFO: task qmail-rspawn:3298 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
qmail-rspawn D 0000000080000000 0 3298 3288 27474 3299 3297 (NOTLB)
ffff8801f0cbfe78 0000000000000286 ffff8801ac41b080 0000000000000000
0000000000000007 ffff8801f136f080 ffff8801abc93080 000000000c3881f4
ffff8801f136f268 ffff8801ac41b080
Call Trace:
[<ffffffff8029cc79>] attach_pid+0x7c/0xa9
[<ffffffff8028a644>] enqueue_task+0x41/0x56
[<ffffffff80262fd7>] wait_for_completion+0x7d/0xaa
[<ffffffff8028acc0>] default_wake_function+0x0/0xe
[<ffffffff80232b10>] do_fork+0x17e/0x1c1
[<ffffffff802602f9>] tracesys+0xab/0xb6
[<ffffffff80260519>] ptregscall_common+0x3d/0x64
INFO: task kjournald:376 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88000116a460 0 376 7 399 353 (L-TLB)
ffff8801fb103dd0 0000000000000246 ffff8801fb103d60 ffffffff80263851
000000000000000a ffff8801fef5d040 ffff8800f89d8830 0000000000000add
ffff8801fef5d228 ffffffff80215e16
Call Trace:
[<ffffffff80263851>] __wait_on_bit+0x60/0x6e
[<ffffffff80215e16>] sync_buffer+0x0/0x3f
[<ffffffff8028a492>] dequeue_task+0x18/0x37
[<ffffffff8028a4d9>] deactivate_task+0x28/0x5f
[<ffffffff8026fa7b>] monotonic_clock+0x35/0x7b
[<ffffffff80264929>] _spin_lock_irqsave+0x9/0x14
[<ffffffff88033639>] :jbd:journal_commit_transaction+0x173/0x10c6
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff8024d2af>] try_to_del_timer_sync+0x7f/0x88
[<ffffffff8803772d>] :jbd:kjournald+0xc1/0x213
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff8029f0e2>] keventd_create_kthread+0x0/0xc4
[<ffffffff8803766c>] :jbd:kjournald+0x0/0x213
[<ffffffff8029f0e2>] keventd_create_kthread+0x0/0xc4
[<ffffffff80233ed3>] kthread+0xfe/0x132
[<ffffffff80260b2c>] child_rip+0xa/0x12
[<ffffffff8029f0e2>] keventd_create_kthread+0x0/0xc4
[<ffffffff80233dd5>] kthread+0x0/0x132
[<ffffffff80260b22>] child_rip+0x0/0x12
INFO: task syslogd:1419 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslogd D 0000000000000002 0 1419 1 1422 1398 (NOTLB)
ffff8801f9d09a48 0000000000000282 0000000000000000 0000000000000000
0000000000000005 ffff8801fe8c1830 ffff88010560b040 00000000000863de
ffff8801fe8c1a18 0000000000000003
Call Trace:
[<ffffffff8020b0be>] get_page_from_freelist+0x1ea/0x3fc
[<ffffffff80264929>] _spin_lock_irqsave+0x9/0x14
[<ffffffff88031fdb>] :jbd:start_this_handle+0x2e9/0x370
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff8803212d>] :jbd:journal_start+0xcb/0x102
[<ffffffff8805053c>] :ext3:ext3_write_begin+0x9a/0x1ce
[<ffffffff802109b0>] generic_file_buffered_write+0x14b/0x640
[<ffffffff880317c5>] :jbd:journal_stop+0x1f7/0x203
[<ffffffff80216eed>] __generic_file_aio_write_nolock+0x369/0x3b6
[<ffffffff8026fa7b>] monotonic_clock+0x35/0x7b
[<ffffffff8028a6af>] __activate_task+0x56/0x6d
[<ffffffff802c2708>] __generic_file_write_nolock+0x8f/0xa8
[<ffffffff80249ffd>] pagevec_lookup_tag+0x1a/0x21
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff8029f2fa>] autoremove_wake_function+0x0/0x2e
[<ffffffff80207116>] kmem_cache_free+0x84/0xd7
[<ffffffff80263920>] mutex_lock+0xd/0x1d
[<ffffffff8020f0c7>] find_get_pages_tag+0x82/0x8d
[<ffffffff802c2769>] generic_file_writev+0x48/0xa3
[<ffffffff8021880e>] do_sync_write+0x0/0x104
[<ffffffff802d73f6>] do_readv_writev+0x172/0x291
[<ffffffff8021880e>] do_sync_write+0x0/0x104
[<ffffffff802d762d>] sys_writev+0x45/0x93
[<ffffffff802602f9>] tracesys+0xab/0xb6
httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff802c353b>] out_of_memory+0x8b/0x203
[<ffffffff8020fac9>] __alloc_pages+0x27f/0x308
[<ffffffff80213a5c>] __do_page_cache_readahead+0xc8/0x1af
[<ffffffff802142d9>] filemap_nopage+0x14c/0x360
[<ffffffff80208e91>] __handle_mm_fault+0x444/0x144f
[<ffffffff80267d48>] do_page_fault+0xf72/0x131b
[<ffffffff8022d6ca>] mntput_no_expire+0x19/0x89
[<ffffffff80233c33>] sys_faccessat+0x148/0x18d
[<ffffffff8026082b>] error_exit+0x0/0x6e
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
cpu 1 hot: high 186, batch 31 used:12
cpu 1 cold: high 62, batch 15 used:44
cpu 2 hot: high 186, batch 31 used:31
cpu 2 cold: high 62, batch 15 used:14
cpu 3 hot: high 186, batch 31 used:18
cpu 3 cold: high 62, batch 15 used:47
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:4
cpu 0 cold: high 62, batch 15 used:54
cpu 1 hot: high 186, batch 31 used:28
cpu 1 cold: high 62, batch 15 used:45
cpu 2 hot: high 186, batch 31 used:102
cpu 2 cold: high 62, batch 15 used:41
cpu 3 hot: high 186, batch 31 used:4
cpu 3 cold: high 62, batch 15 used:50
HighMem per-cpu: empty
Free pages: 29644kB (0kB HighMem)
Active:972322 inactive:1014303 dirty:0 writeback:0 unstable:0 free:7411 slab:8894 mapped-file:574 mapped-anon:1990428 pagetables:37199
DMA free:2040kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9052kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 8064 8064
DMA32 free:21860kB min:5732kB low:7164kB high:8596kB active:1954372kB inactive:1964060kB present:4120800kB pages_scanned:6724522 all_unreclaimable? yes
lowmem_reserve[]: 0 0 4040 4040
Normal free:5744kB min:5752kB low:7188kB high:8628kB active:1934916kB inactive:2093152kB present:4136960kB pages_scanned:15172091 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB
DMA32: 19*4kB 3*8kB 10*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 5*4096kB = 21860kB
Normal: 38*4kB 9*8kB 11*16kB 3*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5744kB
HighMem: empty
6201 pagecache pages
Swap cache: add 22259120, delete 22253616, find 6651148/9072152, race 10+609
Free swap = 0kB
Total swap = 6258680kB
Out of memory: Killed process 28155, UID 48, (httpd).
CentOS release 5.11 (Final)
Kernel 2.6.18-409.el5xen on an x86_64
Also I should mention that I have a very high (Disk write/Disk read) in my server when disk happening. (about 1MB/sec and more)