Mutex Contention in SOlaris 2.6. Please somebody give me a hints. Josè
I have a Solaris System, E10K with 8 CPU of 400 MHZ 6 GB of RAM
and Oracle 7.34, 3 instances and a messagging local packages running.
I detected a Mutex contention, because the CPU (sys) > 20% and > CPU (usr)
besides mpstat (smtx) > 500. sometime CPU (sys)>60 % idle (0%)
How can I fix that problems or detect the fail ?
Is there some scheme or procedure that permit me to localize the problems.
troubleshooting scheme ?
I tried lockstat but the result sometime confused me ?
Please give me some Hints or documents that permit me to clarify the problems
waiting for some hints, recomendation, etc.
Regards,
Josh Mhndez
a little Sun Manager
I 'm attaching some information.
TOP:
last pid: 22463; load averages: 34.05, 34.29, 31.93
10:56:50
2329 processes:2292 sleeping, 27 running, 5 zombie, 5 on cpu
CPU states: 0.0% idle, 40.2% user, 59.8% kernel, 0.0% iowait, 0.0% swap
Memory: 6144M real, 191M free, 41M swap in use, 2638M swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
17223 root 1 0 0 7248K 7144K run 88:49 6.27% cai_UxOs_ProcCo
28824 root 1 0 0 3720K 2576K cpu11 2:30 1.74% top
25059 root 1 0 0 3624K 2568K sleep 105:54 1.65% top
8035 sqltmon 1 0 0 3736K 2584K run 2:03 1.54% top
23080 root 1 0 0 3680K 2576K sleep 15:42 1.28% top
8028 oracle 1 44 0 470M 464M sleep 1:09 1.03% oracle
3084 oracle 1 46 0 470M 460M sleep 119:44 1.03% oracle
1 root 1 59 0 792K 400K sleep 51:02 0.86% init
16473 root 11 59 0 11M 9680K sleep 33:19 0.44% caiUxOs
3329 eculink 1 59 0 4664K 4256K sleep 44:52 0.38% eculink31_2
9964 oracle 1 39 0 470M 464M sleep 0:06 0.38% oracle
27459 oracle 1 33 0 470M 464M sleep 1:17 0.36% oracle
27602 oracle 1 49 0 470M 464M sleep 0:51 0.32% oracle
28263 oracle 1 40 0 470M 464M sleep 0:48 0.28% oracle
27554 oracle 1 9 0 470M 464M sleep 0:01 0.28% oracle
Note:
ca is CA TNG, normally work just for 2 minutes collecting monitoring
information.
eculink is the local messaging application that work strong with sockets
schemes.
top are the desesperated administrator trying to detect something :) waiting
for your collaboration
MPSTAT
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 679 3 12188 499 100 1682 381 292 508 4 10719 36 47 4 13
2 742 1 3604 781 200 1928 419 308 574 6 13298 49 31 4 16
8 637 1 2402 634 100 2203 524 331 632 5 14021 46 34 6 14
9 727 1 3155 2120 1675 1806 436 291 579 4 11067 41 43 3 13
10 594 2 2304 602 100 2156 487 317 598 4 12831 41 39 3 17
11 555 2 2298 577 100 1963 465 291 528 2 11143 40 41 4 15
12 719 2 2664 661 100 2304 558 359 649 6 13672 45 35 5 15
14 667 2 3077 651 100 2418 539 374 676 7 13750 49 28 6 17
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 765 4 15096 735 100 2303 621 300 649 0 16339 54 36 2 8
2 618 2 12073 825 200 1840 464 275 564 0 13710 57 36 1 7
8 791 2 11168 677 100 2028 573 268 570 0 14517 56 37 1 7
9 525 2 13593 2254 1800 1678 424 278 558 0 11961 41 48 2 8
10 596 2 9118 655 100 2059 548 264 586 0 14025 51 42 1 6
11 739 1 4033 654 100 2026 539 273 600 0 12734 50 42 2 6
12 674 3 15864 713 100 2267 602 307 628 0 15451 56 37 2 5
14 570 4 18122 651 100 1989 539 273 571 0 13083 47 47 1 5
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 573 4 2945 512 100 2021 402 366 577 4 13698 51 26 6 18
2 543 1 2817 612 200 1384 237 253 426 7 10103 36 45 2 17
8 551 3 2588 558 100 2274 446 370 622 5 14352 50 27 4 19
9 413 2 1971 2156 1768 1507 302 269 483 4 8653 34 47 3 16
10 730 4 15993 496 100 1853 382 323 552 5 12320 41 36 5 18
11 588 4 3462 570 100 2140 456 350 624 5 12296 41 34 7 18
12 568 4 3324 581 100 2255 462 374 674 6 12673 48 31 5 16
14 414 3 2393 543 100 2080 424 354 610 4 11885 39 38 4 19
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 312 3 2522 517 100 1733 398 306 506 4 12326 45 46 2 8
2 392 6 3382 762 200 1903 391 341 534 8 16208 56 31 5 8
8 801 4 3697 581 100 2027 464 323 579 9 14307 55 33 4 7
9 513 4 2798 2385 1920 1731 380 308 583 6 11683 50 38 5 7
10 366 4 12360 515 100 1762 396 285 506 12 12463 39 51 5 5
11 419 5 2961 574 100 2174 452 362 619 8 15063 49 36 7 8
12 500 5 3127 680 100 2570 559 411 721 8 15136 52 31 5 12
14 681 5 3076 644 100 2327 532 404 682 8 15058 57 31 3 9
PRTMEM
Total memory: 5963 Megabytes
Kernel Memory: 343 Megabytes
Application: 497 Megabytes
Executable & libs: 250 Megabytes
File Cache: 4679 Megabytes
Free, file cache: 73 Megabytes
Free, free: 142 Megabytes
LOCKSTAT sleep 5
the documentation refer the bold hexadecimal number as a hashed locks
how can associate that with something coherent ?
Adaptive mutex spin: 27115 events
Count indv cuml rcnt spin Lock Caller
-----------------------------------------------------------------------------
--
9807 36% 36% 1.00 58 0x70538008 semconv_lock+0x60
6474 24% 60% 1.00 64 0x70538008 cv_wait_sig+0x1b8
2054 8% 68% 1.00 46 0x70538008 cv_wait_sig+0x230
1757 6% 74% 1.00 33 service_queue background+0xdc
1331 5% 79% 1.00 30 service_queue background+0x120
1298 5% 84% 1.00 22 service_queue qenable_locked+0x34
934 3% 87% 1.00 97 0x724b2d30 qfestart+0x1f8
594 2% 89% 1.00 4 tod_lock uniqtime+0x14
184 1% 90% 1.00 3 hblk8_lock sfmmu_hblk_alloc+0xdc
167 1% 91% 1.00 76 0x724b2d30 qfeintr+0x2bc
126 0% 91% 1.00 3 hblk8_lock
sfmmu_hblk_tofreelist+0x38
100 0% 92% 1.00 5 0x704e4018 untimeout+0x18
79 0% 92% 1.00 11 0x704e4018 callout_execute+0xcc
73 0% 92% 1.00 6 0x703234b0 sync_stream_buf+0x24
70 0% 92% 1.00 4 0x704e4018 timeout_common+0x4
68 0% 93% 1.00 6 0x7028731c putnext+0xc
63 0% 93% 1.00 30 service_queue queuerun+0x1cc
58 0% 93% 1.00 9 pr_pidlock pr_p_lock+0xc
56 0% 93% 1.00 15 0x7028473c queuerun+0x198
53 0% 93% 1.00 3 0x7249b220
ufs_lockfs_begin_getpage+0x38
53 0% 94% 1.00 4 0x70239560 untimeout+0x18
53 0% 94% 1.00 4 0x70289f90 putnext+0x88
52 0% 94% 1.00 3 0x70289f90 putnext+0x30
52 0% 94% 1.00 7 service_queue queuerun+0xc
47 0% 94% 1.00 3 0x70239560 timeout_common+0x4
45 0% 95% 1.00 39 0x7028473c background+0xa8
34 0% 95% 1.00 3 0x7028473c putq+0x38
33 0% 95% 1.00 10 0x70239560 callout_execute+0xcc
32 0% 95% 1.00 7 0x7028473c getq_noenab+0x18
31 0% 95% 1.00 2 0x7249b220 ufs_lockfs_end+0x24
30 0% 95% 1.00 12 0x7036b570 timeout_common+0x4
30 0% 95% 1.00 4265 pidlock pr_lookup_procdir+0x9c
26 0% 95% 1.00 2 0x741ed8b0 ufs_getpage+0x86c
25 0% 95% 1.00 2 0x702847c0 putnext+0x88
24 0% 96% 1.00 9 0x70310aa8 callout_execute+0xcc
24 0% 96% 1.00 3 0x702847c0 putnext+0x30
23 0% 96% 1.00 18 0x70388ab8 callout_execute+0xcc
22 0% 96% 1.00 10 0x7036b570 callout_execute+0xcc
20 0% 96% 1.00 8 0x70238020 callout_execute+0xcc
18 0% 96% 1.00 12 0x70388ab8 timeout_common+0x4
18 0% 96% 1.00 16 ph_mutex+0x38 page_lookup_nowait+0x148
17 0% 96% 1.00 11 0x70386038 timeout_common+0x4
17 0% 96% 1.00 9 0x703bb580 timeout_common+0x4
16 0% 96% 1.00 4 0x70238020 untimeout+0x18
16 0% 96% 1.00 6 0x7030e028 callout_execute+0xcc
16 0% 96% 1.00 16 pr_pidlock prinactive+0x38
14 0% 96% 1.00 3 0x70310aa8 untimeout+0x18
14 0% 96% 1.00 4 swapfs_lock swapfs_getvp+0x8
14 0% 96% 1.00 1004 pidlock pr_readdir_procdir+0xc4
13 0% 97% 1.00 6 0x7030e028 untimeout+0x18
13 0% 97% 1.00 7 0x70386038 callout_execute+0xcc
13 0% 97% 1.00 5 0x70310aa8 timeout_common+0x4
13 0% 97% 1.00 22 0x7036b570 untimeout+0x18
12 0% 97% 1.00 1 0x70238020 timeout_common+0x4
12 0% 97% 1.00 2 fnode_cache+0xb4 ufs_lockfs_end+0x24
11 0% 97% 1.00 2 0x7030e028 timeout_common+0x4
9 0% 97% 1.00 26 0x70387578 timeout_common+0x4
8 0% 97% 1.00 9 ph_mutex+0x38 page_create_va+0x698
8 0% 97% 1.00 2129 pidlock waitid+0x90
8 0% 97% 1.00 13 0x70387578 untimeout+0x18
8 0% 97% 1.00 2 0x7023aaa0 untimeout+0x18
8 0% 97% 1.00 1 fnode_cache+0xb4
ufs_lockfs_begin_getpage+0x38
8 0% 97% 1.00 1 0x7249b220 ufs_lockfs_begin+0x38
8 0% 97% 1.00 2 0x702847c0 entersq+0x10
7 0% 97% 1.00 22 service_queue remove_runlist+0x28
7 0% 97% 1.00 8 0x70388ab8 untimeout+0x18
7 0% 97% 1.00 10 0x7023aaa0 callout_execute+0xcc
6 0% 97% 1.00 1 0x7028473c runservice+0x50
6 0% 97% 1.00 73 pidlock thread_exit+0x58
5 0% 97% 1.00 11 0x703bb580 callout_execute+0xcc
5 0% 97% 1.00 6 0x7023aaa0 timeout_common+0x4
5 0% 97% 1.00 1 0x703bb580 untimeout+0x18
5 0% 97% 1.00 38 fpc_mutex+0x28 page_list_add+0xd0
5 0% 97% 1.00 1 anoninfo_lock anon_unresv+0x30
5 0% 97% 1.00 5376 pidlock thread_free+0x1b4
4 0% 97% 1.00 6 0x70410898 getblk_common+0x84
4 0% 97% 1.00 30 0x75062dc0 fifo_write+0x5c
4 0% 97% 1.00 19 0x7036a030 timeout_common+0x4
4 0% 97% 1.00 2 0x72a66620 vn_rele+0x24
4 0% 97% 1.00 21 0x72cdf280
bunyip_name_addcomponent+0x1d0
4 0% 97% 1.00 3 hblk1_lock
sfmmu_hblk_tofreelist+0xb8
4 0% 97% 1.00 48 fpc_mutex+0x20 page_list_add+0xd0
4 0% 97% 1.00 1 anoninfo_lock anon_resvmem+0x2c
4 0% 97% 1.00 5 0x7030e028 callout_execute+0x4
4 0% 97% 1.00 7 0x70387578 callout_execute+0xcc
4 0% 97% 1.00 1 fnode_cache+0xb4 ufs_lockfs_begin+0x38
4 0% 97% 1.00 7261 pidlock cv_wait_sig_swap+0x1d8
4 0% 97% 1.00 16 0x706edf0c segmap_getmapflt+0x338
4 0% 97% 1.00 1 0x702847c0 leavesq+0xc
3 0% 97% 1.00 44 fpc_mutex+0x18 page_list_add+0xd0
3 0% 98% 1.00 45 0x70410898 brelse+0xcc
3 0% 98% 1.00 34 0x814c7510 background+0xa8
3 0% 98% 1.00 11 0x75062140 cv_wait_sig_swap+0x1d8
3 0% 98% 1.00 12 fpc_mutex+0x40
page_get_mnode_freelist+0x140
3 0% 98% 1.00 21 0x7643bcf0 background+0xa8
3 0% 98% 1.00 21 0x75062dc0 cv_wait_sig_swap+0x1d8
3 0% 98% 1.00 765 shm_lock shmget+0x10
3 0% 98% 1.00 2 0x70535ec0 vn_rele+0x24
3 0% 98% 1.00 31 fpc_mutex+0x68
page_get_mnode_freelist+0x140
3 0% 98% 1.00 21 0x8091c450 background+0xa8
3 0% 98% 1.00 24 0x706edf2c segmap_getmapflt+0x338
3 0% 98% 1.00 2 plocks+0x2a0 polldel+0x28
3 0% 98% 1.00 19 0x75ca3998 background+0xa8
3 0% 98% 1.00 40 fpc_mutex+0x48 page_list_add+0xd0
3 0% 98% 1.00 1 hblk1_lock sfmmu_hblk_alloc+0x1ac
3 0% 98% 1.00 5 0x78f5e950 background+0xa8
3 0% 98% 1.00 38 0x72ce1b60
bunyip_name_addcomponent+0x1d0
3 0% 98% 1.00 42 0x72cdab20
bunyip_name_addcomponent+0x1d0
3 0% 98% 1.00 3 0x7190d898 dnlc_lookup+0x9c
3 0% 98% 1.00 2097 pidlock strioctl+0x27cc
3 0% 98% 1.00 9 tcp_mi_lock tcp_snmp_get+0x1d4
3 0% 98% 1.00 13 0x71684bf0 times+0x8
3 0% 98% 1.00 70 volio_spin volkiodone+0x84
3 0% 98% 1.00 43 0x79962bd0 background+0xa8
3 0% 98% 1.00 9 0x70386038 untimeout+0x18
3 0% 98% 1.00 9722 pidlock thread_create+0x250
3 0% 98% 1.00 13 0x71682e50 times+0x8
3 0% 98% 1.00 20 0x76783810 background+0xa8
3 0% 98% 1.00 68 0x79d39000 background+0xa8
3 0% 98% 1.00 1 pse_mutex+0x298 page_trylock+0x18
3 0% 98% 1.00 6 0x70310aa8 callout_execute+0x4
3 0% 98% 1.00 3016 pidlock setpgrp+0xa8
3 0% 98% 1.00 8 0x703bb580 callout_thread+0x38
3 0% 98% 1.00 2 pse_mutex+0x190 page_trylock+0x18
2 0% 98% 1.00 23 door_knob door_call+0x358
2 0% 98% 1.00 49 sfmmu_page_lock+0x1d0 sfmmu_page_enter+0x6c
2 0% 98% 1.00 2 pse_mutex+0x180 page_unlock+0x1c
2 0% 98% 1.00 67 0x7fcc9ba0 fifo_close+0x70
2 0% 98% 1.00 52 0x7479e820 fifo_write+0x5c
2 0% 98% 1.00 14 anonhash_lock+0x98 anon_dup+0x7c
2 0% 98% 1.00 9 0x75062a00 fifo_read+0x24
2 0% 98% 1.00 64 0x75062a00 fifo_write+0x5c
2 0% 98% 1.00 26 0x7fad4550 background+0xa8
2 0% 98% 1.00 8 0x7036a030 callout_execute+0xcc
2 0% 98% 1.00 6 0x1068a620
sfmmu_tteload_acquire_hashbucket+0x88
2 0% 98% 1.00 29 ctr_mutex+0x8 page_ctr_add+0x30
2 0% 98% 1.00 19 0x70410898 getblk_common+0x184
2 0% 98% 1.00 76 0x7479e1e0 cv_wait_sig_swap+0x1d8
2 0% 98% 1.00 15 0x8136eac8 background+0xa8
2 0% 98% 1.00 24 0x7f3b8800 background+0xa8
2 0% 98% 1.00 11 0x76fb3170 background+0xa8
2 0% 98% 1.00 6 0x735742f8 background+0xa8
2 0% 98% 1.00 28 0x756218e8 background+0xa8
2 0% 98% 1.00 6 0x704e4018 callout_schedule_1+0x4
2 0% 98% 1.00 24 0x105cfb40 sfmmu_mlist_enter+0x84
2 0% 98% 1.00 2 ph_mutex+0x188 page_create_va+0x698
2 0% 98% 1.00 1390 pidlock sigqkill+0x60
2 0% 98% 1.00 76 fpc_mutex+0x68 page_list_add+0xd0
2 0% 98% 1.00 36 0x75062a00 fifo_close+0x280
2 0% 98% 1.00 59 0x75062a00 fifo_close+0x70
2 0% 98% 1.00 2 pse_mutex+0x388 page_unlock+0x1c
2 0% 98% 1.00 93 pse_mutex+0x78 page_unlock+0x1c
2 0% 98% 1.00 35 0x7efeb6b8 background+0xa8
2 0% 98% 1.00 7 0x706edf2c segmap_smapadd+0xc
2 0% 98% 1.00 30 ctr_mutex+0x70 page_ctr_add+0x30
2 0% 98% 1.00 16 0x7f376ff8 background+0xa8
2 0% 98% 1.00 7 0x71684680 times+0x8
2 0% 98% 1.00 25 0x71682d08 times+0x8
2 0% 98% 1.00 2 0x741ed8b0 ufs_getattr+0x14c
2 0% 98% 1.00 4 0x741ed8b0 rdip+0x104
2 0% 98% 1.00 12 0x71682918 times+0x8
2 0% 98% 1.00 17 0x7ef57bb0 background+0xa8
2 0% 98% 1.00 6 0x803e1cf0 background+0xa8
2 0% 98% 1.00 55 0x7f413408 background+0xa8
2 0% 98% 1.00 1 ftable_lock fifo_inactive+0xc
2 0% 98% 1.00 32 0x79169568 background+0xa8
2 0% 98% 1.00 39 0x7479e460 fifo_read+0x24
2 0% 98% 1.00 37 0x7479e460 fifo_write+0x5c
2 0% 98% 1.00 4 smap_mtx+0x90 segmap_getmapflt+0x2b8
2 0% 98% 1.00 1 anonhash_lock+0x98 anon_decref+0x48
2 0% 98% 1.00 1 pse_mutex+0x348 page_trylock+0x18
2 0% 98% 1.00 53 0x72ede060 fifo_write+0x5c
2 0% 98% 1.00 7 0x72a66620 dnlc_lookup+0x12c
2 0% 98% 1.00 16 0x72a66620
bunyip_name_addcomponent+0xb0
2 0% 98% 1.00 9 0x7190cb48 dnlc_lookup+0x9c
2 0% 98% 1.00 44 0x72a66620 traverse+0x8
2 0% 98% 1.00 27 mod_lock mod_release_stub+0x14
2 0% 98% 1.00 3749 pidlock exit+0x414
2 0% 98% 1.00 44 0x1068d1e8 hat_unload+0x16c
2 0% 98% 1.00 42 0x7479e1e0 fifo_write+0x5c
2 0% 98% 1.00 103 0x72e906c0 fifo_close+0x70
2 0% 98% 1.00 35 0x72f6ada8 background+0xa8
2 0% 98% 1.00 18 0x71684a50 times+0x8
2 0% 98% 1.00 2 0x75c27f70 background+0xa8
2 0% 98% 1.00 20 0x7557add8 background+0xa8
2 0% 98% 1.00 1 pse_mutex+0x30
page_try_reclaim_lock+0x18
2 0% 98% 1.00 59 0x71681658 times+0x8
2 0% 98% 1.00 26 0x803e01f0 background+0xa8
2 0% 98% 1.00 29 0x7f412608 background+0xa8
2 0% 98% 1.00 14 0x75bcd468 background+0xa8
2 0% 98% 1.00 15 0x72e90080 fifo_read+0x24
2 0% 98% 1.00 34 0x72e90080 fifo_write+0x5c
2 0% 98% 1.00 2 0x70239560 callout_execute+0x4
2 0% 98% 1.00 2 0x706edf0c segmap_smapadd+0xc
2 0% 98% 1.00 2 pse_mutex+0x48 page_unlock+0x1c
2 0% 98% 1.00 11490 pidlock getproc+0xd4
2 0% 98% 1.00 28 0x796620b0 background+0xa8
2 0% 98% 1.00 17 0x7fea8ab0 background+0xa8
2 0% 98% 1.00 5 0x71683a58 times+0x8
2 0% 98% 1.00 8 0x755f15e0 background+0xa8
2 0% 98% 1.00 12 0x71684128 xsetitimer+0xec
2 0% 98% 1.00 31 0x7a9788b0 background+0xa8
2 0% 98% 1.00 14312 pidlock exit+0x2f8
2 0% 98% 1.00 64 0x70388ab8 callout_thread+0x38
2 0% 98% 1.00 16 0x71683b08 xsetitimer+0xec
2 0% 98% 1.00 18 0x7036a030 untimeout+0x18
2 0% 98% 1.00 15 0x7036b570 callout_schedule_1+0x4
2 0% 98% 1.00 122 0x7fcc9ba0 fifo_write+0x5c
2 0% 98% 1.00 45 0x782426b8 background+0xa8
2 0% 99% 1.00 14 0x73c38f28 background+0xa8
2 0% 99% 1.00 26 0x71681cb0 times+0x8
2 0% 99% 1.00 3 0x7253ba30 ufs_getpage+0x86c
2 0% 99% 1.00 17 0x802b62d8 background+0xa8
2 0% 99% 1.00 62 0x7fcc92e0 fifo_close+0x70
2 0% 99% 1.00 9 0x7479e820 fifo_close+0x70
1 0% 99% 1.00 62 door_knob door_call+0xa4
1 0% 99% 1.00 22 door_knob door_return+0xa4
1 0% 99% 1.00 2 plocks+0x350 polldel+0x28
1 0% 99% 1.00 2 door_knob door_info+0x74
1 0% 99% 1.00 40 0x7fcc9ba0 fifo_close+0x280
1 0% 99% 1.00 1 0x70535ec0 lookuppn+0xc4
1 0% 99% 1.00 29 0x754a6a18 background+0xa8
1 0% 99% 1.00 17 0x105cf448 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 2 pse_mutex+0x3c0
page_try_reclaim_lock+0x18
1 0% 99% 1.00 31 0x71682ca8 times+0x8
1 0% 99% 1.00 85 0x71681388 times+0x8
1 0% 99% 1.00 125 0x7f62c120 background+0xa8
1 0% 99% 1.00 23 0x756992f0 background+0xa8
1 0% 99% 1.00 2 0x803418e0 background+0xa8
1 0% 99% 1.00 22 0x74920c20 background+0xa8
1 0% 99% 1.00 25 0x782db92c strrput+0x390
1 0% 99% 1.00 2 0x71684660 sigaction+0xc4
1 0% 99% 1.00 2 0x7279ad20 ufs_getpage+0x86c
1 0% 99% 1.00 2 0x782db92c strclean+0x10
1 0% 99% 1.00 20 0x7557aed8 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x3f8 page_unlock+0x1c
1 0% 99% 1.00 2 0x71680c80 xsetitimer+0xec
1 0% 99% 1.00 2 0x71683938 times+0x8
1 0% 99% 1.00 1 pse_mutex+0x288
page_try_reclaim_lock+0x18
1 0% 99% 1.00 19 smap_mtx+0x98 segmap_getmapflt+0x2b8
1 0% 99% 1.00 2 pse_mutex+0x1d0 page_unlock+0x1c
1 0% 99% 1.00 2 fpc_mutex+0x8
page_get_mnode_freelist+0x140
1 0% 99% 1.00 2 fpc_mutex+0x20
page_get_mnode_freelist+0x140
1 0% 99% 1.00 30 fpc_mutex+0x30 page_list_add+0xd0
1 0% 99% 1.00 1 pse_mutex+0x238 page_unlock+0x1c
1 0% 99% 1.00 48 0x71680308 times+0x8
1 0% 99% 1.00 1 0x73005e38 ufs_getpage+0x86c
1 0% 99% 1.00 2 anonhash_lock+0x18 anon_decref+0x48
1 0% 99% 1.00 37 anonhash_lock+0x118 anon_decref+0x48
1 0% 99% 1.00 2 0x73834610
bunyip_name_addcomponent+0xb0
1 0% 99% 1.00 1 0x75856920 queuerun+0x198
1 0% 99% 1.00 66 0x801247d0 background+0xa8
1 0% 99% 1.00 2 0x71681900 times+0x8
1 0% 99% 1.00 20 0x7567239c strrput+0x4c0
1 0% 99% 1.00 3 0x7567239c strclean+0x10
1 0% 99% 1.00 1 0x804c9608 background+0xa8
1 0% 99% 1.00 30 0x7af3e4d8 background+0xa8
1 0% 99% 1.00 29 0x71684bd0 xsetitimer+0xec
1 0% 99% 1.00 8 0x71684530 times+0x8
1 0% 99% 1.00 8 0x70386038 callout_schedule_1+0x4
1 0% 99% 1.00 48 0x80897248 background+0xa8
1 0% 99% 1.00 25 0x80eeba70 background+0xa8
1 0% 99% 1.00 15 0x750891c0 background+0xa8
1 0% 99% 1.00 5 0x7f412708 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x320 page_unlock+0x1c
1 0% 99% 1.00 3 0x716846a8 xsetitimer+0xec
1 0% 99% 1.00 2 pse_mutex+0x3e8 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x2f8
page_try_reclaim_lock+0x18
1 0% 99% 1.00 30 0x79903bc8 background+0xa8
1 0% 99% 1.00 1 pse_mutex+0x308 page_trylock+0x18
1 0% 99% 1.00 2 0x7410dbf0 putq+0x38
1 0% 99% 1.00 2 0x71682718 times+0x8
1 0% 99% 1.00 32 0x716801d8 times+0x8
1 0% 99% 1.00 36 0x7fc33468 background+0xa8
1 0% 99% 1.00 46 0x8050e628 background+0xa8
1 0% 99% 1.00 23 0x80e5dc68 background+0xa8
1 0% 99% 1.00 32 0x763e15e8 background+0xa8
1 0% 99% 1.00 106 0x80e5c968 background+0xa8
1 0% 99% 1.00 39 0x7382eb18 background+0xa8
1 0% 99% 1.00 4 0x71681b08 clock+0x2b0
1 0% 99% 1.00 12 0x71682490 times+0x8
1 0% 99% 1.00 2 0x105cd360 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 27 0x75ce45a0 background+0xa8
1 0% 99% 1.00 1 0x741ed4c0 ufs_getpage+0x86c
1 0% 99% 1.00 8 0x716840b8 xsetitimer+0xec
1 0% 99% 1.00 31 0x71682720 times+0x8
1 0% 99% 1.00 11 0x1068a620 hat_unlock+0xfc
1 0% 99% 1.00 36 0x106e8fb8 hat_unload+0x16c
1 0% 99% 1.00 8 0x72af1738 traverse+0xa8
1 0% 99% 1.00 3 ctr_mutex+0x8 page_ctr_sub+0x30
1 0% 99% 1.00 81 fpc_mutex+0x48
page_get_mnode_freelist+0x140
1 0% 99% 1.00 2 ctr_mutex+0x48 page_ctr_add+0x30
1 0% 99% 1.00 1 pse_mutex+0x1d0 page_trylock+0x18
1 0% 99% 1.00 2 fpc_mutex+0x30
page_get_mnode_freelist+0x140
1 0% 99% 1.00 28 fpc_mutex+0x28
page_get_mnode_freelist+0x140
1 0% 99% 1.00 2 0x105d5598 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 50 0x7fc8da70 background+0xa8
1 0% 99% 1.00 16 0x71682328 times+0x8
1 0% 99% 1.00 2 0x71682328 xsetitimer+0xec
1 0% 99% 1.00 25 0x75062140 fifo_write+0x5c
1 0% 99% 1.00 21 0x7502b7b0 background+0xa8
1 0% 99% 1.00 4 0x75ca3198 background+0xa8
1 0% 99% 1.00 57 0x787327f0 background+0xa8
1 0% 99% 1.00 19 0x79b3d3e8 background+0xa8
1 0% 99% 1.00 1 0x71683ba8 times+0x8
1 0% 99% 1.00 130 0x71684ef0 times+0x8
1 0% 99% 1.00 2 0x71682908 times+0x8
1 0% 99% 1.00 42 0x75062000 fifo_close+0x280
1 0% 99% 1.00 1 0x73834a00 vn_rele+0x24
1 0% 99% 1.00 45 0x71683518 times+0x8
1 0% 99% 1.00 22 0x70239560 callout_schedule_1+0x4
1 0% 99% 1.00 22 0x71681980 times+0x8
1 0% 99% 1.00 16 0x716840b8 times+0x8
1 0% 99% 1.00 2 0x70289f90 qwriter_inner+0x8
1 0% 99% 1.00 52 0x7f0767c8 background+0xa8
1 0% 99% 1.00 62 0x716846f0 times+0x8
1 0% 99% 1.00 37 0x71680ad8 times+0x8
1 0% 99% 1.00 6 0x70238020 callout_execute+0x4
1 0% 99% 1.00 28 0x7023d020 dmpstrategy+0x58
1 0% 99% 1.00 32 0x7efeb0b8 background+0xa8
1 0% 99% 1.00 16 0x71681cb0 xsetitimer+0xec
1 0% 99% 1.00 12 0x724642c0 fifo_close+0x70
1 0% 99% 1.00 10 0x75062dc0 fifo_close+0x280
1 0% 99% 1.00 2 0x7289fc58 background+0xa8
1 0% 99% 1.00 27 0x812a64b8 background+0xa8
1 0% 99% 1.00 30 0x7a25c750 background+0xa8
1 0% 99% 1.00 2 0x756b39f8 background+0xa8
1 0% 99% 1.00 2 0x75bcd268 background+0xa8
1 0% 99% 1.00 64 0x716826c8 xsetitimer+0xec
1 0% 99% 1.00 21 0x71682ef8 times+0x8
1 0% 99% 1.00 43 0x71684030 times+0x8
1 0% 99% 1.00 2 0x71684e00 issig_forreal+0x18
1 0% 99% 1.00 39 0x75062140 fifo_close+0x280
1 0% 99% 1.00 30 fpc_mutex+0x78 page_list_add+0xd0
1 0% 99% 1.00 2 0x105d4488 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 2 volio_spin voliod_loop+0x174
1 0% 99% 1.00 1 anonhash_lock+0x1c8 anon_decref+0x48
1 0% 99% 1.00 10 0x7fcc96a0 cv_wait_sig_swap+0x1d8
1 0% 99% 1.00 3 0x71681898 sigaction+0xc4
1 0% 99% 1.00 22 0x7f181e10 leavesq+0xc
1 0% 99% 1.00 13 0x7191c898 dnlc_lookup+0x9c
1 0% 99% 1.00 1 0x72a66690 ufs_getattr+0x14c
1 0% 99% 1.00 15 0x105d3c48 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 51 0x719690d8 dnlc_lookup+0x9c
1 0% 99% 1.00 1 pse_mutex+0x1b0 page_trylock+0x18
1 0% 99% 1.00 2 0x7036a030 callout_thread+0x38
1 0% 99% 1.00 15 0x80565730 background+0xa8
1 0% 99% 1.00 53 0x750063a0 background+0xa8
1 0% 99% 1.00 52 0x8139d6d8 background+0xa8
1 0% 99% 1.00 55 0x7f181de8 background+0xa8
1 0% 99% 1.00 8 0x75057ab8 background+0xa8
1 0% 99% 1.00 2 0x7502b2b0 background+0xa8
1 0% 99% 1.00 18 0x71684530 xsetitimer+0xec
1 0% 99% 1.00 18 0x7479fd60 cv_wait_sig_swap+0x1d8
1 0% 99% 1.00 158 0x106a41e0
sfmmu_shadow_hcreate+0xf8
1 0% 99% 1.00 2 pse_mutex+0x2a0
page_try_reclaim_lock+0x18
1 0% 99% 1.00 2 0x71681980 xsetitimer+0xec
1 0% 99% 1.00 2 pse_mutex+0x1d8 page_unlock+0x1c
1 0% 99% 1.00 3 0x70388ab8 callout_schedule_1+0x4
1 0% 99% 1.00 2 pse_mutex+0x20 page_unlock+0x1c
1 0% 99% 1.00 6 pse_mutex+0x30 page_unlock+0x1c
1 0% 99% 1.00 1 pse_mutex+0x150 page_unlock+0x1c
1 0% 99% 1.00 13 0x7479f4a0 fifo_close+0x70
1 0% 99% 1.00 22 0x724642c0 fifo_read+0x24
1 0% 99% 1.00 3 0x724642c0 fifo_write+0x5c
1 0% 99% 1.00 1 pse_mutex+0x1c8
page_try_reclaim_lock+0x18
1 0% 99% 1.00 9 0x791db670 background+0xa8
1 0% 99% 1.00 10 0x79600fa8 background+0xa8
1 0% 99% 1.00 20 0x8142f500 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x170 page_unlock+0x1c
1 0% 99% 1.00 2 0x71684be8 sigaction+0xc4
1 0% 99% 1.00 22 0x7be5b240 background+0xa8
1 0% 99% 1.00 7 pse_mutex+0x228 page_unlock+0x1c
1 0% 99% 1.00 1 0x716850b0 times+0x8
1 0% 99% 1.00 59 0x716850b0 prinactive+0x64
1 0% 99% 1.00 1 pse_mutex+0xf0 page_trylock+0x18
1 0% 99% 1.00 2 pse_mutex+0xf8 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x170 page_trylock+0x18
1 0% 99% 1.00 2 pse_mutex+0x338 page_unlock+0x1c
1 0% 99% 1.00 2 pse_mutex+0xc8
page_try_reclaim_lock+0x18
1 0% 99% 1.00 29 0x71680cc0 xsetitimer+0xec
1 0% 99% 1.00 8 0x71684168 pr_read_psinfo+0x38
1 0% 99% 1.00 42 0x71681678 times+0x8
1 0% 99% 1.00 2 0x105cda60 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 2 pr_pidlock prfree+0x20
1 0% 99% 1.00 2 0x105d13a0 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 17 pidlock strioctl+0x2a6c
1 0% 99% 1.00 1 0x75831718 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x358 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x230 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x388
page_try_reclaim_lock+0x18
1 0% 99% 1.00 1 pse_mutex+0x148 page_hashout+0x160
1 0% 99% 1.00 7 0x71682450 prinactive+0x64
1 0% 99% 1.00 2 pse_mutex+0x78 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x1a0
page_try_reclaim_lock+0x18
1 0% 99% 1.00 2 pse_mutex+0xe0 page_unlock+0x1c
1 0% 99% 1.00 31 0x75ea88b0 background+0xa8
1 0% 99% 1.00 113 0x80897648 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x2a8 page_hashout+0x160
1 0% 99% 1.00 2 pse_mutex+0x198 page_unlock+0x1c
1 0% 99% 1.00 2 pse_mutex+0x338 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x118 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x398 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x3f0 page_trylock+0x18
1 0% 99% 1.00 2 pse_mutex+0x280
page_try_reclaim_lock+0x18
1 0% 99% 1.00 2 pse_mutex+0x380 page_trylock+0x18
1 0% 99% 1.00 8 0x80ffed98 background+0xa8
1 0% 99% 1.00 2 pse_mutex+0x290 page_trylock+0x18
1 0% 99% 1.00 2 pse_mutex+0x70 page_unlock+0x1c
1 0% 99% 1.00 2 pse_mutex+0x138 page_unlock+0x1c
1 0% 99% 1.00 2 pse_mutex+0x3f8 page_trylock+0x18
1 0% 99% 1.00 1 pse_mutex+0x50 page_trylock+0x18
1 0% 99% 1.00 68 0x760150c0 background+0xa8
1 0% 99% 1.00 38 0x7af3fad8 background+0xa8
1 0% 99% 1.00 8 0x80660d10 background+0xa8
1 0% 99% 1.00 2 0x105d0120 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 7 0x105d7c40 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 1 pse_mutex+0x288 page_trylock+0x18
1 0% 99% 1.00 2 0x7fcc96a0 fifo_close+0x70
1 0% 99% 1.00 2 0x7c6ff0cc strclean+0x10
1 0% 99% 1.00 3 0x7908c660 background+0xa8
1 0% 99% 1.00 21 0x750892c0 background+0xa8
1 0% 99% 1.00 63 0x105d7a88 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 29 0x716833a8 times+0x8
1 0% 99% 1.00 2 fpc_mutex+0x58
page_get_mnode_freelist+0x140
1 0% 99% 1.00 2 smap_mtx+0x90 segmap_release+0xb0
1 0% 99% 1.00 2 0x105cddc8 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 2 0x105d0260 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 1 0x73ee5b28 dnlc_lookup+0x12c
1 0% 99% 1.00 17 0x105d6db8 sfmmu_mlist_enter+0x84
1 0% 99% 1.00 1 0x71682d08 clock+0x2b0
1 0% 99% 1.00 2 0x7253ba30 ufs_addmap+0x20
1 0% 99% 1.00 46 0x750104a8 background+0xa8
1 0% 99% 1.00 20 0x78e4c640 background+0xa8
1 0% 99% 1.00 1 0x71683a60 times+0x8
1 0% 99% 1.00 13 ctr_mutex+0x70 page_ctr_sub+0x30
1 0% 99% 1.00 2 plocks+0x280 polldel+0x28
1 0% 99% 1.00 9 0x71682650 xsetitimer+0xec
1 0% 99% 1.00 22 0x72cdd950
bunyip_name_addcomponent+0x1d0
1 0% 99% 1.00 47 0x7f3766f8 background+0xa8
1 0% 99% 1.00 19 0x7f86d830 background+0xa8
1 0% 99% 1.00 13 0x796639b0 background+0xa8
1 0% 99% 1.00 13 0x703bb580 callout_schedule_1+0x4
1 0% 99% 1.00 10 0x7f412608 queuerun+0x198
1 0% 99% 1.00 8 fpc_mutex+0x60 page_list_add+0xd0
1 0% 99% 1.00 33 0x79b3d5e8 background+0xa8
1 0% 99% 1.00 59 0x7f91e540 background+0xa8
1 0% 99% 1.00 14 0x71684dc8 times+0x8
1 0% 99% 1.00 116 0x71681218 times+0x8
1 0% 99% 1.00 18 0x716851c0 clock+0x2b0
1 0% 99% 1.00 48 0x71684390 clock+0x2b0
1 0% 99% 1.00 9 0x75856220 queuerun+0x198
1 0% 99% 1.00 79 0x716824a0 times+0x8
1 0% 99% 1.00 16 0x7fe72fa8 background+0xa8
1 0% 99% 1.00 56 0x71681ba0 times+0x8
1 0% 99% 1.00 42 0x78d78f38 background+0xa8
1 0% 99% 1.00 21 0x716850a0 times+0x8
1 0% 99% 1.00 57 0x7ca677e8 queuerun+0x198
1 0% 99% 1.00 2 0x7fffc5c0 queuerun+0x198
1 0% 99% 1.00 51 0x761bfbd0 background+0xa8
1 0% 99% 1.00 95 0x716840d0 times+0x8
1 0% 99% 1.00 2 0x80ffedc0 rwnext+0x248
1 0% 99% 1.00 2 0x7f86c530 background+0xa8
1 0% 99% 1.00 19 0x75063cc0 cv_wait_sig_swap+0x1d8
1 0% 99% 1.00 5 0x7023aaa0 callout_execute+0x4
1 0% 99% 1.00 2 0x807dc368 drain_syncq+0x250
1 0% 99% 1.00 20 0x75062dc0 fifo_read+0x24
1 0% 99% 1.00 93 0x716843a0 times+0x8
1 0% 99% 1.00 100 0x789e7b10 background+0xa8
1 0% 99% 1.00 34 0x716824a0 xsetitimer+0xec
1 0% 99% 1.00 1 plocks+0x380 polldel+0x28
1 0% 99% 1.00 2 fpc_mutex+0x70
page_get_mnode_freelist+0x140
1 0% 99% 1.00 23 0x71681f78 clock+0x2b0
1 0% 99% 1.00 2 pse_mutex+0x300 page_hashout+0x160
1 0% 100% 1.00 13 0x71684050 times+0x8
1 0% 100% 1.00 70 0x72ede060 fifo_close+0x280
1 0% 100% 1.00 93 0x75062c80 fifo_close+0x280
1 0% 100% 1.00 47 0x75062c80 cv_wait_sig_swap+0x1d8
1 0% 100% 1.00 26 0x71681d60 times+0x8
1 0% 100% 1.00 98 0x7479e460 fifo_close+0x280
1 0% 100% 1.00 116 0x7479e460 fifo_close+0x70
1 0% 100% 1.00 137 0x7fcc9ba0 cv_wait_sig_swap+0x1d8
1 0% 100% 1.00 2 0x7fc0a43c strclean+0x10
1 0% 100% 1.00 14 0x71684e00 clock+0x2b0
1 0% 100% 1.00 1 pse_mutex+0x188 page_trylock+0x18
1 0% 100% 1.00 52 0x7830a6c0 background+0xa8
1 0% 100% 1.00 36 ctr_mutex+0x20 page_ctr_add+0x30
1 0% 100% 1.00 2 0x71684bc8 times+0x8
1 0% 100% 1.00 3 0x716833e8 times+0x8
1 0% 100% 1.00 2 0x79a77de0 background+0xa8
1 0% 100% 1.00 2 0x73896148 ufs_getpage+0x86c
1 0% 100% 1.00 63 0x71683010 times+0x8
1 0% 100% 1.00 2 0x73e72690 ufs_getpage+0x86c
1 0% 100% 1.00 52 0x764d84f8 background+0xa8
1 0% 100% 1.00 46 0x74b67410 background+0xa8
1 0% 100% 1.00 34 0x71682fd8 times+0x8
1 0% 100% 1.00 49 0x72ede060 fifo_read+0x24
1 0% 100% 1.00 23 0x71683c90 xsetitimer+0xec
1 0% 100% 1.00 1783 0x7303ac18 pollwakeup_safe+0x1ac
1 0% 100% 1.00 2 plocks+0x380 pollwakeup_safe+0x3c
1 0% 100% 1.00 2 0x803e1cf0 queuerun+0x198
1 0% 100% 1.00 58 0x79a772e0 background+0xa8
1 0% 100% 1.00 27 ph_mutex+0xf0 page_create_va+0x698
1 0% 100% 1.00 9 0x706edf6c segmap_smapadd+0xc
1 0% 100% 1.00 6 0x71684ef0 xsetitimer+0xec
1 0% 100% 1.00 61 0x71681280 times+0x8
1 0% 100% 1.00 2 0x73834760 vn_rele+0x24
1 0% 100% 1.00 2 0x738347d0 ufs_getattr+0x14c
1 0% 100% 1.00 9 shash_mtx+0x148 segmap_getmapflt+0x108
1 0% 100% 1.00 46 0x10685868 hat_unlock+0xfc
1 0% 100% 1.00 86 fpc_mutex+0x8 page_list_add+0xd0
1 0% 100% 1.00 8 ph_mutex+0x1a8 page_lookup+0x230
1 0% 100% 1.00 2 sfmmu_page_lock+0x1f8 sfmmu_page_enter+0x6c
1 0% 100% 1.00 27 anonhash_lock+0xa8 anon_decref+0x48
1 0% 100% 1.00 1 pcf+0x308 page_create_va+0x948
1 0% 100% 1.00 1 0x105d7dc0 sfmmu_mlist_enter+0x84
1 0% 100% 1.00 2 fpc_mutex+0x38
page_get_mnode_freelist+0x140
1 0% 100% 1.00 81 0x799f20d8 background+0xa8
1 0% 100% 1.00 45 0x71680d70 times+0x8
1 0% 100% 1.00 1 sfmmu_page_lock+0x20 sfmmu_page_enter+0x6c
1 0% 100% 1.00 19 0x71683b08 times+0x8
1 0% 100% 1.00 46 0x71682628 xsetitimer+0x160
1 0% 100% 1.00 54 0x81264ab0 background+0xa8
1 0% 100% 1.00 2 0x70310aa8 callout_schedule_1+0x4
1 0% 100% 1.00 62 0x71684500 times+0x8
1 0% 100% 1.00 45 0x7023aaa0 callout_schedule_1+0x4
1 0% 100% 1.00 10 0x75ca3098 background+0xa8
1 0% 100% 1.00 33 0x71682a38 times+0x8
1 0% 100% 1.00 31 0x7a177740 background+0xa8
1 0% 100% 1.00 74 0x71682dd8 times+0x8
1 0% 100% 1.00 36 0x105d3af0 sfmmu_mlist_enter+0x84
1 0% 100% 1.00 1 0x75062000 fifo_close+0x70
1 0% 100% 1.00 40 0x71684dc8 xsetitimer+0xec
1 0% 100% 1.00 34 0x784264d0 background+0xa8
1 0% 100% 1.00 420 0x1068e808 sfmmu_hblk_alloc+0x274
1 0% 100% 1.00 4 0x1068e808 hat_unload+0x16c
1 0% 100% 1.00 6 0x105d0658 sfmmu_mlist_enter+0x84
1 0% 100% 1.00 2 anonhash_lock+0x1e0 anon_decref+0x48
1 0% 100% 1.00 5 0x785210e0 background+0xa8
1 0% 100% 1.00 45 0x75c26170 background+0xa8
1 0% 100% 1.00 29 0x813ca4e8 background+0xa8
1 0% 100% 1.00 49 0x71681258 times+0x8
1 0% 100% 1.00 2 0x71682bc8 times+0x8
1 0% 100% 1.00 24 0x71680ed0 times+0x8
1 0% 100% 1.00 1 0x7028443c background+0xa8
1 0% 100% 1.00 3 0x71684e00 xsetitimer+0xec
1 0% 100% 1.00 2 0x75c32a90 background+0xa8
1 0% 100% 1.00 40 0x716841b8 times+0x8
1 0% 100% 1.00 2 0x71b355b8 socal_cq_enque+0x40
1 0% 100% 1.00 26 0x7b38ebf8 background+0xa8
1 0% 100% 1.00 27 0x71682598 times+0x8
1 0% 100% 1.00 20 0x71682598 xsetitimer+0xec
1 0% 100% 1.00 57 0x799636d0 background+0xa8
1 0% 100% 1.00 2 volio_spin voldiskiodone+0x2e0
1 0% 100% 1.00 36 0x803fc800 background+0xa8
1 0% 100% 1.00 8 0x756996f0 background+0xa8
1 0% 100% 1.00 2 0x71b02010 iommu_dma_bindhdl+0x250
1 0% 100% 1.00 17 0x7e414a50 background+0xa8
1 0% 100% 1.00 2 pio_mutex+0x2a8 page_io_unlock+0x18
1 0% 100% 1.00 11 pse_mutex+0x2a8 page_trylock+0x18
1 0% 100% 1.00 59 0x71683058 xsetitimer+0xec
1 0% 100% 1.00 13 0x105cf4f0 sfmmu_mlist_enter+0x84
1 0% 100% 1.00 8 0x79bb50f0 background+0xa8
1 0% 100% 1.00 2 0x79d9c808 background+0xa8
1 0% 100% 1.00 2 0x71684298 times+0x8
1 0% 100% 1.00 8 0x7f3b8900 background+0xa8
1 0% 100% 1.00 22 0x8141b56c putnext+0xc
1 0% 100% 1.00 32 0x7ac77560 background+0xa8
1 0% 100% 1.00 61 0x7f503a10 background+0xa8
1 0% 100% 1.00 2 0x71684388 xsetitimer+0xec
1 0% 100% 1.00 20 pidlock strioctl+0x29cc
1 0% 100% 1.00 20 0x7036b570 callout_thread+0x38
1 0% 100% 1.00 193 0x71684128 xsetitimer+0x160
1 0% 100% 1.00 1 0x71684128 pr_p_lock+0x38
1 0% 100% 1.00 2 0x71684cb8 times+0x8
1 0% 100% 1.00 2 0x71682e10 nanosleep+0x128
1 0% 100% 1.00 90 volio_spin voliod_iohandle+0x88
1 0% 100% 1.00 47 0x71683b08 pr_p_lock+0x38
1 0% 100% 1.00 8 0x71680e40 times+0x8
1 0% 100% 1.00 42 0x7f3764f8 background+0xa8
1 0% 100% 1.00 37 0x71681c80 times+0x8
1 0% 100% 1.00 6 plocks+0x340 prpoll+0xe0
1 0% 100% 1.00 204 0x7fcc97e0 fifo_close+0x70
1 0% 100% 1.00 1 sfmmu_page_lock+0xf0 sfmmu_page_enter+0x6c
1 0% 100% 1.00 75 0x737e0210 background+0xa8
1 0% 100% 1.00 12 0x756215e8 background+0xa8
1 0% 100% 1.00 47 0x76de2c58 background+0xa8
1 0% 100% 1.00 12 0x71682720 clock+0x2b0
1 0% 100% 1.00 2 0x71684228 times+0x8
1 0% 100% 1.00 2 pse_mutex+0xc8 page_trylock+0x18
1 0% 100% 1.00 17 0x7557fd38 background+0xa8
1 0% 100% 1.00 20 0x71681e38 times+0x8
1 0% 100% 1.00 2 0x105cd780 sfmmu_mlist_enter+0x84
1 0% 100% 1.00 2 pse_mutex+0x148 page_unlock+0x1c
1 0% 100% 1.00 45 pidlock sigcld_repost+0x48
1 0% 100% 1.00 48 0x716826c8 times+0x8
1 0% 100% 1.00 2 0x71684668 clock+0x2b0
1 0% 100% 1.00 2 0x706edf4c segmap_smapadd+0xc
1 0% 100% 1.00 44 0x7e520668 background+0xa8
1 0% 100% 1.00 6 ctr_mutex+0x60 page_ctr_sub+0x30
1 0% 100% 1.00 2 0x81392ba4 strrput+0x4c0
1 0% 100% 1.00 20 0x81392ba4 strclean+0x10
1 0% 100% 1.00 29 0x71684f60 xsetitimer+0xec
1 0% 100% 1.00 8 0x71681b48 xsetitimer+0xec
1 0% 100% 1.00 2 pse_mutex+0x100
page_try_reclaim_lock+0x18
1 0% 100% 1.00 57 pidlock sess_create+0x48
1 0% 100% 1.00 49 0x71683a58 xsetitimer+0xec
1 0% 100% 1.00 2 0x7453fba0 background+0xa8
1 0% 100% 1.00 21 0x71682710 times+0x8
1 0% 100% 1.00 2 plocks+0x160 polldel+0x28
-----------------------------------------------------------------------------
--
Adaptive mutex block: 766 events
Count indv cuml rcnt nsec Lock Caller
-----------------------------------------------------------------------------
--
342 45% 45% 1.00 1406000 0x70538008 cv_wait_sig+0x1b8
215 28% 73% 1.00 1396551 0x70538008 semconv_lock+0x60
43 6% 78% 1.00 446313 0x724b2d30 qfestart+0x1f8
25 3% 82% 1.00 95639 service_queue qenable_locked+0x34
24 3% 85% 1.00 128170 service_queue background+0x120
21 3% 87% 1.00 123177 service_queue background+0xdc
20 3% 90% 1.00 73733 0x724b2d30 qfeintr+0x2bc
7 1% 91% 1.00 98611 volio_spin voliod_loop+0x174
6 1% 92% 1.00 316864 0x70538008 cv_wait_sig+0x230
6 1% 93% 1.00 2229996 tod_lock uniqtime+0x14
5 1% 93% 1.00 544026 0x7028731c putnext+0xc
5 1% 94% 1.00 3553192 pidlock pr_lookup_procdir+0x9c
4 1% 94% 1.00 596840 hblk8_lock sfmmu_hblk_alloc+0xdc
3 0% 95% 1.00 670927 0x704e4018 untimeout+0x18
2 0% 95% 1.00 765259 hblk8_lock
sfmmu_hblk_tofreelist+0x38
2 0% 95% 1.00 159438 0x70387578 timeout_common+0x4
2 0% 96% 1.00 29227 0x7028473c putq+0x38
2 0% 96% 1.00 47421 0x70239560 timeout_common+0x4
2 0% 96% 1.00 138366 0x70388ab8 timeout_common+0x4
1 0% 96% 1.00 57997 0x79662fb0 background+0xa8
1 0% 96% 1.00 81407 0x702847c0 putnext+0x30
1 0% 96% 1.00 251520 0x75bcd468 background+0xa8
1 0% 97% 1.00 116846 0x7a05a95c cv_wait_sig+0x1b8
1 0% 97% 1.00 132330 0x75258ed0 background+0xa8
1 0% 97% 1.00 30498 0x71682628 clock+0x2b0
1 0% 97% 1.00 19374 0x71684540 clock+0x2b0
1 0% 97% 1.00 35064 0x78e5c38c cv_wait_sig+0x1b8
1 0% 97% 1.00 96629 0x7249b220 ufs_lockfs_end+0x24
1 0% 97% 1.00 163760 0x799634d0 background+0xa8
1 0% 98% 1.00 681127 0x7190d898 dnlc_lookup+0x9c
1 0% 98% 1.00 302395 0x76783c10 background+0xa8
1 0% 98% 1.00 22604 0x81392ba4 cv_wait_sig+0x1b8
1 0% 98% 1.00 193856716 kstat_chain_lock read_kstat_data+0x34
1 0% 98% 1.00 92620 ph_mutex+0x38 page_create_va+0x698
1 0% 98% 1.00 26383 0x70388ab8 callout_schedule_1+0x4
1 0% 98% 1.00 1210276 0x7fc0b3fc cv_wait_sig+0x1b8
1 0% 98% 1.00 36509 0x71684170 clock+0x2b0
1 0% 99% 1.00 55299 0x756cff00 background+0xa8
1 0% 99% 1.00 177998 0x70239560 untimeout+0x18
1 0% 99% 1.00 22047 0x8141b86c cv_wait_sig+0x1b8
1 0% 99% 1.00 125510 0x702847c0 leavesq+0xc
1 0% 99% 1.00 135051 0x76bb9c40 background+0xa8
1 0% 99% 1.00 60529 0x7249b220
ufs_lockfs_begin_getpage+0x38
1 0% 99% 1.00 85796 0x784a57d8 background+0xa8
1 0% 99% 1.00 24589 0x70289f90 putnext+0x88
1 0% 100% 1.00 73274 lwpchanlock+0x118 lwp_sema_wait+0x3e8
1 0% 100% 1.00 167944 0x75b66460 background+0xa8
1 0% 100% 1.00 128238 0x7479e820 cv_wait_sig_swap+0x1d8
1 0% 100% 1.00 62379 0x704e4018 timeout_common+0x4
-----------------------------------------------------------------------------
--
Spin lock spin: 5559 events
Count indv cuml rcnt spin Lock Caller
-----------------------------------------------------------------------------
--
1063 19% 19% 1.00 546 sleepq_head+0x79c cv_signal+0x24
272 5% 24% 1.00 438 cpu[0]+0x44 disp_getbest+0x4
261 5% 29% 1.00 191 cpu[0]+0x44 setbackdq+0x188
254 5% 33% 1.00 463 cpu[12]+0x44 disp_getbest+0x4
246 4% 38% 1.00 484 cpu[11]+0x44 disp_getbest+0x4
243 4% 42% 1.00 503 cpu[14]+0x44 disp_getbest+0x4
230 4% 46% 1.00 501 cpu[10]+0x44 disp_getbest+0x4
218 4% 50% 1.00 461 cpu[8]+0x44 disp_getbest+0x4
209 4% 54% 1.00 513 cpu[9]+0x44 disp_getbest+0x4
170 3% 57% 1.00 185 0x72b9ff7b
mutex_adaptive_release+0x78
151 3% 60% 1.00 173 cpu[14]+0x44 setbackdq+0x188
149 3% 62% 1.00 629 0x72b9ff3b
mutex_adaptive_release+0x78
141 3% 65% 1.00 206 cpu[8]+0x44 setbackdq+0x188
140 3% 67% 1.00 513 cpu[2]+0x44 disp_getbest+0x4
135 2% 70% 1.00 147 cpu[12]+0x44 setbackdq+0x188
123 2% 72% 1.00 164 cpu[11]+0x44 setbackdq+0x188
123 2% 74% 1.00 170 cpu[10]+0x44 setbackdq+0x188
117 2% 76% 1.00 737 0x72b9ff3b mutex_vector_enter+0x208
104 2% 78% 1.00 165 cpu[0]+0x44 disp+0x78
94 2% 80% 1.00 165 0x72b9ff7b mutex_vector_enter+0x208
92 2% 82% 1.00 199 cpu[14]+0x44 setfrontdq+0x128
79 1% 83% 1.00 246 cpu[14]+0x44 disp+0x78
77 1% 84% 1.00 228 cpu[0]+0x44 setfrontdq+0x128
76 1% 86% 1.00 151 cpu[11]+0x44 disp+0x78
76 1% 87% 1.00 179 cpu[9]+0x44 setbackdq+0x188
69 1% 88% 1.00 164 cpu[8]+0x44 disp+0x78
69 1% 90% 1.00 263 cpu[12]+0x44 setfrontdq+0x128
65 1% 91% 1.00 192 cpu[12]+0x44 disp+0x78
61 1% 92% 1.00 243 cpu[11]+0x44 setfrontdq+0x128
57 1% 93% 1.00 182 cpu[10]+0x44 disp+0x78
55 1% 94% 1.00 228 cpu[8]+0x44 setfrontdq+0x128
53 1% 95% 1.00 267 cpu[10]+0x44 setfrontdq+0x128
51 1% 96% 1.00 144 cpu[9]+0x44 disp+0x78
49 1% 97% 1.00 236 sleepq_head+0x79c cv_block+0x70
45 1% 97% 1.00 276 cpu[9]+0x44 setfrontdq+0x128
32 1% 98% 1.00 200 cpu[2]+0x44 setfrontdq+0x128
28 1% 99% 1.00 100 0x7053800e mutex_vector_enter+0x13c
21 0% 99% 1.00 218 cpu[2]+0x44 disp+0x78
20 0% 99% 1.00 254 cpu[2]+0x44 setbackdq+0x188
12 0% 99% 1.00 181 service_queue+0x6 mutex_vector_enter+0x13c
3 0% 100% 1.00 85 sleepq_head+0x2ac sema_v+0x10
3 0% 100% 1.00 38 hres_lock+0x3 hr_clock_lock+0x10
2 0% 100% 1.00 111 hblk8_lock+0x6 mutex_vector_enter+0x13c
2 0% 100% 1.00 324 sleepq_head+0x2ac sema_tryp+0x14
1 0% 100% 1.00 1 sleepq_head+0xdfc cv_broadcast+0x24
1 0% 100% 1.00 198 0x72b9ffdb
mutex_adaptive_release+0x78
1 0% 100% 1.00 896 sleepq_head+0x5d4 cv_block+0x70
1 0% 100% 1.00 49 sleepq_head+0xb7c cv_broadcast+0x24
1 0% 100% 1.00 231 sleepq_head+0x9dc cv_block+0x70
1 0% 100% 1.00 1047 sleepq_head+0x7d4 cv_block+0x70
1 0% 100% 1.00 412 sleepq_head+0x22c cv_signal+0x24
1 0% 100% 1.00 614 sleepq_head+0x30c cv_block+0x70
1 0% 100% 1.00 41 0x724b2d36 mutex_vector_enter+0x13c
1 0% 100% 1.00 676 sleepq_head+0x42c sema_v+0x10
1 0% 100% 1.00 263 sleepq_head+0xea4 cv_broadcast+0x24
1 0% 100% 1.00 868 sleepq_head+0xcdc cv_block+0x70
1 0% 100% 1.00 157 sleepq_head+0xf44 cv_block+0x70
1 0% 100% 1.00 1 sleepq_head+0xec4 cv_broadcast+0x24
1 0% 100% 1.00 309 sleepq_head+0x994 cv_block+0x70
1 0% 100% 1.00 292 sleepq_head+0x44c cv_block+0x70
1 0% 100% 1.00 635 0x7600dfb4 rw_exit_wakeup+0x94
1 0% 100% 1.00 23 sleepq_head+0x1bc cv_block+0x70
1 0% 100% 1.00 4 0x72b9ff7b mutex_vector_enter+0x174
-----------------------------------------------------------------------------
--
Thread lock spin: 83 events
Count indv cuml rcnt spin Lock Caller
-----------------------------------------------------------------------------
--
29 35% 35% 1.00 302 sleepq_head+0x79c setrun+0x4
5 6% 41% 1.00 212 0x72b9ff3b ts_tick+0x8
3 4% 45% 1.00 167 cpu[0]+0x44 ts_tick+0x8
2 2% 47% 1.00 36 sleepq_head+0x79c pi_fix_epri+0x4c
2 2% 49% 1.00 38 cpu[8]+0x68 cv_wait_sig+0xb0
2 2% 52% 1.00 13 cpu[11]+0x68 preempt+0x1c
1 1% 53% 1.00 71 cpu[9]+0x68 sigtoproc+0x2c4
1 1% 54% 1.00 294 sleepq_head+0xd4c setrun+0x4
1 1% 55% 1.00 217 sleepq_head+0xabc ts_tick+0x8
1 1% 57% 1.00 256 sleepq_head+0xafc ts_tick+0x8
1 1% 58% 1.00 238 sleepq_head+0x624 setrun+0x4
1 1% 59% 1.00 89 cpu[9]+0x68 preempt+0x1c
1 1% 60% 1.00 84 cpu[12]+0x44 sigtoproc+0x2c4
1 1% 61% 1.00 132 sleepq_head+0xd74 ts_tick+0x8
1 1% 63% 1.00 282 sleepq_head+0x4ac ts_tick+0x8
1 1% 64% 1.00 80 cpu[9]+0x44 ts_tick+0x8
1 1% 65% 1.00 26 sleepq_head+0xda4 ts_update_list+0x60
1 1% 66% 1.00 6 cpu[0]+0x68 cv_wait_sig+0xb0
1 1% 67% 1.00 115 cpu[8]+0x44 sigtoproc+0x2c4
1 1% 69% 1.00 161 sleepq_head+0xddc ts_tick+0x8
1 1% 70% 1.00 5 sleepq_head+0xbfc sigtoproc+0x2c4
1 1% 71% 1.00 20 cpu[11]+0x44 sigtoproc+0x2c4
1 1% 72% 1.00 293 cpu[12]+0x44 prchoose+0x68
1 1% 73% 1.00 241 cpu[14]+0x44 ts_tick+0x8
1 1% 75% 1.00 10 cpu[14]+0x44 sigtoproc+0x2c4
1 1% 76% 1.00 22 cpu[9]+0x44 sigtoproc+0x2c4
1 1% 77% 1.00 225 cpu[0]+0x44 sigtoproc+0x2c4
1 1% 78% 1.00 50 cpu[11]+0x44 ts_tick+0x8
1 1% 80% 1.00 178 sleepq_head+0xfc4 ts_tick+0x8
1 1% 81% 1.00 243 sleepq_head+0x42c ts_tick+0x8
1 1% 82% 1.00 86 sleepq_head+0x6c ts_tick+0x8
1 1% 83% 1.00 406 sleepq_head+0xcdc ts_tick+0x8
1 1% 84% 1.00 133 sleepq_head+0xde4 ts_tick+0x8
1 1% 86% 1.00 5 sleepq_head+0xcec ts_update_list+0x60
1 1% 87% 1.00 77 cpu[0]+0x68 ts_tick+0x8
1 1% 88% 1.00 63 sleepq_head+0x9cc setrun+0x4
1 1% 89% 1.00 114 sleepq_head+0xdc4 ts_tick+0x8
1 1% 90% 1.00 108 sleepq_head+0x4cc ts_tick+0x8
1 1% 92% 1.00 41 cpu[12]+0x68 preempt+0x1c
1 1% 93% 1.00 520 sleepq_head+0xcdc setrun+0x4
1 1% 94% 1.00 183 sleepq_head+0x8bc ts_tick+0x8
1 1% 95% 1.00 129 sleepq_head+0xc5c ts_tick+0x8
1 1% 96% 1.00 13 cpu[10]+0x68 cv_wait_sig+0xb0
1 1% 98% 1.00 382 sleepq_head+0x634 prchoose+0x68
1 1% 99% 1.00 117 sleepq_head+0xd4c ts_update_list+0x60
1 1% 100% 1.00 294 sleepq_head+0x3bc ts_tick+0x8
-----------------------------------------------------------------------------
--
R/W writer blocked by writer: 11 events
Count indv cuml rcnt nsec Lock Caller
-----------------------------------------------------------------------------
--
5 45% 45% 1.00 26679356 0x73bbbf48 write+0x84
2 18% 64% 1.00 8311176 0x759428d8 pwrite64+0x170
2 18% 82% 1.00 3529571 0x7db33b68 pwrite64+0x170
1 9% 91% 1.00 14441149 0x73a72790 write+0x84
1 9% 100% 1.00 12546186 0x73a72640 write+0x84
-----------------------------------------------------------------------------
--
R/W writer blocked by readers: 1 events
Count indv cuml rcnt nsec Lock Caller
-----------------------------------------------------------------------------
--
1 100% 100% 1.00 162874 0x7600dfac as_map+0x6c
-----------------------------------------------------------------------------
--
R/W reader blocked by writer: 2 events
Count indv cuml rcnt nsec Lock Caller
-----------------------------------------------------------------------------
--
1 50% 50% 1.00 11314336 0x73bbbf48 pread+0xdc
1 50% 100% 1.00 14818874 0x73a72640 pread+0xdc
-----------------------------------------------------------------------------
--
Lockstat record failure: 1 events
Count indv cuml rcnt Lock Caller
-----------------------------------------------------------------------------
--
1 100% 100% 0.00 lockstat_lock lockstat_add_to_hash
-----------------------------------------------------------------------------
--
I have volume manager 2.5
my system file is:
set msgsys:msginfo_msgmap=258
set msgsys:msginfo_msgmax=32768
set msgsys:msginfo_msgmnb=65535
set msgsys:msginfo_msgmni=128
set msgsys:msginfo_msgtql=256
set msgsys:msginfo_msgseg=7168
set msgsys:msginfo_msgssz=32
set shmsys:shminfo_shmmin=1
*set shmsys:shminfo_shmmax=268435456
set shmsys:shminfo_shmmax=2684354560
set shmsys:shminfo_shmseg=200
set shmsys:shminfo_shmmni=1000
set semsys:seminfo_semmap=258
*set semsys:seminfo_semmni=2000
set semsys:seminfo_semmni=1000
set semsys:seminfo_semmns=10000
*set semsys:seminfo_semmns=4500
set semsys:seminfo_semmnu=1000
*set semsys:seminfo_semmsl=600
set semsys:seminfo_semmsl=4500
*set semsys:seminfo_semopm=20
set semsys:seminfo_semopm=200
set semsys:seminfo_semvmx=32767
set semsys:seminfo_semaem=16384
set semsys:seminfo_semume=100
forceload: sys/msgsys
forceload: sys/shmsys
forceload: sys/semsys
*
set pln:pln_enable_detach_suspend=1
set soc:soc_enable_detach_suspend=1
* SUNWecsr end
* Number of Users JM3
set pt_cnt=1024
set npty=1024
set rlim_fd_max=8096
* jm3
set qfe:qfe_adv_autoneg_cap=0
set qfe:qfe_adv_100T4_cap=0
set qfe:qfe_adv_100fdx_cap=1
set qfe:qfe_adv_100hdx_cap=0
set qfe:qfe_adv_10fdx_cap=0
set qfe:qfe_adv_10hdx_cap=0
* Priority Paging. jm3
set priority_paging=1
set fastscan=16000
set handspreadpages=16000
set maxpgio=65536
* DR jm3.
set dr_max_mem=16384
#set dr-max-mem=16384
# Network queue JM3
set sq_max_size=0x30
* Monitoring RAM State JM3
set report_ce_console=1
* VM FS Cached JM3
set bufhwm=56000
set ncsize=36000
set ufs_ninode=36000
* vxvm_START (do not remove)
forceload: drv/vxdmp
forceload: drv/vxio
forceload: drv/vxspec
forceload: drv/ssd
forceload: drv/sf
forceload: drv/SUNW,socal
forceload: drv/sbus
forceload: drv/sd
forceload: drv/pln
forceload: drv/soc
forceload: drv/socal
rootdev:/pseudo/vxio at 0:0
set vxio:vol_rootdev_is_volume=1
* vxvm_END (do not remove)
