Latch, mutex and beyond

July 30, 2012

Mutex waits. Part III. Contemporary Oracle wait schemes diversity.

Filed under: 11.2,Contention,DTrace,Instrumentation,Mutex,OS tuning,Patches,Spinlock — andreynikolaev @ 12:21 pm

Several months have passed since my previous “mutex wait” post. I was so busy with work and conference presentations. Thanks to all my listeners at UKOUG2011, Hotsos2012 and Medias2012 conferences and several seminars for inspiring questions and conversations.

I. Unexpected change.

Now it is time to discuss how contemporary Oracle waits for mutexes. My previous posts described evolution of “invisible and aggressive” 10.2-11.1 mutex waits into fully accounted and less aggressive 11gR2 mutexes. Surprisingly Oracle (or PSU2) appeared in April 2011 demonstrated almost negligible CPU consumption during mutex waits. (more…)

March 25, 2012

Database under fire

Filed under: Corruption,Patches — andreynikolaev @ 6:58 pm

This post does not discuss mutexes or latches. I wrote it under the impression of recent escalation due to database corruption.

This was Oracle database on HP-UX. Database files underwent recurrent corruption. Most commonly blocks were corrupted in system and undo tablespaces. This resulted in corrupted dictionary tables and indexes. Neither db_block_checking/db_block_checksum nor installation of latest PSU and OS patches helped. SCNs showed that these dictionary blocks were not changed for years. It looks likes some program other than DBWR overwrites the blocks. Corruptions caused downtimes to restore and recover datafiles.

October 25, 2011

Mutex waits. Part II. “Cursor: Pin S” in Oracle 11.2 _mutex_wait_scheme=0. Steps out of shadow.

Filed under: 11.2,Contention,DTrace,Instrumentation,Mutex,OS tuning,Patches — andreynikolaev @ 4:23 pm

I would like to describe how Oracle versions waited for mutexes. This algorithm also appears to be used in post- PSUs and new patchset as _mutex_wait_scheme=0.

My previous post demonstrated that before version 11.2:

  • “Cursor: pin S” was pure wait for CPU. Long “cursor: pin S” waits indicated CPU starvation.
  • Mutex contention was almost invisible to Oracle Wait Interface
  • Spin time to acquire mutex was accounted as CPU time. It was service time, not waiting time.

Things changed. Mutex waits in Oracle 11.2 significantly differ from previous versions. Contemporary mutex waits are not CPU aggressive anymore, completely visible to Oracle Wait Interface and highly tunable.

July 9, 2011

Mutex waits. Part 1. “Cursor: Pin S” in Oracle 10.2-11.1. Invisible and aggressive.

Filed under: Contention,History,Instrumentation,Mutex,Patches,Spinlock — andreynikolaev @ 1:18 pm

Oracle KGX mutexes appeared more than 7 years ago. However, mutex waits are still obscure. Oracle Documentation provided only brief description of mutex wait events without any information about wait durations and timeouts.
Look at the following timeline:


May 1, 2011

Divide and conquer the “true” mutex contention

Filed under: 11.2,Contention,Mutex,Patches — andreynikolaev @ 7:16 pm

Oracle contains enhancements 9282521 and 9239863 named “Library cache: mutex X” for objects highly contended for. Part I and II. These enhancements introduce new interesting possibilities to tune some types of the mutex contention.

Contention for heavily accessed objects can now be divided between multiple copies of object in the library cache. According to notes 9282521.8 and 9239863.8 describing the patches, the enhancements should be used:
When there is true contention on a specific library cache object….
Let me investigate this deeper. I will use Oracle for Solaris SPARC (64-bit) on 8Cores/32Threads Sun Fire T2000. I chose this platform in order to emphasize how the enhancements work. (more…)

March 19, 2011 is the right patchset for mutexes

Filed under: 11.2,Mutex,Patches — andreynikolaev @ 10:05 am

Many people asked me about the second part of my blog title – the mutex. This is the first post about it. Mutexes is another Oracle spinlock, which was appeared in version Despite being known since 2005, Oracle mutex internals is still Terra incognita.

This post is inspired by several recent escalations due to mutex contention. It occurs that patchset contains extraordinary number of mutex related changes. Some enhancements like 10411618 exist only for The following patches even changed the mutex architecture: (more…)

Blog at