Deadlock when using AudioUnit

There was a deadlock occured when we tried to integrate my implementation for audio 5.1 into Firefox. You can see the bug here. It only happens on OSX. After analysis, I wrote a test to prevent others from getting into the same problem. The test is added to cubeb, which is our cross-platform audio library for Firefox. We reproduced a simpler version of the deadlock) in the test.

However, the code is not easy enough for those who are not familir with cubeb, so I wrote a general version to highlight the issue to everyone who uses AudioUnit in their audio backend. You can find the code on gist here

The APIs called to play and stop the audio stream is:

The key why deadlock happend is that the audio callback thread holds a mutex(hereafter referred to as Mutex-AU) shared with AudioUnit. The Mutex-AU is held inside it’s framework, so you don’t notice it.

Thus, if the callback thread requests another mutex M held by the another thread, without releasing mutex-AU, then it will cause a deadlock when the another thread, which holds the mutex M, request to use AudioUnit.

That is, if we have a thread T, holding the mutex M

and one callback thread which holds the mutex-AU,

The deadlock will occur when the callback thread requests the mutex M (the callback thread is blocked for waiting the mutex M)

and the thread T requests the mutex-AU to use AudioUnit