During a performance tuning engagement, ATEAM had noticed an increase in page response times when the number of concurrent users increased to a value greater than 1000. Further investigation revealed that the bottleneck was coming from the usage of Content Presenter Template(s).
Using a jRockit Mission Control Flight Recording (JFR), ATEAM was able to determine the cause: Threads were being parked; each waiting to resume. So what was causing all of the parked threads? Digging deeper in stack traces provided by the JFR the root cause was discovered: java.util.concurrent.locks.ReentrantLock.lock(), which is called by UCMBridge.getAccessLevel(IdcContext, DataObject, ITrace). From within the Mission Control Event Graph view, the evidence of how this issue (identified by each of the grey blocks) significantly effected the page response times (identified by the lime-green blocks).
Stack trace view:
Although this has been identified as a BUG (17330713), there is a patch fix that is now available for 126.96.36.199.1.