It has been observed that very rarely is the maximum number of threads reached on execution queues, under heavy load.
During several stress tests, at certain points there has been heavy activity on only one of the queues, while the rest were idle. Even if the configured max number of threads for the busy queue is lower than the global number of threads, this value has never been reached, and ISBI still had idle threads while queue depth on the busy queues increased progressively.
It would appear that the thread scheduling algorithm tries to preserve certain threads available for the rest of queues. This results in lower average utilisation of system resources, since there are threads not doing any work that could be used to supplement busy queues.
One way to optimize the number of threads used on average is to distribute workload among idle queues, but this can only be achieved with complex logic and the Execution Control Service (to switch a WFC context to a different queue under specific circumstances. It would be better if 'fallback' queues could be defined for Business Processes, so that when a (configurable) certain queue depth threshold is exceeded, additional Business Process instances can be put on queues other than their predefined one - the 'fallback' queues.