Revision c6ba647935e9108d139cc8914091790917567ad7 authored by IngoSchuster on 30 June 2017, 03:16:09 UTC, committed by Wenchen Fan on 30 June 2017, 03:16:19 UTC
## What changes were proposed in this pull request?
Please see also https://issues.apache.org/jira/browse/SPARK-21176

This change limits the number of selector threads that jetty creates to maximum 8 per proxy servlet (Jetty default is number of processors / 2).
The newHttpClient for Jettys ProxyServlet class is overwritten to avoid the Jetty defaults (which are designed for high-performance http servers).
Once https://github.com/eclipse/jetty.project/issues/1643 is available, the code could be cleaned up to avoid the method override.

I really need this on v2.1.1 - what is the best way for a backport automatic merge works fine)? Shall I create another PR?

## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
The patch was tested manually on a Spark cluster with a head node that has 88 processors using JMX to verify that the number of selector threads is now limited to 8 per proxy.

gurvindersingh zsxwing can you please review the change?

Author: IngoSchuster <ingo.schuster@de.ibm.com>
Author: Ingo Schuster <ingo.schuster@de.ibm.com>

Closes #18437 from IngoSchuster/master.

(cherry picked from commit 88a536babf119b7e331d02aac5d52b57658803bf)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 8de67e3
History
File Mode Size
slaves.sh -rwxr-xr-x 2.7 KB
spark-config.sh -rwxr-xr-x 1.4 KB
spark-daemon.sh -rwxr-xr-x 5.6 KB
spark-daemons.sh -rwxr-xr-x 1.2 KB
start-all.sh -rwxr-xr-x 1.2 KB
start-history-server.sh -rwxr-xr-x 1.2 KB
start-master.sh -rwxr-xr-x 2.0 KB
start-mesos-dispatcher.sh -rwxr-xr-x 1.8 KB
start-mesos-shuffle-service.sh -rwxr-xr-x 1.4 KB
start-shuffle-service.sh -rwxr-xr-x 1.2 KB
start-slave.sh -rwxr-xr-x 3.1 KB
start-slaves.sh -rwxr-xr-x 1.5 KB
start-thriftserver.sh -rwxr-xr-x 1.8 KB
stop-all.sh -rwxr-xr-x 1.4 KB
stop-history-server.sh -rwxr-xr-x 1.0 KB
stop-master.sh -rwxr-xr-x 1.1 KB
stop-mesos-dispatcher.sh -rwxr-xr-x 1.2 KB
stop-mesos-shuffle-service.sh -rwxr-xr-x 1.1 KB
stop-shuffle-service.sh -rwxr-xr-x 1.0 KB
stop-slave.sh -rwxr-xr-x 1.5 KB
stop-slaves.sh -rwxr-xr-x 1.0 KB
stop-thriftserver.sh -rwxr-xr-x 1.0 KB

back to top