https://github.com/apache/spark
Revision 45561cd9fcf0445274d82da495401d32b62aa915 authored by Thomas Graves on 03 May 2014, 17:59:05 UTC, committed by Aaron Davidson on 03 May 2014, 18:13:06 UTC
Move the doAs in Executor higher up so that we only have 1 ugi and aren't leaking filesystems.
Fix spark on yarn to work when the cluster is running as user "yarn" but the clients are launched as the user and want to read/write to hdfs as the user.

Note this hasn't been fully tested yet.  Need to test in standalone mode.

Putting this up for people to look at and possibly test.  I don't have access to a mesos cluster.

This is alternative to https://github.com/apache/spark/pull/607

Author: Thomas Graves <tgraves@apache.org>

Closes #621 from tgravescs/SPARK-1676 and squashes the following commits:

244d55a [Thomas Graves] fix line length
44163d4 [Thomas Graves] Rework
9398853 [Thomas Graves] change to have doAs in executor higher up.

(cherry picked from commit 3d0a02dff3011e8894d98d903cd086bc95e56807)
Signed-off-by: Aaron Davidson <aaron@databricks.com>

Conflicts:
	core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
	core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
	yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
	yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
	yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
1 parent 54c3b7e
History
Tip revision: 45561cd9fcf0445274d82da495401d32b62aa915 authored by Thomas Graves on 03 May 2014, 17:59:05 UTC
[WIP] SPARK-1676: Cache Hadoop UGIs by default to prevent FileSystem leak
Tip revision: 45561cd

README.md

back to top