https://github.com/apache/spark
Revision 401ac20d22f46f4a0e279c8aa08ceb00d8bdc617 authored by Devaraj K on 20 September 2017, 23:22:36 UTC, committed by Marcelo Vanzin on 20 September 2017, 23:22:46 UTC
## What changes were proposed in this pull request? When the libraries temp directory(i.e. __spark_libs__*.zip dir) file system and staging dir(destination) file systems are the same then the __spark_libs__*.zip is not copying to the staging directory. But after making this decision the libraries zip file is getting deleted immediately and becoming unavailable for the Node Manager's localization. With this change, client copies the files to remote always when the source scheme is "file". ## How was this patch tested? I have verified it manually in yarn/cluster and yarn/client modes with hdfs and local file systems. Author: Devaraj K <devaraj@apache.org> Closes #19141 from devaraj-kavali/SPARK-21384. (cherry picked from commit 55d5fa79db883e4d93a9c102a94713c9d2d1fb55) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
1 parent 5d10586
Tip revision: 401ac20d22f46f4a0e279c8aa08ceb00d8bdc617 authored by Devaraj K on 20 September 2017, 23:22:36 UTC
[SPARK-21384][YARN] Spark + YARN fails with LocalFileSystem as default FS
[SPARK-21384][YARN] Spark + YARN fails with LocalFileSystem as default FS
Tip revision: 401ac20
.gitignore
*#*#
*.#*
*.iml
*.ipr
*.iws
*.pyc
*.pyo
*.swp
*~
.DS_Store
.cache
.classpath
.ensime
.ensime_cache/
.ensime_lucene
.generated-mima*
.idea/
.idea_modules/
.project
.pydevproject
.scala_dependencies
.settings
/lib/
R-unit-tests.log
R/unit-tests.out
R/cran-check.out
R/pkg/vignettes/sparkr-vignettes.html
build/*.jar
build/apache-maven*
build/scala*
build/zinc*
cache
checkpoint
conf/*.cmd
conf/*.conf
conf/*.properties
conf/*.sh
conf/*.xml
conf/java-opts
conf/slaves
dependency-reduced-pom.xml
derby.log
dev/create-release/*final
dev/create-release/*txt
dev/pr-deps/
dist/
docs/_site
docs/api
lib_managed/
lint-r-report.log
log/
logs/
out/
project/boot/
project/build/target/
project/plugins/lib_managed/
project/plugins/project/build.properties
project/plugins/src_managed/
project/plugins/target/
python/lib/pyspark.zip
python/deps
python/pyspark/python
reports/
scalastyle-on-compile.generated.xml
scalastyle-output.xml
scalastyle.txt
spark-*-bin-*.tgz
spark-tests.log
src_managed/
streaming-tests.log
target/
unit-tests.log
work/
# For Hive
TempStatsStore/
metastore/
metastore_db/
sql/hive-thriftserver/test_warehouses
warehouse/
spark-warehouse/
# For R session data
.RData
.RHistory
.Rhistory
*.Rproj
*.Rproj.*
.Rproj.user
Computing file changes ...
