https://github.com/apache/spark
Revision 401ac20d22f46f4a0e279c8aa08ceb00d8bdc617 authored by Devaraj K on 20 September 2017, 23:22:36 UTC, committed by Marcelo Vanzin on 20 September 2017, 23:22:46 UTC
## What changes were proposed in this pull request? When the libraries temp directory(i.e. __spark_libs__*.zip dir) file system and staging dir(destination) file systems are the same then the __spark_libs__*.zip is not copying to the staging directory. But after making this decision the libraries zip file is getting deleted immediately and becoming unavailable for the Node Manager's localization. With this change, client copies the files to remote always when the source scheme is "file". ## How was this patch tested? I have verified it manually in yarn/cluster and yarn/client modes with hdfs and local file systems. Author: Devaraj K <devaraj@apache.org> Closes #19141 from devaraj-kavali/SPARK-21384. (cherry picked from commit 55d5fa79db883e4d93a9c102a94713c9d2d1fb55) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
1 parent 5d10586
Tip revision: 401ac20d22f46f4a0e279c8aa08ceb00d8bdc617 authored by Devaraj K on 20 September 2017, 23:22:36 UTC
[SPARK-21384][YARN] Spark + YARN fails with LocalFileSystem as default FS
[SPARK-21384][YARN] Spark + YARN fails with LocalFileSystem as default FS
Tip revision: 401ac20
| File | Mode | Size |
|---|---|---|
| .github | ||
| R | ||
| assembly | ||
| bin | ||
| build | ||
| common | ||
| conf | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| external | ||
| graphx | ||
| launcher | ||
| licenses | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .gitattributes | -rw-r--r-- | 40 bytes |
| .gitignore | -rw-r--r-- | 1.2 KB |
| .travis.yml | -rw-r--r-- | 1.7 KB |
| CONTRIBUTING.md | -rw-r--r-- | 995 bytes |
| LICENSE | -rw-r--r-- | 17.5 KB |
| NOTICE | -rw-r--r-- | 24.1 KB |
| README.md | -rw-r--r-- | 3.7 KB |
| appveyor.yml | -rw-r--r-- | 1.9 KB |
| pom.xml | -rw-r--r-- | 94.8 KB |
| scalastyle-config.xml | -rw-r--r-- | 17.4 KB |
Computing file changes ...
