https://github.com/apache/spark
Revision ede0e1a982f146da54fdd187a1c217bee8d0e1b4 authored by Wing Yew Poon on 10 November 2017, 00:20:55 UTC, committed by Shixiong Zhu on 10 November 2017, 00:21:06 UTC
## What changes were proposed in this pull request? When run in YARN cluster mode, the StructuredKafkaWordCount example fails because Spark tries to create a temporary checkpoint location in a subdirectory of the path given by java.io.tmpdir, and YARN sets java.io.tmpdir to a path in the local filesystem that usually does not correspond to an existing path in the distributed filesystem. Add an optional checkpointLocation argument to the StructuredKafkaWordCount example so that users can specify the checkpoint location and avoid this issue. ## How was this patch tested? Built and ran the example manually on YARN client and cluster mode. Author: Wing Yew Poon <wypoon@cloudera.com> Closes #19703 from wypoon/SPARK-22403. (cherry picked from commit 11c4021044f3a302449a2ea76811e73f5c99a26a) Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
1 parent 0e97c8e
Tip revision: ede0e1a982f146da54fdd187a1c217bee8d0e1b4 authored by Wing Yew Poon on 10 November 2017, 00:20:55 UTC
[SPARK-22403][SS] Add optional checkpointLocation argument to StructuredKafkaWordCount example
[SPARK-22403][SS] Add optional checkpointLocation argument to StructuredKafkaWordCount example
Tip revision: ede0e1a
| File | Mode | Size |
|---|---|---|
| .github | ||
| R | ||
| assembly | ||
| bin | ||
| build | ||
| common | ||
| conf | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| external | ||
| graphx | ||
| launcher | ||
| licenses | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .gitattributes | -rw-r--r-- | 40 bytes |
| .gitignore | -rw-r--r-- | 1.2 KB |
| .travis.yml | -rw-r--r-- | 1.7 KB |
| CONTRIBUTING.md | -rw-r--r-- | 995 bytes |
| LICENSE | -rw-r--r-- | 17.5 KB |
| NOTICE | -rw-r--r-- | 24.1 KB |
| README.md | -rw-r--r-- | 3.7 KB |
| appveyor.yml | -rw-r--r-- | 1.9 KB |
| pom.xml | -rw-r--r-- | 94.8 KB |
| scalastyle-config.xml | -rw-r--r-- | 17.4 KB |
Computing file changes ...
