https://github.com/apache/spark
Revision ede0e1a982f146da54fdd187a1c217bee8d0e1b4 authored by Wing Yew Poon on 10 November 2017, 00:20:55 UTC, committed by Shixiong Zhu on 10 November 2017, 00:21:06 UTC
## What changes were proposed in this pull request? When run in YARN cluster mode, the StructuredKafkaWordCount example fails because Spark tries to create a temporary checkpoint location in a subdirectory of the path given by java.io.tmpdir, and YARN sets java.io.tmpdir to a path in the local filesystem that usually does not correspond to an existing path in the distributed filesystem. Add an optional checkpointLocation argument to the StructuredKafkaWordCount example so that users can specify the checkpoint location and avoid this issue. ## How was this patch tested? Built and ran the example manually on YARN client and cluster mode. Author: Wing Yew Poon <wypoon@cloudera.com> Closes #19703 from wypoon/SPARK-22403. (cherry picked from commit 11c4021044f3a302449a2ea76811e73f5c99a26a) Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
1 parent 0e97c8e
Tip revision: ede0e1a982f146da54fdd187a1c217bee8d0e1b4 authored by Wing Yew Poon on 10 November 2017, 00:20:55 UTC
[SPARK-22403][SS] Add optional checkpointLocation argument to StructuredKafkaWordCount example
[SPARK-22403][SS] Add optional checkpointLocation argument to StructuredKafkaWordCount example
Tip revision: ede0e1a
CONTRIBUTING.md
## Contributing to Spark
*Before opening a pull request*, review the
[Contributing to Spark guide](http://spark.apache.org/contributing.html).
It lists steps that are required before creating a PR. In particular, consider:
- Is the change important and ready enough to ask the community to spend time reviewing?
- Have you searched for existing, related JIRAs and pull requests?
- Is this a new feature that can stand alone as a [third party project](http://spark.apache.org/third-party-projects.html) ?
- Is the change being proposed clearly explained and motivated?
When you contribute code, you affirm that the contribution is your original work and that you
license the work to the project under the project's open source license. Whether or not you
state this explicitly, by submitting any copyrighted material via pull request, email, or
other means you agree to license the material under the project's open source license and
warrant that you have the legal authority to do so.
Computing file changes ...
