https://github.com/apache/spark
Revision 371be22b1e56e6b56ad59130bcd9381a2ee4a014 authored by Santiago Saavedra on 10 November 2017, 18:57:58 UTC, committed by Shixiong Zhu on 10 November 2017, 18:58:10 UTC
## What changes were proposed in this pull request? It seems that recovering from a checkpoint can replace the old driver and executor IP addresses, as the workload can now be taking place in a different cluster configuration. It follows that the bindAddress for the master may also have changed. Thus we should not be keeping the old one, and instead be added to the list of properties to reset and recreate from the new environment. ## How was this patch tested? This patch was tested via manual testing on AWS, using the experimental (not yet merged) Kubernetes scheduler, which uses bindAddress to bind to a Kubernetes service (and thus was how I first encountered the bug too), but it is not a code-path related to the scheduler and this may have slipped through when merging SPARK-4563. Author: Santiago Saavedra <ssaavedra@openshine.com> Closes #19427 from ssaavedra/fix-checkpointing-master. (cherry picked from commit 5ebdcd185f2108a90e37a1aa4214c3b6c69a97a4) Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
1 parent eb49c32
Tip revision: 371be22b1e56e6b56ad59130bcd9381a2ee4a014 authored by Santiago Saavedra on 10 November 2017, 18:57:58 UTC
[SPARK-22294][DEPLOY] Reset spark.driver.bindAddress when starting a Checkpoint
[SPARK-22294][DEPLOY] Reset spark.driver.bindAddress when starting a Checkpoint
Tip revision: 371be22
| File | Mode | Size |
|---|---|---|
| .github | ||
| R | ||
| assembly | ||
| bin | ||
| build | ||
| common | ||
| conf | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| external | ||
| graphx | ||
| launcher | ||
| licenses | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .gitattributes | -rw-r--r-- | 40 bytes |
| .gitignore | -rw-r--r-- | 1.2 KB |
| .travis.yml | -rw-r--r-- | 1.7 KB |
| CONTRIBUTING.md | -rw-r--r-- | 995 bytes |
| LICENSE | -rw-r--r-- | 17.5 KB |
| NOTICE | -rw-r--r-- | 24.1 KB |
| README.md | -rw-r--r-- | 3.7 KB |
| appveyor.yml | -rw-r--r-- | 1.9 KB |
| pom.xml | -rw-r--r-- | 94.8 KB |
| scalastyle-config.xml | -rw-r--r-- | 17.4 KB |
Computing file changes ...
