Revision 7076ab40f86fe606cd9b813dad506e921501383e authored by Yash Sharma on 16 May 2017, 22:08:05 UTC, committed by Burak Yavuz on 16 May 2017, 22:08:46 UTC
## What changes were proposed in this pull request? The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES. This change is critical for kinesis checkpoint recovery when the kinesis backed rdd is huge. Following happens in a typical kinesis recovery : - kinesis throttles large number of requests while recovering - retries in case of throttling are not able to recover due to the small wait period - kinesis throttles per second, the wait period should be configurable for recovery The patch picks the spark kinesis configs from: - spark.streaming.kinesis.retry.wait.time - spark.streaming.kinesis.retry.max.attempts Jira : https://issues.apache.org/jira/browse/SPARK-20140 ## How was this patch tested? Modified the KinesisBackedBlockRDDSuite.scala to run kinesis tests with the modified configurations. Wasn't able to test the patch with actual throttling. Author: Yash Sharma <ysharma@atlassian.com> Closes #17467 from yssharma/ysharma/spark-kinesis-retries. (cherry picked from commit 38f4e8692ce3b6cbcfe0c1aff9b5e662f7a308b7) Signed-off-by: Burak Yavuz <brkyvz@gmail.com>
1 parent 75e5ea2
File | Mode | Size |
---|---|---|
create-release | ||
deps | ||
sparktestsupport | ||
tests | ||
.gitignore | -rw-r--r-- | 9 bytes |
.rat-excludes | -rw-r--r-- | 1.7 KB |
README.md | -rw-r--r-- | 197 bytes |
appveyor-guide.md | -rw-r--r-- | 7.9 KB |
appveyor-install-dependencies.ps1 | -rw-r--r-- | 3.7 KB |
change-scala-version.sh | -rwxr-xr-x | 2.0 KB |
change-version-to-2.10.sh | -rwxr-xr-x | 1018 bytes |
change-version-to-2.11.sh | -rwxr-xr-x | 1018 bytes |
check-license | -rwxr-xr-x | 2.5 KB |
checkstyle-suppressions.xml | -rw-r--r-- | 2.3 KB |
checkstyle.xml | -rw-r--r-- | 7.9 KB |
github_jira_sync.py | -rwxr-xr-x | 5.2 KB |
lint-java | -rwxr-xr-x | 1.2 KB |
lint-python | -rwxr-xr-x | 3.9 KB |
lint-r | -rwxr-xr-x | 1.3 KB |
lint-r.R | -rw-r--r-- | 1.5 KB |
lint-scala | -rwxr-xr-x | 920 bytes |
make-distribution.sh | -rwxr-xr-x | 7.9 KB |
merge_spark_pr.py | -rwxr-xr-x | 18.8 KB |
mima | -rwxr-xr-x | 1.7 KB |
pip-sanity-check.py | -rw-r--r-- | 1.4 KB |
requirements.txt | -rw-r--r-- | 64 bytes |
run-pip-tests | -rwxr-xr-x | 4.6 KB |
run-tests | -rwxr-xr-x | 895 bytes |
run-tests-jenkins | -rwxr-xr-x | 1.2 KB |
run-tests-jenkins.py | -rwxr-xr-x | 8.9 KB |
run-tests.py | -rwxr-xr-x | 23.1 KB |
scalastyle | -rwxr-xr-x | 1.3 KB |
test-dependencies.sh | -rwxr-xr-x | 3.7 KB |
tox.ini | -rw-r--r-- | 848 bytes |
Computing file changes ...