Revision 7076ab40f86fe606cd9b813dad506e921501383e authored by Yash Sharma on 16 May 2017, 22:08:05 UTC, committed by Burak Yavuz on 16 May 2017, 22:08:46 UTC
## What changes were proposed in this pull request?

The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES.

This change is critical for kinesis checkpoint recovery when the kinesis backed rdd is huge.
Following happens in a typical kinesis recovery :
- kinesis throttles large number of requests while recovering
- retries in case of throttling are not able to recover due to the small wait period
- kinesis throttles per second, the wait period should be configurable for recovery

The patch picks the spark kinesis configs from:
- spark.streaming.kinesis.retry.wait.time
- spark.streaming.kinesis.retry.max.attempts

Jira : https://issues.apache.org/jira/browse/SPARK-20140

## How was this patch tested?

Modified the KinesisBackedBlockRDDSuite.scala to run kinesis tests with the modified configurations. Wasn't able to test the patch with actual throttling.

Author: Yash Sharma <ysharma@atlassian.com>

Closes #17467 from yssharma/ysharma/spark-kinesis-retries.

(cherry picked from commit 38f4e8692ce3b6cbcfe0c1aff9b5e662f7a308b7)
Signed-off-by: Burak Yavuz <brkyvz@gmail.com>
1 parent 75e5ea2
History
File Mode Size
create-release
deps
sparktestsupport
tests
.gitignore -rw-r--r-- 9 bytes
.rat-excludes -rw-r--r-- 1.7 KB
README.md -rw-r--r-- 197 bytes
appveyor-guide.md -rw-r--r-- 7.9 KB
appveyor-install-dependencies.ps1 -rw-r--r-- 3.7 KB
change-scala-version.sh -rwxr-xr-x 2.0 KB
change-version-to-2.10.sh -rwxr-xr-x 1018 bytes
change-version-to-2.11.sh -rwxr-xr-x 1018 bytes
check-license -rwxr-xr-x 2.5 KB
checkstyle-suppressions.xml -rw-r--r-- 2.3 KB
checkstyle.xml -rw-r--r-- 7.9 KB
github_jira_sync.py -rwxr-xr-x 5.2 KB
lint-java -rwxr-xr-x 1.2 KB
lint-python -rwxr-xr-x 3.9 KB
lint-r -rwxr-xr-x 1.3 KB
lint-r.R -rw-r--r-- 1.5 KB
lint-scala -rwxr-xr-x 920 bytes
make-distribution.sh -rwxr-xr-x 7.9 KB
merge_spark_pr.py -rwxr-xr-x 18.8 KB
mima -rwxr-xr-x 1.7 KB
pip-sanity-check.py -rw-r--r-- 1.4 KB
requirements.txt -rw-r--r-- 64 bytes
run-pip-tests -rwxr-xr-x 4.6 KB
run-tests -rwxr-xr-x 895 bytes
run-tests-jenkins -rwxr-xr-x 1.2 KB
run-tests-jenkins.py -rwxr-xr-x 8.9 KB
run-tests.py -rwxr-xr-x 23.1 KB
scalastyle -rwxr-xr-x 1.3 KB
test-dependencies.sh -rwxr-xr-x 3.7 KB
tox.ini -rw-r--r-- 848 bytes

README.md

back to top