Skip to main content

Browse the archive

Revision 7a04def920438ef0e08b66a95befeec981e5571e authored by Xianyang Liu on 07 August 2017, 09:04:53 UTC, committed by Wenchen Fan on 07 August 2017, 09:05:02 UTC 
[SPARK-21621][CORE] Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called
 
## What changes were proposed in this pull request?

We should reset numRecordsWritten to zero after DiskBlockObjectWriter.commitAndGet called.
Because when `revertPartialWritesAndClose` be called, we decrease the written records in `ShuffleWriteMetrics` . However, we decreased the written records to zero, this should be wrong, we should only decreased the number reords after the last `commitAndGet` called.

## How was this patch tested?
Modified existing test.

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Xianyang Liu <xianyang.liu@intel.com>

Closes #18830 from ConeyLiu/DiskBlockObjectWriter.

(cherry picked from commit 534a063f7c693158437d13224f50d4ae789ff6fb)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 098aaec
History
File Mode Size
_data
_includes
_layouts
_plugins
css
img
js
README.md -rw-r--r-- 3.6 KB
_config.yml -rw-r--r-- 615 bytes
api.md -rw-r--r-- 314 bytes
building-spark.md -rw-r--r-- 10.3 KB
cluster-overview.md -rw-r--r-- 6.5 KB
configuration.md -rw-r--r-- 85.4 KB
contributing-to-spark.md -rw-r--r-- 308 bytes
ec2-scripts.md -rw-r--r-- 217 bytes
graphx-programming-guide.md -rw-r--r-- 51.7 KB
hadoop-provided.md -rw-r--r-- 1.0 KB
hardware-provisioning.md -rw-r--r-- 3.8 KB
index.md -rw-r--r-- 7.7 KB
job-scheduling.md -rw-r--r-- 15.5 KB
ml-advanced.md -rw-r--r-- 6.2 KB
ml-ann.md -rw-r--r-- 264 bytes
ml-classification-regression.md -rw-r--r-- 47.5 KB
ml-clustering.md -rw-r--r-- 7.3 KB
ml-collaborative-filtering.md -rw-r--r-- 8.2 KB
ml-decision-tree.md -rw-r--r-- 210 bytes
ml-ensembles.md -rw-r--r-- 224 bytes
ml-features.md -rw-r--r-- 66.0 KB
ml-frequent-pattern-mining.md -rw-r--r-- 4.1 KB
ml-guide.md -rw-r--r-- 6.3 KB
ml-linear-methods.md -rw-r--r-- 195 bytes
ml-migration-guides.md -rw-r--r-- 20.1 KB
ml-pipeline.md -rw-r--r-- 13.0 KB
ml-statistics.md -rw-r--r-- 3.3 KB
ml-survival-regression.md -rw-r--r-- 225 bytes
ml-tuning.md -rw-r--r-- 7.0 KB
mllib-classification-regression.md -rw-r--r-- 1.7 KB
mllib-clustering.md -rw-r--r-- 23.8 KB
mllib-collaborative-filtering.md -rw-r--r-- 6.1 KB
mllib-data-types.md -rw-r--r-- 30.3 KB
mllib-decision-tree.md -rw-r--r-- 14.5 KB
mllib-dimensionality-reduction.md -rw-r--r-- 5.9 KB
mllib-ensembles.md -rw-r--r-- 15.9 KB
mllib-evaluation-metrics.md -rw-r--r-- 25.3 KB
mllib-feature-extraction.md -rw-r--r-- 15.9 KB
mllib-frequent-pattern-mining.md -rw-r--r-- 7.5 KB
mllib-guide.md -rw-r--r-- 2.8 KB
mllib-isotonic-regression.md -rw-r--r-- 4.6 KB
mllib-linear-methods.md -rw-r--r-- 23.4 KB
mllib-migration-guides.md -rw-r--r-- 319 bytes
mllib-naive-bayes.md -rw-r--r-- 4.2 KB
mllib-optimization.md -rw-r--r-- 12.8 KB
mllib-pmml-model-export.md -rw-r--r-- 2.0 KB
mllib-statistics.md -rw-r--r-- 17.3 KB
monitoring.md -rw-r--r-- 20.4 KB
programming-guide.md -rw-r--r-- 152 bytes
quick-start.md -rw-r--r-- 17.8 KB
rdd-programming-guide.md -rw-r--r-- 80.1 KB
running-on-mesos.md -rw-r--r-- 23.8 KB
running-on-yarn.md -rw-r--r-- 31.1 KB
security.md -rw-r--r-- 10.4 KB
spark-standalone.md -rw-r--r-- 21.3 KB
sparkr.md -rw-r--r-- 25.6 KB
sql-programming-guide.md -rw-r--r-- 94.2 KB
storage-openstack-swift.md -rw-r--r-- 4.8 KB
streaming-custom-receivers.md -rw-r--r-- 9.9 KB
streaming-flume-integration.md -rw-r--r-- 10.1 KB
streaming-kafka-0-10-integration.md -rw-r--r-- 16.9 KB
streaming-kafka-0-8-integration.md -rw-r--r-- 15.6 KB
streaming-kafka-integration.md -rw-r--r-- 1.5 KB
streaming-kinesis-integration.md -rw-r--r-- 14.2 KB
streaming-programming-guide.md -rw-r--r-- 116.5 KB
structured-streaming-kafka-integration.md -rw-r--r-- 20.2 KB
structured-streaming-programming-guide.md -rw-r--r-- 85.8 KB
submitting-applications.md -rw-r--r-- 11.2 KB
tuning.md -rw-r--r-- 19.9 KB

README.md

back to top