Skip to main content

Browse the archive

Revision 6764408f68495e2ca7c1b9959db53ee12cabb197 authored by Taaffy on 19 September 2017, 09:20:04 UTC, committed by Sean Owen on 19 September 2017, 09:20:14 UTC 
[SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala
 
Current implementation for processingRate-total uses wrong metric:
mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond

## What changes were proposed in this pull request?
Adjust processingRate-total from using inputRowsPerSecond to processedRowsPerSecond

## How was this patch tested?

Built spark from source with proposed change and tested output with correct parameter. Before change the csv metrics file for inputRate-total and processingRate-total displayed the same values due to the error. After changing MetricsReporter.scala the processingRate-total csv file displayed the correct metric.
<img width="963" alt="processed rows per second" src="https://user-images.githubusercontent.com/32072374/30554340-82eea12c-9ca4-11e7-8370-8168526ff9a2.png">

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Taaffy <32072374+Taaffy@users.noreply.github.com>

Closes #19268 from Taaffy/patch-1.

(cherry picked from commit 1bc17a6b8add02772a8a0a1048ac6a01d045baf4)
Signed-off-by: Sean Owen <sowen@cloudera.com>
1 parent d0234eb
History
File Mode Size
LICENSE-AnchorJS.txt -rw-r--r-- 1.1 KB
LICENSE-DPark.txt -rw-r--r-- 1.5 KB
LICENSE-Mockito.txt -rw-r--r-- 1.1 KB
LICENSE-SnapTree.txt -rw-r--r-- 1.7 KB
LICENSE-antlr.txt -rw-r--r-- 1.5 KB
LICENSE-boto.txt -rw-r--r-- 1.0 KB
LICENSE-cloudpickle.txt -rw-r--r-- 1.6 KB
LICENSE-d3.min.js.txt -rw-r--r-- 1.4 KB
LICENSE-dagre-d3.txt -rw-r--r-- 1.0 KB
LICENSE-f2j.txt -rw-r--r-- 1.5 KB
LICENSE-graphlib-dot.txt -rw-r--r-- 1.0 KB
LICENSE-heapq.txt -rw-r--r-- 14.2 KB
LICENSE-javolution.txt -rw-r--r-- 1.4 KB
LICENSE-jbcrypt.txt -rw-r--r-- 836 bytes
LICENSE-jline.txt -rw-r--r-- 1.5 KB
LICENSE-jpmml-model.txt -rw-r--r-- 1.5 KB
LICENSE-jquery.txt -rw-r--r-- 1.1 KB
LICENSE-junit-interface.txt -rw-r--r-- 1.3 KB
LICENSE-kryo.txt -rw-r--r-- 1.5 KB
LICENSE-minlog.txt -rw-r--r-- 1.5 KB
LICENSE-modernizr.txt -rw-r--r-- 1.1 KB
LICENSE-netlib.txt -rw-r--r-- 2.2 KB
LICENSE-paranamer.txt -rw-r--r-- 1.6 KB
LICENSE-postgresql.txt -rw-r--r-- 1.2 KB
LICENSE-protobuf.txt -rw-r--r-- 2.1 KB
LICENSE-py4j.txt -rw-r--r-- 1.4 KB
LICENSE-pyrolite.txt -rw-r--r-- 1.3 KB
LICENSE-reflectasm.txt -rw-r--r-- 1.5 KB
LICENSE-sbt-launch-lib.txt -rw-r--r-- 1.5 KB
LICENSE-scala.txt -rw-r--r-- 1.5 KB
LICENSE-scalacheck.txt -rw-r--r-- 1.5 KB
LICENSE-scopt.txt -rw-r--r-- 1.1 KB
LICENSE-slf4j.txt -rw-r--r-- 1.1 KB
LICENSE-sorttable.js.txt -rw-r--r-- 937 bytes
LICENSE-spire.txt -rw-r--r-- 1.0 KB
LICENSE-xmlenc.txt -rw-r--r-- 1.5 KB
back to top