Skip to main content

Browse the archive

Revision 6764408f68495e2ca7c1b9959db53ee12cabb197 authored by Taaffy on 19 September 2017, 09:20:04 UTC, committed by Sean Owen on 19 September 2017, 09:20:14 UTC 
[SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala
 
Current implementation for processingRate-total uses wrong metric:
mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond

## What changes were proposed in this pull request?
Adjust processingRate-total from using inputRowsPerSecond to processedRowsPerSecond

## How was this patch tested?

Built spark from source with proposed change and tested output with correct parameter. Before change the csv metrics file for inputRate-total and processingRate-total displayed the same values due to the error. After changing MetricsReporter.scala the processingRate-total csv file displayed the correct metric.
<img width="963" alt="processed rows per second" src="https://user-images.githubusercontent.com/32072374/30554340-82eea12c-9ca4-11e7-8370-8168526ff9a2.png">

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Taaffy <32072374+Taaffy@users.noreply.github.com>

Closes #19268 from Taaffy/patch-1.

(cherry picked from commit 1bc17a6b8add02772a8a0a1048ac6a01d045baf4)
Signed-off-by: Sean Owen <sowen@cloudera.com>
1 parent d0234eb
Raw File
appveyor.yml 
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: "{build}-{branch}"

shallow_clone: true

platform: x64
configuration: Debug

branches:
  only:
    - master

only_commits:
  files:
    - appveyor.yml
    - dev/appveyor-install-dependencies.ps1
    - R/
    - sql/core/src/main/scala/org/apache/spark/sql/api/r/
    - core/src/main/scala/org/apache/spark/api/r/
    - mllib/src/main/scala/org/apache/spark/ml/r/

cache:
  - C:\Users\appveyor\.m2

install:
  # Install maven and dependencies
  - ps: .\dev\appveyor-install-dependencies.ps1
  # Required package for R unit tests
  - cmd: R -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='http://cran.us.r-project.org')"
  - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown'); packageVersion('testthat'); packageVersion('e1071'); packageVersion('survival')"

build_script:
  - cmd: mvn -DskipTests -Psparkr -Phive -Phive-thriftserver package

environment:
  NOT_CRAN: true

test_script:
  - cmd: .\bin\spark-submit2.cmd --conf spark.hadoop.fs.defaultFS="file:///" R\pkg\tests\run-all.R

notifications:
  - provider: Email
    on_build_success: false
    on_build_failure: false
    on_build_status_changed: false
back to top