Skip to main content

Browse the archive

Revision c59ad420b5fda29567f4a06b5f71df76e70e269a authored by Liang-Chi Hsieh on 24 May 2017, 16:35:40 UTC, committed by Wenchen Fan on 24 May 2017, 16:36:01 UTC 
[SPARK-20848][SQL] Shutdown the pool after reading parquet files
 
## What changes were proposed in this pull request?

From JIRA: On each call to spark.read.parquet, a new ForkJoinPool is created. One of the threads in the pool is kept in the WAITING state, and never stopped, which leads to unbounded growth in number of threads.

We should shutdown the pool after reading parquet files.

## How was this patch tested?

Added a test to ParquetFileFormatSuite.

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #18073 from viirya/SPARK-20848.

(cherry picked from commit f72ad303f05a6d99513ea3b121375726b177199c)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 83aeac9
Raw File
appveyor.yml 
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: "{build}-{branch}"

shallow_clone: true

platform: x64
configuration: Debug

branches:
  only:
    - master

only_commits:
  files:
    - R/
    - sql/core/src/main/scala/org/apache/spark/sql/api/r/
    - core/src/main/scala/org/apache/spark/api/r/
    - mllib/src/main/scala/org/apache/spark/ml/r/

cache:
  - C:\Users\appveyor\.m2

install:
  # Install maven and dependencies
  - ps: .\dev\appveyor-install-dependencies.ps1
  # Required package for R unit tests
  - cmd: R -e "install.packages('testthat', repos='http://cran.us.r-project.org')"
  - cmd: R -e "packageVersion('testthat')"
  - cmd: R -e "install.packages('e1071', repos='http://cran.us.r-project.org')"
  - cmd: R -e "packageVersion('e1071')"
  - cmd: R -e "install.packages('survival', repos='http://cran.us.r-project.org')"
  - cmd: R -e "packageVersion('survival')"

build_script:
  - cmd: mvn -DskipTests -Psparkr -Phive -Phive-thriftserver package

environment:
  NOT_CRAN: true

test_script:
  - cmd: .\bin\spark-submit2.cmd --conf spark.hadoop.fs.defaultFS="file:///" R\pkg\tests\run-all.R

notifications:
  - provider: Email
    on_build_success: false
    on_build_failure: false
    on_build_status_changed: false
back to top