Skip to main content

Browse the archive

https://github.com/apache/spark
28 July 2024, 20:17:12 UTC
sort by:
Revision Author Date Message Commit Date
7531b50 Patrick Wendell 05 April 2015, 12:17:47 UTC Preparing Spark release v1.2.2-rc1 05 April 2015, 12:17:47 UTC
86d1715 Patrick Wendell 05 April 2015, 12:17:49 UTC [HOTFIX] Bumping versions for Spark 1.2.2 05 April 2015, 12:17:49 UTC
eac9525 Patrick Wendell 05 April 2015, 12:15:33 UTC [HOTFIX] Updating CHANGES.txt for Spark 1.2.2 05 April 2015, 12:15:33 UTC
f4a9c41 KaiXinXiaoLei 03 April 2015, 03:24:31 UTC [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory The config option is spark.history.fs.logDirectory, not spark.fs.history.logDirectory. So the descriptionof should be changed. Thanks. Author: KaiXinXiaoLei <huleilei1@huawei.com> Closes #5332 from KaiXinXiaoLei/historyConfig and squashes the following commits: 5ffbfb5 [KaiXinXiaoLei] the describe of jobHistory config is error (cherry picked from commit 8a0aa81ca37d337423db60edb09cf264cc2c6498) Signed-off-by: Andrew Or <andrew@databricks.com> 03 April 2015, 03:24:48 UTC
2991dd0 seayi 03 February 2015, 00:06:52 UTC [SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes #3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala 02 April 2015, 23:57:57 UTC
d82e732 Reynold Xin 02 April 2015, 21:51:00 UTC [SPARK-6578] [core] Fix thread-safety issue in outbound path of network library. While the inbound path of a netty pipeline is thread-safe, the outbound path is not. That means that multiple threads can compete to write messages to the next stage of the pipeline. The network library sometimes breaks a single RPC message into multiple buffers internally to avoid copying data (see MessageEncoder). This can result in the following scenario (where "FxBy" means "frame x, buffer y"): T1 F1B1 F1B2 \ \ \ \ socket F1B1 F2B1 F1B2 F2B2 / / / / T2 F2B1 F2B2 And the frames now cannot be rebuilt on the receiving side because the different messages have been mixed up on the wire. The fix wraps these multi-buffer messages into a `FileRegion` object so that these messages are written "atomically" to the next pipeline handler. Author: Reynold Xin <rxin@databricks.com> Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #5336 from vanzin/SPARK-6578-1.2 and squashes the following commits: 4d3395e [Reynold Xin] [SPARK-6578] Small rewrite to make the logic more clear in MessageWithHeader.transferTo. 526f230 [Marcelo Vanzin] [SPARK-6578] [core] Fix thread-safety issue in outbound path of network library. 02 April 2015, 21:51:00 UTC
8fa09a4 Hung Lin 02 April 2015, 21:01:43 UTC SPARK-6414: Spark driver failed with NPE on job cancelation Use Option for ActiveJob.properties to avoid NPE bug Author: Hung Lin <hung.lin@gmail.com> Closes #5124 from hunglin/SPARK-6414 and squashes the following commits: 2290b6b [Hung Lin] [SPARK-6414][core] Fix NPE in SparkContext.cancelJobGroup() (cherry picked from commit e3202aa2e9bd140effbcf2a7a02b90cb077e760b) Signed-off-by: Josh Rosen <joshrosen@databricks.com> Conflicts: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala Conflicts: core/src/test/scala/org/apache/spark/SparkContextSuite.scala 02 April 2015, 21:16:45 UTC
a73055f Davies Liu 02 April 2015, 19:18:33 UTC [SPARK-6667] [PySpark] remove setReuseAddress The reused address on server side had caused the server can not acknowledge the connected connections, remove it. This PR will retry once after timeout, it also add a timeout at client side. Author: Davies Liu <davies@databricks.com> Closes #5324 from davies/collect_hang and squashes the following commits: e5a51a2 [Davies Liu] remove setReuseAddress 7977c2f [Davies Liu] do retry on client side b838f35 [Davies Liu] retry after timeout (cherry picked from commit 0cce5451adfc6bf4661bcf67aca3db26376455fe) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 02 April 2015, 19:21:04 UTC
758ebf7 Sean Owen 26 March 2015, 15:00:23 UTC SPARK-6480 [CORE] histogram() bucket function is wrong in some simple edge cases Fix fastBucketFunction for histogram() to handle edge conditions more correctly. Add a test, and fix existing one accordingly Author: Sean Owen <sowen@cloudera.com> Closes #5148 from srowen/SPARK-6480 and squashes the following commits: 974a0a0 [Sean Owen] Additional test of huge ranges, and a few more comments (and comment fixes) 23ec01e [Sean Owen] Fix fastBucketFunction for histogram() to handle edge conditions more correctly. Add a test, and fix existing one accordingly (cherry picked from commit fe15ea976073edd738c006af1eb8d31617a039fc) Signed-off-by: Sean Owen <sowen@cloudera.com> 26 March 2015, 15:00:42 UTC
61c059a Josh Rosen 17 March 2015, 16:18:57 UTC [SPARK-3266] Use intermediate abstract classes to fix type erasure issues in Java APIs This PR addresses a Scala compiler bug ([SI-8905](https://issues.scala-lang.org/browse/SI-8905)) that was breaking some of the Spark Java APIs. In a nutshell, it seems that methods whose implementations are inherited from generic traits sometimes have their type parameters erased to Object. This was causing methods like `DoubleRDD.min()` to throw confusing NoSuchMethodErrors at runtime. The fix implemented here is to introduce an intermediate layer of abstract classes and inherit from those instead of directly extends the `Java*Like` traits. This should not break binary compatibility. I also improved the test coverage of the Java API, adding several new tests for methods that failed at runtime due to this bug. Author: Josh Rosen <joshrosen@databricks.com> Closes #5050 from JoshRosen/javardd-si-8905-fix and squashes the following commits: 2feb068 [Josh Rosen] Use intermediate abstract classes to work around SPARK-3266 d5f3e5d [Josh Rosen] Add failing regression tests for SPARK-3266 (cherry picked from commit 0f673c21f68ee3d5df3c01ae405709d3c1f4909b) Signed-off-by: Josh Rosen <joshrosen@databricks.com> Conflicts: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala core/src/test/java/org/apache/spark/JavaAPISuite.java 24 March 2015, 20:47:19 UTC
8ef6995 Kousuke Saruta 24 March 2015, 16:13:25 UTC [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 52 times over 10.094849836 seconds. Last failure message: Error connecting to localhost/127.0.0.1:23456. at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116) at org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.scalatest.Suite$class.withFixture(Suite.scala:1122) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175) This error is caused by check-then-act logic when it find free-port . /** Find a free port */ private def findFreePort(): Int = { Utils.startServiceOnPort(23456, (trialPort: Int) => { val socket = new ServerSocket(trialPort) socket.close() (null, trialPort) }, conf)._2 } Removing the check-then-act is not easy but we can reduce the chance of having the error by choosing random value for initial port instead of 23456. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #4337 from sarutak/SPARK-5559 and squashes the following commits: 16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark into SPARK-5559 1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 33357e3 [Kousuke Saruta] Changed "findFreePort" method in MQTTStreamSuite and FlumeStreamSuite so that it can choose valid random port a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from 23456 to random value (cherry picked from commit 85cf0636825d1997d64d0bdc04618f29b7222da1) Signed-off-by: Sean Owen <sowen@cloudera.com> 24 March 2015, 16:22:22 UTC
e080cc3 Anselme Vignon 23 March 2015, 19:00:50 UTC [SPARK-5775] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table The Bug solved here was due to a change in PartitionTableScan, when reading a partitioned table. - When the Partititon column is requested out of a parquet table, the Table Scan needs to add the column back to the output Rows. - To update the Row object created by PartitionTableScan, the Row was first casted in SpecificMutableRow, before being updated. - This casting was unsafe, since there are no guarantee that the newHadoopRDD used internally will instanciate the output Rows as MutableRow. Particularly, when reading a Table with complex (e.g. struct or Array) types, the newHadoopRDD uses a parquet.io.api.RecordMateralizer, that is produced by the org.apache.spark.sql.parquet.RowReadSupport . This consumer will be created as a org.apache.spark.sql.parquet.CatalystGroupConverter (a) and not a org.apache.spark.sql.parquet.CatalystPrimitiveRowConverter (b), when there are complex types involved (in the org.apache.spark.sql.parquet.CatalystConverter.createRootConverter factory ) The consumer (a) will output GenericRow, while the consumer (b) produces SpecificMutableRow. Therefore any request selecting a partition columns, plus a complex type column, are returned as GenericRows, and fails into an unsafe casting pit (see https://issues.apache.org/jira/browse/SPARK-5775 for an example. ) The fix proposed here originally replaced the unsafe class casting by a case matching on the Row type, updating the Row if it is of a mutable type, and recreating a Row otherwise. This PR now implements the solution updated by liancheng on aa39460d4bb4c41084d350ccb1c5a56cd61239b7 : The fix checks if every requested requested columns are primitiveType, in a manner symmetrical to the check in org.apache.spark.sql.parquet.CatalystConverter.createRootConverter. - If all columns are primitive type, the Row can safely be casted to a MutableRow. - Otherwise a new GenericRow is created, and the partition column is written this new row structure This fix is unit-tested in sql/hive/src/test/scala/org/apache/spark/sql/parquet/parquetSuites.scala Author: Anselme Vignon <anselme.vignon@flaminem.com> Author: Cheng Lian <lian@databricks.com> Closes #4697 from anselmevignon/local_dev and squashes the following commits: 6a4c53d [Anselme Vignon] style corrections 52f73fc [Cheng Lian] cherry-pick & merge from aa39460d4bb4c41084d350ccb1c5a56cd61239b7 8fc6a8c [Anselme Vignon] correcting tests on temporary tables 24928ea [Anselme Vignon] corrected mirror bug (see SPARK-5775) for newParquet 7c829cb [Anselme Vignon] bugfix, hopefully correct this time 005a7f8 [Anselme Vignon] added test cleanup 22cec52 [Anselme Vignon] lint compatible changes ae48f7c [Anselme Vignon] unittesting SPARK-5775 f876dea [Anselme Vignon] starting to write tests dbceaa3 [Anselme Vignon] cutting lines 4eb04e9 [Anselme Vignon] bugfix SPARK-5775 23 March 2015, 19:00:50 UTC
abdcec6 Andrew Or 04 March 2015, 04:49:45 UTC [SPARK-6132][HOTFIX] ContextCleaner InterruptedException should be quiet If the cleaner is stopped, we shouldn't print a huge stack trace when the cleaner thread is interrupted because we purposefully did this. Author: Andrew Or <andrew@databricks.com> Closes #4882 from andrewor14/cleaner-interrupt and squashes the following commits: 8652120 [Andrew Or] Just a hot fix 22 March 2015, 13:05:11 UTC
06d883c Andrew Or 03 March 2015, 21:44:05 UTC [SPARK-6132] ContextCleaner race condition across SparkContexts The problem is that `ContextCleaner` may clean variables that belong to a different `SparkContext`. This can happen if the `SparkContext` to which the cleaner belongs stops, and a new one is started immediately afterwards in the same JVM. In this case, if the cleaner is in the middle of cleaning a broadcast, for instance, it will do so through `SparkEnv.get.blockManager`, which could be one that belongs to a different `SparkContext`. JoshRosen and I suspect that this is the cause of many flaky tests, most notably the `JavaAPISuite`. We were able to reproduce the failure locally (though it is not deterministic and very hard to reproduce). Author: Andrew Or <andrew@databricks.com> Closes #4869 from andrewor14/cleaner-masquerade and squashes the following commits: 29168c0 [Andrew Or] Synchronize ContextCleaner stop 22 March 2015, 13:05:11 UTC
a2a94a1 nemccarthy 17 March 2015, 16:33:11 UTC [SPARK-6313] Add config option to disable file locks/fetchFile cache to ... ...support NFS mounts. This is a work around for now with the goal to find a more permanent solution. https://issues.apache.org/jira/browse/SPARK-6313 Author: nemccarthy <nathan@nemccarthy.me> Closes #5036 from nemccarthy/master and squashes the following commits: 2eaaf42 [nemccarthy] [SPARK-6313] Update config wording doc for spark.files.useFetchCache 5de7eb4 [nemccarthy] [SPARK-6313] Add config option to disable file locks/fetchFile cache to support NFS mounts (cherry picked from commit 4cca3917dc30ee907e6cbd6a569b6ac58af963f7) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 17 March 2015, 16:33:45 UTC
9ebd6f1 Davies Liu 12 March 2015, 22:19:17 UTC [SPARK-6294] [PySpark] fix take of PythonRDD in JVM (branch 1.2) The Thread.interrupt() can not terminate the thread in some cases, so we should not wait for the writerThread of PythonRDD. This PR also ignore some exception during clean up. cc mengxr Author: Davies Liu <davies@databricks.com> Closes #5003 from davies/fix_take2 and squashes the following commits: 2f2f893 [Davies Liu] fix take of PythonRDD in JVM 12 March 2015, 22:19:17 UTC
c684e5f Yuhao Yang 12 March 2015, 08:40:40 UTC [SPARK-5186][branch-1.2] Vector.hashCode is not efficient Backport hhbyyh 's hasCode implementation to branch-1.2. The old implementation causes performance issues with PySpark, which calls hashCode (https://issues.apache.org/jira/browse/SPARK-6288). Author: Yuhao Yang <hhbyyh@gmail.com> Closes #4985 from mengxr/SPARK-5186-1.2 and squashes the following commits: 155e559 [Yuhao Yang] backport SPARK-5186 12 March 2015, 08:40:40 UTC
d7c359b Davies Liu 09 March 2015, 23:24:06 UTC [SPARK-6194] [SPARK-677] [PySpark] fix memory leak in collect() Because circular reference between JavaObject and JavaMember, an Java object can not be released until Python GC kick in, then it will cause memory leak in collect(), which may consume lots of memory in JVM. This PR change the way we sending collected data back into Python from local file to socket, which could avoid any disk IO during collect, also avoid any referrers of Java object in Python. cc JoshRosen Author: Davies Liu <davies@databricks.com> Closes #4923 from davies/fix_collect and squashes the following commits: d730286 [Davies Liu] address comments 24c92a4 [Davies Liu] fix style ba54614 [Davies Liu] use socket to transfer data from JVM 9517c8f [Davies Liu] fix memory leak in collect() (cherry picked from commit 8767565cef01d847f57b7293d8b63b2422009b90) Signed-off-by: Josh Rosen <joshrosen@databricks.com> Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala python/pyspark/rdd.py python/pyspark/sql/dataframe.py 10 March 2015, 01:19:46 UTC
e753f9c Makoto Fukuhara 23 February 2015, 09:24:33 UTC [EXAMPLES] fix typo. Author: Makoto Fukuhara <fukuo33@gmail.com> Closes #4724 from fukuo33/fix-typo and squashes the following commits: 8c806b9 [Makoto Fukuhara] fix typo. 04 March 2015, 22:40:13 UTC
77a8c06 Sean Owen 04 March 2015, 11:42:50 UTC SPARK-1911 [DOCS] Backport. Warn users if their assembly jars are not built with Java 6 Add warning about building with Java 7+ and running the JAR on early Java 6. Author: Sean Owen <sowen@cloudera.com> Closes #4888 from srowen/SPARK-1911.2 and squashes the following commits: 3f4ed5f [Sean Owen] Add warning about building with Java 7+ and running the JAR on early Java 6. 04 March 2015, 11:42:50 UTC
a91c1c5 Andrew Or 03 March 2015, 23:09:57 UTC [SPARK-6133] Make sc.stop() idempotent Before we would get the following (benign) error if we called `sc.stop()` twice. This is because the listener bus would try to post the end event again even after it has already stopped. This happens occasionally when flaky tests fail, usually as a result of other sources of error. Either way we shouldn't be logging this error when it is not the cause of the failure. ``` ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerApplicationEnd(1425348445682) ``` Author: Andrew Or <andrew@databricks.com> Closes #4871 from andrewor14/sc-stop and squashes the following commits: a14afc5 [Andrew Or] Move code after code 915db16 [Andrew Or] Move code into code (cherry picked from commit 6c20f35290e220e4a659a0222d62575ff959d703) Signed-off-by: Andrew Or <andrew@databricks.com> 03 March 2015, 23:10:03 UTC
eb30fc1 Andrew Or 03 March 2015, 21:04:50 UTC Revert "[SPARK-5423][Core] Cleanup resources in DiskMapIterator.finalize to ensure deleting the temp file" This reverts commit 61bde0049fac324b5004eadfa22b02cd76cf2187. 03 March 2015, 21:04:50 UTC
5226dc7 Sean Owen 01 March 2015, 09:20:36 UTC SPARK-5628 [EC2] Backport: Add version option to spark-ec2 Backport of https://github.com/apache/spark/pull/4414 to 1.2: Add version option to spark-ec2 nchammas JoshRosen : is this about the right backporting of this change? Author: Sean Owen <sowen@cloudera.com> Closes #4833 from srowen/SPARK-5628.2 and squashes the following commits: e723911 [Sean Owen] Backport of https://github.com/apache/spark/pull/4414 to 1.2: Add version option to spark-ec2 01 March 2015, 09:20:36 UTC
576fc54 Davies Liu 28 February 2015, 04:04:16 UTC [SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2) The eq of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released. Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython). This PR also improve the performance of inferSchema (avoid the unnecessary converter of object). Author: Davies Liu <davies@databricks.com> Closes #4809 from davies/leak2 and squashes the following commits: 65c222f [Davies Liu] Update sql.py 9b4dadc [Davies Liu] fix __eq__ of singleton b576107 [Davies Liu] fix tests 6c2909a [Davies Liu] fix incorrect DataType.__eq__ 28 February 2015, 04:04:16 UTC
17b7cc7 Ryan Williams 28 January 2015, 21:04:52 UTC [SPARK-5417] Remove redundant executor-id set() call This happens inside SparkEnv initialization as of #4194 Author: Ryan Williams <ryan.blake.williams@gmail.com> Closes #4213 from ryan-williams/exec-id-set and squashes the following commits: b3e4f7b [Ryan Williams] Remove redundant executor-id set() call 27 February 2015, 15:20:48 UTC
6b22741 Nicholas Chammas 28 January 2015, 20:56:03 UTC [SPARK-5434] [EC2] Preserve spaces in EC2 path Fixes [SPARK-5434](https://issues.apache.org/jira/browse/SPARK-5434). Simple demonstration of the problem and the fix: ``` $ spacey_path="/path/with some/spaces" $ dirname $spacey_path usage: dirname path $ echo $? 1 $ dirname "$spacey_path" /path/with some $ echo $? 0 ``` Author: Nicholas Chammas <nicholas.chammas@gmail.com> Closes #4224 from nchammas/patch-1 and squashes the following commits: 960711a [Nicholas Chammas] [EC2] Preserve spaces in EC2 path 27 February 2015, 13:46:11 UTC
d4ce702 许鹏 27 February 2015, 07:05:56 UTC fix spark-6033, clarify the spark.worker.cleanup behavior in standalone mode jira case spark-6033 https://issues.apache.org/jira/browse/SPARK-6033 In standalone deploy mode, the cleanup will only remove the stopped application's directories. The original description about the cleanup behavior is incorrect. Author: 许鹏 <peng.xu@fraudmetrix.cn> Closes #4803 from hseagle/spark-6033 and squashes the following commits: 927a6a0 [许鹏] fix the incorrect description about the spark.worker.cleanup in standalone mode 27 February 2015, 07:07:18 UTC
d0bf938 Sean Owen 27 February 2015, 01:35:09 UTC SPARK-4579 [WEBUI] Scheduling Delay appears negative Ensure scheduler delay handles unfinished task case, and ensure delay is never negative even due to rounding Author: Sean Owen <sowen@cloudera.com> Closes #4796 from srowen/SPARK-4579 and squashes the following commits: ad6713c [Sean Owen] Ensure scheduler delay handles unfinished task case, and ensure delay is never negative even due to rounding 27 February 2015, 01:35:59 UTC
58b3aa6 moussa taifi 26 February 2015, 22:19:43 UTC Add a note for context termination for History server on Yarn The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway. Related to SPARK-2972 and SPARK-3458 Author: moussa taifi <moutai10@gmail.com> Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits: 9f5b6c3 [moussa taifi] Fix upper case typo for YARN 3ad3db4 [moussa taifi] Add context termination for History server on Yarn (cherry picked from commit c871e2dae0182e914135560d14304242e1f97f7e) Signed-off-by: Andrew Or <andrew@databricks.com> 26 February 2015, 22:20:43 UTC
64e0cbc Sean Owen 26 February 2015, 22:08:56 UTC SPARK-4300 [CORE] Race condition during SparkWorker shutdown Close appender saving stdout/stderr before destroying process to avoid exception on reading closed input stream. (This also removes a redundant `waitFor()` although it was harmless) CC tdas since I think you wrote this method. Author: Sean Owen <sowen@cloudera.com> Closes #4787 from srowen/SPARK-4300 and squashes the following commits: e0cdabf [Sean Owen] Close appender saving stdout/stderr before destroying process to avoid exception on reading closed input stream (cherry picked from commit 3fb53c0298761ba227890525ae79ce4ec6300deb) Signed-off-by: Andrew Or <andrew@databricks.com> 26 February 2015, 22:09:03 UTC
2d83442 Sean Owen 26 February 2015, 22:07:09 UTC SPARK-794 [CORE] Backport. Remove sleep() in ClusterScheduler.stop Backport https://github.com/apache/spark/pull/3851 to branch 1.2: remove Thread.sleep(1000) in TaskSchedulerImpl. Teeing this up for Jenkins per discussion in the JIRA / PR. Author: Sean Owen <sowen@cloudera.com> Closes #4793 from srowen/SPARK-795.2 and squashes the following commits: 5f5db4a [Sean Owen] Backport https://github.com/apache/spark/pull/3851 to branch 1.2: remove Thread.sleep(1000) in TaskSchedulerImpl 26 February 2015, 22:07:09 UTC
e21475d Cheolsoo Park 26 February 2015, 21:53:49 UTC [SPARK-6018] [YARN] NoSuchMethodError in Spark app is swallowed by YARN AM Author: Cheolsoo Park <cheolsoop@netflix.com> Closes #4773 from piaozhexiu/SPARK-6018 and squashes the following commits: 2a919d5 [Cheolsoo Park] Rename e with cause to avoid duplicate names 1e71d2d [Cheolsoo Park] Replace placeholder with throwable eb5750d [Cheolsoo Park] NoSuchMethodError in Spark app is swallowed by YARN AM (cherry picked from commit 5f3238b3b0157091d28803aa3b1d248dfa6cdc59) Signed-off-by: Andrew Or <andrew@databricks.com> 26 February 2015, 21:54:04 UTC
94faf4c Li Zhihui 26 February 2015, 21:07:07 UTC Modify default value description for spark.scheduler.minRegisteredResourcesRatio on docs. The configuration is not supported in mesos mode now. See https://github.com/apache/spark/pull/1462 Author: Li Zhihui <zhihui.li@intel.com> Closes #4781 from li-zhihui/fixdocconf and squashes the following commits: 63e7a44 [Li Zhihui] Modify default value description for spark.scheduler.minRegisteredResourcesRatio on docs. (cherry picked from commit 10094a523e3993b775111ae9b22ca31cc0d76e03) Signed-off-by: Andrew Or <andrew@databricks.com> 26 February 2015, 21:08:02 UTC
602d5c1 Sean Owen 26 February 2015, 20:56:54 UTC SPARK-4704 [CORE] SparkSubmitDriverBootstrap doesn't flush output Join on output threads to make sure any lingering output from process reaches stdout, stderr before exiting CC andrewor14 since I believe he created this section of code Author: Sean Owen <sowen@cloudera.com> Closes #4788 from srowen/SPARK-4704 and squashes the following commits: ad7114e [Sean Owen] Join on output threads to make sure any lingering output from process reaches stdout, stderr before exiting (cherry picked from commit cd5c8d7bbd3ea410df08af6cdd3833e0ed4b91a0) Signed-off-by: Andrew Or <andrew@databricks.com> 26 February 2015, 20:57:02 UTC
cc7313d Davies Liu 26 February 2015, 19:54:17 UTC [SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe Removing elements from a mutable HashSet while iterating over it can cause the iteration to incorrectly skip over entries that were not removed. If this happened, PythonRDD would write fewer broadcast variables than the Python worker was expecting to read, which would cause the Python worker to hang indefinitely. Author: Davies Liu <davies@databricks.com> Closes #4776 from davies/fix_hang and squashes the following commits: a4384a5 [Davies Liu] fix bug: remvoe() inside iterator is not safe (cherry picked from commit 7fa960e653a905fc48d4097b49ce560cff919fa2) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 26 February 2015, 19:57:07 UTC
015895a Davies Liu 26 February 2015, 18:45:29 UTC [SPARK-6015] fix links to source code in Python API docs Author: Davies Liu <davies@databricks.com> Closes #4772 from davies/source_link and squashes the following commits: 389f0c6 [Davies Liu] fix link to source code in Pyton API docs 26 February 2015, 18:45:29 UTC
00112ba Brennon York 25 February 2015, 22:11:12 UTC [SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD. Author: Brennon York <brennon.york@capitalone.com> Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits: 0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs (cherry picked from commit 9f603fce78fcc997926e9a72dec44d48cbc396fc) Signed-off-by: Ankur Dave <ankurdave@gmail.com> 25 February 2015, 22:14:22 UTC
a9abcaa Davies Liu 24 February 2015, 22:50:00 UTC [SPARK-5973] [PySpark] fix zip with two RDDs with AutoBatchedSerializer Author: Davies Liu <davies@databricks.com> Closes #4745 from davies/fix_zip and squashes the following commits: 2124b2c [Davies Liu] Update tests.py b5c828f [Davies Liu] increase the number of records c1e40fd [Davies Liu] fix zip with two RDDs with AutoBatchedSerializer (cherry picked from commit da505e59274d1c838653c1109db65ad374e65304) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> 24 February 2015, 22:50:28 UTC
3ad00ee Tathagata Das 24 February 2015, 19:02:47 UTC [Spark-5967] [UI] Correctly clean JobProgressListener.stageIdToActiveJobIds Patch should be self-explanatory pwendell JoshRosen Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4741 from tdas/SPARK-5967 and squashes the following commits: 653b5bb [Tathagata Das] Fixed the fix and added test e2de972 [Tathagata Das] Clear stages which have no corresponding active jobs. (cherry picked from commit 64d2c01ff1048de83b9b8efce987b55e457298f9) Signed-off-by: Andrew Or <andrew@databricks.com> 24 February 2015, 19:03:02 UTC
2c9d965 Andrew Or 24 February 2015, 18:52:19 UTC Revert "[SPARK-4808] Removing minimum number of elements read before spill check" This reverts commit 5cea859fd27dc6a216fa9d31d293c93407fbff01. 24 February 2015, 18:52:19 UTC
71173de Davies Liu 24 February 2015, 01:29:25 UTC [SPARK-5722] [SQL] [PySpark] infer int as LongType in Python (for 1.2 branch) This PR change to use LongType for int in Python, when inferSchema(), because IntegerType in SQL is not enough for int in Python (which is 64-bit on 64-bit machines). Closes #4521 cc dondrake marmbrus Author: Davies Liu <davies@databricks.com> Closes #4681 from davies/long2 and squashes the following commits: 05ef1c8 [Davies Liu] infer LongType for int in Python 24 February 2015, 01:29:25 UTC
5cea859 mcheah 20 February 2015, 02:09:22 UTC [SPARK-4808] Removing minimum number of elements read before spill check In the general case, Spillable's heuristic of checking for memory stress on every 32nd item after 1000 items are read is good enough. In general, we do not want to be enacting the spilling checks until later on in the job; checking for disk-spilling too early can produce unacceptable performance impact in trivial cases. However, there are non-trivial cases, particularly if each serialized object is large, where checking for the necessity to spill too late would allow the memory to overflow. Consider if every item is 1.5 MB in size, and the heap size is 1000 MB. Then clearly if we only try to spill the in-memory contents to disk after 1000 items are read, we would have already accumulated 1500 MB of RAM and overflowed the heap. Patch #3656 attempted to circumvent this by checking the need to spill on every single item read, but that would cause unacceptable performance in the general case. However, the convoluted cases above should not be forced to be refactored to shrink the data items. Therefore it makes sense that the memory spilling thresholds be configurable. Author: mcheah <mcheah@palantir.com> Closes #4420 from mingyukim/memory-spill-configurable and squashes the following commits: 6e2509f [mcheah] [SPARK-4808] Removing minimum number of elements read before spill check 20 February 2015, 02:10:02 UTC
18fbed5 Zhan Zhang 19 February 2015, 23:13:02 UTC [Spark-5889] Remove pid file after stopping service. Currently the pid file is not deleted, and potentially may cause some problem after service is stopped. The fix remove the pid file after service stopped. Author: Zhan Zhang <zhazhan@gmail.com> Closes #4676 from zhzhan/spark-5889 and squashes the following commits: eb01be1 [Zhan Zhang] solve review comments b4c009e [Zhan Zhang] solve review comments 018110a [Zhan Zhang] spark-5889: remove pid file after stopping service 088d2a2 [Zhan Zhang] squash all commits c1f1fa5 [Zhan Zhang] test (cherry picked from commit ad6b169dee84df175b51933b7a3ad7f0bbc52cf3) Signed-off-by: Sean Owen <sowen@cloudera.com> 19 February 2015, 23:14:56 UTC
856fdcb Cheng Hao 19 February 2015, 20:07:51 UTC [SPARK-5825] [Spark Submit] Remove the double checking instance name when stopping the service `spark-daemon.sh` will confirm the process id by fuzzy matching the class name while stopping the service, however, it will fail if the java process arguments is very long (greater than 4096 characters). This PR looses the check for the service process. Author: Cheng Hao <hao.cheng@intel.com> Closes #4611 from chenghao-intel/stopping_service and squashes the following commits: a0051f6 [Cheng Hao] loosen the process checking while stopping a service (cherry picked from commit 94cdb05ff7e6b8fc5b3a574202ba8bc8e5bbe689) Signed-off-by: Andrew Or <andrew@databricks.com> 19 February 2015, 20:08:03 UTC
61bde00 zsxwing 19 February 2015, 18:37:31 UTC [SPARK-5423][Core] Cleanup resources in DiskMapIterator.finalize to ensure deleting the temp file This PR adds a `finalize` method in DiskMapIterator to clean up the resources even if some exception happens during processing data. Author: zsxwing <zsxwing@gmail.com> Closes #4219 from zsxwing/SPARK-5423 and squashes the following commits: d4b2ca6 [zsxwing] Cleanup resources in DiskMapIterator.finalize to ensure deleting the temp file (cherry picked from commit 90095bf3ce9304d09a32ceffaa99069079071b59) Signed-off-by: Ubuntu <ubuntu@ip-172-31-36-14.us-west-2.compute.internal> 19 February 2015, 18:38:00 UTC
f6ee80b Kay Ousterhout 19 February 2015, 02:03:56 UTC [SPARK-5846] Correctly set job description and pool for SQL jobs This is #4630 but modified for the 1.2 branch, because I'm guessing it makes sense to fix this issue in that branch (again, unless I missed something obvious here...) Author: Kay Ousterhout <kayousterhout@gmail.com> Closes #4631 from kayousterhout/SPARK-5846_1.2.1 and squashes the following commits: ffe8ff2 [Kay Ousterhout] [SPARK-5846] Correctly set job description and pool for SQL jobs 19 February 2015, 02:03:56 UTC
36e15b4 Yin Huai 18 February 2015, 21:59:55 UTC [SPARK-4903][SQL]Backport the bug fix for SPARK-4903 The original fix was a part of https://issues.apache.org/jira/browse/SPARK-4912 (commit https://github.com/apache/spark/commit/6463e0b9e8067cce70602c5c9006a2546856a9d6). Author: Yin Huai <yhuai@databricks.com> Closes #4671 from yhuai/SPARK-4903-1.2 and squashes the following commits: 3168b4b [Yin Huai] Uncache table in drop table command. 18 February 2015, 21:59:55 UTC
068ba45 MechCoder 18 February 2015, 10:13:28 UTC SPARK-4610 addendum: [Minor] [MLlib] Minor doc fix in GBT classification example numClassesForClassification has been renamed to numClasses. Author: MechCoder <manojkumarsivaraj334@gmail.com> Closes #4672 from MechCoder/minor-doc and squashes the following commits: d2ddb7f [MechCoder] Minor doc fix in GBT classification example (cherry picked from commit e79a7a626d9ac2e2474b9d5008c6b5d07df5c6f1) Signed-off-by: Sean Owen <sowen@cloudera.com> 18 February 2015, 10:14:31 UTC
6be36d5 Josh Rosen 17 February 2015, 15:48:27 UTC Revert "[SPARK-5363] [PySpark] check ending mark in non-block way" This reverts commits ac6fe67e1d8bf01ee565f9cc09ad48d88a275829 and c06e42f2c1e5fcf123b466efd27ee4cb53bbed3f. 17 February 2015, 15:51:37 UTC
432ceca Patrick Wendell 17 February 2015, 06:10:39 UTC HOTFIX: Style issue causing build break Caused by #4601 17 February 2015, 06:15:02 UTC
0df26bb Davies Liu 17 February 2015, 04:32:03 UTC [SPARK-5363] [PySpark] check ending mark in non-block way There is chance of dead lock that the Python process is waiting for ending mark from JVM, but which is eaten by corrupted stream. This PR checks the ending mark from Python in non-block way, so it will not blocked by Python process. There is a small chance that the ending mark is sent by Python process but not available right now, then Python process will not be used. cc JoshRosen pwendell Author: Davies Liu <davies@databricks.com> Closes #4601 from davies/freeze and squashes the following commits: e15a8c3 [Davies Liu] update logging 890329c [Davies Liu] Merge branch 'freeze' of github.com:davies/spark into freeze 2bd2228 [Davies Liu] add more logging 656d544 [Davies Liu] Update PythonRDD.scala 05e1085 [Davies Liu] check ending mark in non-block way (cherry picked from commit ac6fe67e1d8bf01ee565f9cc09ad48d88a275829) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 17 February 2015, 04:37:24 UTC
a39da17 Davies Liu 30 January 2015, 01:28:37 UTC [SPARK-5395] [PySpark] fix python process leak while coalesce() Currently, the Python process is released into pool only after the task had finished, it cause many process forked if coalesce() is called. This PR will change it to release the process as soon as read all the data from it (finish the partition), then a process could be reused to process multiple partitions in a single task. Author: Davies Liu <davies@databricks.com> Closes #4238 from davies/py_leak and squashes the following commits: ec80a43 [Davies Liu] add @volatile 6da437a [Davies Liu] address comments 24ed322 [Davies Liu] fix python process leak while coalesce() (cherry picked from commit 5c746eedda8cff2fc1692cf6dce376f4b0ca6fac) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 17 February 2015, 04:35:02 UTC
f468688 Davies Liu 17 February 2015, 01:57:14 UTC [SPARK-5788] [PySpark] capture the exception in python write thread The exception in Python writer thread will shutdown executor. Author: Davies Liu <davies@databricks.com> Closes #4577 from davies/exception and squashes the following commits: eb0ceff [Davies Liu] Update PythonRDD.scala 139b0db [Davies Liu] capture the exception in python write thread (cherry picked from commit b1bd1dd3228ef50fa7310d466afd834b8cb1f22e) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 17 February 2015, 01:57:45 UTC
6f47114 Winston Chen 28 January 2015, 19:08:44 UTC [SPARK-5361]Multiple Java RDD <-> Python RDD conversions not working correctly This is found through reading RDD from `sc.newAPIHadoopRDD` and writing it back using `rdd.saveAsNewAPIHadoopFile` in pyspark. It turns out that whenever there are multiple RDD conversions from JavaRDD to PythonRDD then back to JavaRDD, the exception below happens: ``` 15/01/16 10:28:31 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 7) java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.util.ArrayList at org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:157) at org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:153) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) ``` The test case code below reproduces it: ``` from pyspark.rdd import RDD dl = [ (u'2', {u'director': u'David Lean'}), (u'7', {u'director': u'Andrew Dominik'}) ] dl_rdd = sc.parallelize(dl) tmp = dl_rdd._to_java_object_rdd() tmp2 = sc._jvm.SerDe.javaToPython(tmp) t = RDD(tmp2, sc) t.count() tmp = t._to_java_object_rdd() tmp2 = sc._jvm.SerDe.javaToPython(tmp) t = RDD(tmp2, sc) t.count() # it blows up here during the 2nd time of conversion ``` Author: Winston Chen <wchen@quid.com> Closes #4146 from wingchen/master and squashes the following commits: 903df7d [Winston Chen] SPARK-5361, update to toSeq based on the PR 5d90a83 [Winston Chen] SPARK-5361, make python pretty, so to pass PEP 8 checks 126be6b [Winston Chen] SPARK-5361, add in test case 4cf1187 [Winston Chen] SPARK-5361, add in test case 9f1a097 [Winston Chen] add in tuple handling while converting form python RDD back to JavaRDD (cherry picked from commit 453d7999b88be87bda30d9e73038eb484ee063bd) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 17 February 2015, 00:51:26 UTC
1af7ca1 Michael Nazario 28 January 2015, 21:55:01 UTC [SPARK-5441][pyspark] Make SerDeUtil PairRDD to Python conversions more robust SerDeUtil.pairRDDToPython and SerDeUtil.pythonToPairRDD now both support empty RDDs by checking the result of take(1) instead of calling first which throws an exception. Author: Michael Nazario <mnazario@palantir.com> Closes #4236 from mnazario/feature/empty-first and squashes the following commits: a531c0c [Michael Nazario] Added regression tests for SPARK-5441 e3b2fb6 [Michael Nazario] Added acceptance of the empty case 17 February 2015, 00:31:44 UTC
7f19c7c Josh Rosen 16 February 2015, 23:41:38 UTC [SPARK-1600] Refactor FileInputStream tests to remove Thread.sleep() calls and SystemClock usage (branch-1.2 backport) (This PR backports #3801 into `branch-1.2` (1.2.2)) This patch refactors Spark Streaming's FileInputStream tests to remove uses of Thread.sleep() and SystemClock, which should hopefully resolve some longstanding flakiness in these tests (see SPARK-1600). Key changes: - Modify FileInputDStream to use the scheduler's Clock instead of System.currentTimeMillis(); this allows it to be tested using ManualClock. - Fix a synchronization issue in ManualClock's `currentTime` method. - Add a StreamingTestWaiter class which allows callers to block until a certain number of batches have finished. - Change the FileInputStream tests so that files' modification times are manually set based off of ManualClock; this eliminates many Thread.sleep calls. - Update these tests to use the withStreamingContext fixture. Author: Josh Rosen <joshrosen@databricks.com> Closes #4633 from JoshRosen/spark-1600-b12-backport and squashes the following commits: e5d3dc4 [Josh Rosen] [SPARK-1600] Refactor FileInputStream tests to remove Thread.sleep() calls and SystemClock usage 16 February 2015, 23:41:38 UTC
f9d8c5e Emre Sevinç 14 February 2015, 15:06:45 UTC SPARK-5819 Backported the fix described in SPARK-5805 Fixes SPARK-5819 Author: Emre Sevinç <emre.sevinc@gmail.com> Closes #4605 from emres/SPARK-5819 and squashes the following commits: 33bd4b6 [Emre Sevinç] SPARK-5819 Backported the fix described in SPARK-5805 14 February 2015, 15:06:45 UTC
26410a2 Josh Rosen 14 February 2015, 01:45:31 UTC [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite This patch fixes two difficult-to-reproduce Jenkins test failures in InputOutputMetricsSuite (SPARK-5227 and SPARK-5679). The problem was that WholeTextFileRecordReaderSuite modifies the `fs.local.block.size` Hadoop configuration and this change was affecting subsequent test suites due to Hadoop's caching of FileSystem instances (see HADOOP-8490 for more details). The fix implemented here is to disable FileSystem caching in WholeTextFileRecordReaderSuite. Author: Josh Rosen <joshrosen@databricks.com> Closes #4599 from JoshRosen/inputoutputsuite-fix and squashes the following commits: 47dc447 [Josh Rosen] [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite (cherry picked from commit d06d5ee9b33505774ef1e5becc01b47492f1a2dc) Signed-off-by: Patrick Wendell <patrick@databricks.com> 14 February 2015, 01:45:45 UTC
c2b4633 Sean Owen 13 February 2015, 14:25:54 UTC SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later Backport of SPARK-4267 (https://github.com/apache/spark/commit/de7806048ac49a8bfdf44d8f87bc11cea1dfb242) for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string. Author: Sean Owen <sowen@cloudera.com> Closes #4575 from srowen/SPARK-4267-1.2 and squashes the following commits: 7bfc75b [Sean Owen] Backport of SPARK-4267 for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string. 13 February 2015, 14:25:54 UTC
222ce9f Sean Owen 11 February 2015, 08:13:51 UTC SPARK-5728 [STREAMING] MQTTStreamSuite leaves behind ActiveMQ database files Use temp dir for ActiveMQ database Author: Sean Owen <sowen@cloudera.com> Closes #4517 from srowen/SPARK-5728 and squashes the following commits: 1d3aeb8 [Sean Owen] Use temp dir for ActiveMQ database (cherry picked from commit da89720bf4023392436e75b6ed5e10ed8588a132) Signed-off-by: Sean Owen <sowen@cloudera.com> 13 February 2015, 10:41:33 UTC
a5d72fd Iulian Dragos 02 February 2015, 22:00:33 UTC [SPARK-4631][streaming][FIX] Wait for a receiver to start before publishing test data. This fixes two sources of non-deterministic failures in this test: - wait for a receiver to be up before pushing data through MQTT - gracefully handle the case where the MQTT client is overloaded. There’s a hard-coded limit of 10 in-flight messages, and this test may hit it. Instead of crashing, we retry sending the message. Both of these are needed to make the test pass reliably on my machine. Author: Iulian Dragos <jaguarul@gmail.com> Closes #4270 from dragos/issue/fix-flaky-test-SPARK-4631 and squashes the following commits: f66c482 [Iulian Dragos] [SPARK-4631][streaming] Wait for a receiver to start before publishing test data. d408a8e [Iulian Dragos] Install callback before connecting to MQTT broker. (cherry picked from commit e908322cd5991e6cbdaaafb8cd494759dac01225) Signed-off-by: Sean Owen <sowen@cloudera.com> # Conflicts: # external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala 13 February 2015, 10:40:07 UTC
2b9dbdd WangTaoTheTonic 13 February 2015, 10:27:23 UTC [SPARK-4832][Deploy]some other processes might take the daemon pid Some other processes might use the pid saved in pid file. In that case we should ignore it and launch daemons. JIRA is down for maintenance. I will file one once it return. Author: WangTaoTheTonic <barneystinson@aliyun.com> Author: WangTaoTheTonic <wangtao111@huawei.com> Closes #3683 from WangTaoTheTonic/otherproc and squashes the following commits: daa86a1 [WangTaoTheTonic] some bash style fix 8befee7 [WangTaoTheTonic] handle the mistake scenario cf4ecc6 [WangTaoTheTonic] remove redundant condition f36cfb4 [WangTaoTheTonic] some other processes might take the pid (cherry picked from commit 1768bd51438670c493ca3ca02988aee3ae31e87e) Signed-off-by: Sean Owen <sowen@cloudera.com> 13 February 2015, 10:27:52 UTC
0ba065f Andrew Or 13 February 2015, 00:18:27 UTC Revert "[SPARK-5762] Fix shuffle write time for sort-based shuffle" This reverts commit 9c5454d06e56917521a15697c36f76a33a94dd1e. 13 February 2015, 00:18:27 UTC
d24971a Vladimir Grigor 12 February 2015, 23:26:24 UTC [SPARK-5335] Fix deletion of security groups within a VPC Please see https://issues.apache.org/jira/browse/SPARK-5335. The fix itself is in e58a8b01a8bedcbfbbc6d04b1c1489255865cf87 commit. Two earlier commits are fixes of another VPC related bug waiting to be merged. I should have created former bug fix in own branch then this fix would not have former fixes. :( This code is released under the project's license. Author: Vladimir Grigor <vladimir@kiosked.com> Author: Vladimir Grigor <vladimir@voukka.com> Closes #4122 from voukka/SPARK-5335_delete_sg_vpc and squashes the following commits: 090dca9 [Vladimir Grigor] fixes as per review: removed printing of group_id and added comment 730ec05 [Vladimir Grigor] fix for SPARK-5335: Destroying cluster in VPC with "--delete-groups" fails to remove security groups (cherry picked from commit ada993e954e2825c0fe13326fc23b0e1a567cd55) Signed-off-by: Sean Owen <sowen@cloudera.com> 12 February 2015, 23:26:50 UTC
c7bac57 Davies Liu 12 February 2015, 22:54:38 UTC [SPARK-5780] [PySpark] Mute the logging during unit tests There a bunch of logging coming from driver and worker, it's noisy and scaring, and a lots of exception in it, people are confusing about the tests are failing or not. This PR will mute the logging during tests, only show them if any one failed. Author: Davies Liu <davies@databricks.com> Closes #4572 from davies/mute and squashes the following commits: 1e9069c [Davies Liu] mute the logging during python tests (cherry picked from commit 0bf031582588723dd5a4ca42e6f9f36bc2da1a0b) Signed-off-by: Andrew Or <andrew@databricks.com> 12 February 2015, 22:54:51 UTC
9c5454d Kay Ousterhout 12 February 2015, 22:46:37 UTC [SPARK-5762] Fix shuffle write time for sort-based shuffle mateiz was excluding the time to write this final file from the shuffle write time intentional? Author: Kay Ousterhout <kayousterhout@gmail.com> Closes #4559 from kayousterhout/SPARK-5762 and squashes the following commits: 5c6f3d9 [Kay Ousterhout] Use foreach 94e4237 [Kay Ousterhout] Removed open time metrics added inadvertently ace156c [Kay Ousterhout] Moved metrics to finally block d773276 [Kay Ousterhout] Use nano time 5a59906 [Kay Ousterhout] [SPARK-5762] Fix shuffle write time for sort-based shuffle (cherry picked from commit 47c73d410ab533c3196184d2b6004081e79daeaa) Signed-off-by: Andrew Or <andrew@databricks.com> 12 February 2015, 22:46:53 UTC
b78a686 Venkata Ramana Gollamudi 12 February 2015, 22:44:21 UTC [SPARK-5765][Examples]Fixed word split problem in run-example and compute-classpath Author: Venkata Ramana G <ramana.gollamudihuawei.com> Author: Venkata Ramana Gollamudi <ramana.gollamudi@huawei.com> Closes #4561 from gvramana/word_split and squashes the following commits: 285c8d4 [Venkata Ramana Gollamudi] Fixed word split problem in run-example and compute-classpath (cherry picked from commit 629d0143eeb3c153dac9c65e7b556723c6b4bfc7) Signed-off-by: Andrew Or <andrew@databricks.com> 12 February 2015, 22:44:38 UTC
64254ee Andrew Rowson 12 February 2015, 18:41:39 UTC [SPARK-5655] Don't chmod700 application files if running in YARN [Was previously PR4507] As per SPARK-5655, recently committed code chmod 700s all application files created on the local fs by a spark executor. This is both unnecessary and broken on YARN, where files created in the nodemanager's working directory are already owned by the user running the job and the 'yarn' group. Group read permission is also needed for the auxiliary shuffle service to be able to read the files, as this is running as the 'yarn' user. Author: Andrew Rowson <github@growse.com> Closes #4509 from growse/master and squashes the following commits: 7ca993c [Andrew Rowson] Moved chmod700 functionality into Utils.getOrCreateLocalRootDirs f57ce6b [Andrew Rowson] [SPARK-5655] Don't chmod700 application files if running in a YARN container (cherry picked from commit 466b1f671b21f575d28f9c103f51765790914fe3) Signed-off-by: Sean Owen <sowen@cloudera.com> 12 February 2015, 19:12:16 UTC
53de237 Andrew Or 10 February 2015, 05:18:48 UTC [SPARK-5703] AllJobsPage throws empty.max exception If you have a `SparkListenerJobEnd` event without the corresponding `SparkListenerJobStart` event, then `JobProgressListener` will create an empty `JobUIData` with an empty `stageIds` list. However, later in `AllJobsPage` we call `stageIds.max`. If this is empty, it will throw an exception. This crashed my history server. Author: Andrew Or <andrew@databricks.com> Closes #4490 from andrewor14/jobs-page-max and squashes the following commits: 21797d3 [Andrew Or] Check nonEmpty before calling max (cherry picked from commit a95ed52157473fb0e42e910ee15270e7f0edf943) Signed-off-by: Andrew Or <andrew@databricks.com> 10 February 2015, 05:18:59 UTC
515f658 Andrew Or 10 February 2015, 01:33:29 UTC [SPARK-5698] Do not let user request negative # of executors Otherwise we might crash the ApplicationMaster. Why? Please see https://issues.apache.org/jira/browse/SPARK-5698. sryza I believe this is also relevant in your patch #4168. Author: Andrew Or <andrew@databricks.com> Closes #4483 from andrewor14/da-negative and squashes the following commits: 53ed955 [Andrew Or] Throw IllegalArgumentException instead 0e89fd5 [Andrew Or] Check against negative requests 10 February 2015, 01:34:02 UTC
63eee52 Hari Shreedharan 09 February 2015, 22:17:14 UTC [SPARK-4905][STREAMING] FlumeStreamSuite fix. Using String constructor instead of CharsetDecoder to see if it fixes the issue of empty strings in Flume test output. Author: Hari Shreedharan <hshreedharan@apache.org> Closes #4371 from harishreedharan/Flume-stream-attempted-fix and squashes the following commits: 550d363 [Hari Shreedharan] Fix imports. 8695950 [Hari Shreedharan] Use Charsets.UTF_8 instead of "UTF-8" in String constructors. af3ba14 [Hari Shreedharan] [SPARK-4905][STREAMING] FlumeStreamSuite fix. (cherry picked from commit 0765af9b21e9204c410c7a849c7201bc3eda8cc3) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 09 February 2015, 22:18:59 UTC
97541b2 mcheah 09 February 2015, 18:58:58 UTC [SPARK-5691] Fixing wrong data structure lookup for dupe app registration In Master's registerApplication method, it checks if the application had already registered by examining the addressToWorker hash map. In reality, it should refer to the addressToApp data structure, as this is what really tracks which apps have been registered. 09 February 2015, 21:21:18 UTC
4bad854 Jacek Lewandowski 08 February 2015, 03:16:07 UTC SPARK-5425: Use synchronised methods in system properties to create SparkConf SPARK-5425: Fixed usages of system properties This patch fixes few problems caused by the fact that the Scala wrapper over system properties is not thread-safe and is basically invalid because it doesn't take into account the default values which could have been set in the properties object. The problem is fixed by modifying `Utils.getSystemProperties` method so that it uses `stringPropertyNames` method of the `Properties` class, which is thread-safe (internally it creates a defensive copy in a synchronized method) and returns keys of the properties which were set explicitly and which are defined as defaults. The other related problem, which is fixed here. was in `ResetSystemProperties` mix-in. It created a copy of the system properties in the wrong way. This patch also introduces a test case for thread-safeness of SparkConf creation. Refer to the discussion in https://github.com/apache/spark/pull/4220 for more details. Author: Jacek Lewandowski <lewandowski.jacek@gmail.com> Closes #4221 from jacek-lewandowski/SPARK-5425-1.2 and squashes the following commits: 87951a2 [Jacek Lewandowski] SPARK-5425: Modified Utils.getSystemProperties to return a map of all system properties - explicit + defaults 01dd5cb [Jacek Lewandowski] SPARK-5425: Use SerializationUtils to save properties in ResetSystemProperties trait 94aeacf [Jacek Lewandowski] SPARK-5425: Use synchronised methods in system properties to create SparkConf 08 February 2015, 03:16:07 UTC
d89964f Kashish Jain 06 February 2015, 21:47:23 UTC SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart. [SPARK-5613] Added a catch block to catch the ApplicationNotFoundException. Without this catch block the thread gets killed on occurrence of this exception. This Exception occurs when yarn restarts and tries to find an application id for a spark job which got interrupted due to yarn getting stopped. See the stacktrace in the bug for more details. Author: Kashish Jain <kashish.jain@guavus.com> Closes #4392 from kasjain/branch-1.2 and squashes the following commits: 4831000 [Kashish Jain] SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart. 06 February 2015, 21:47:23 UTC
36f70de GenTang 06 February 2015, 21:27:34 UTC [SPARK-4983] Insert waiting time before tagging EC2 instances The boto API doesn't support tag EC2 instances in the same call that launches them. We add a five-second wait so EC2 has enough time to propagate the information so that the tagging can succeed. Author: GenTang <gen.tang86@gmail.com> Author: Gen TANG <gen.tang86@gmail.com> Closes #3986 from GenTang/spark-4983 and squashes the following commits: 13e257d [Gen TANG] modification of comments 47f06755 [GenTang] print the information ab7a931 [GenTang] solve the issus spark-4983 by inserting waiting time 3179737 [GenTang] Revert "handling exceptions about adding tags to ec2" 6a8b53b [GenTang] Revert "the improvement of exception handling" 13e97a6 [GenTang] Revert "typo" 63fd360 [GenTang] typo 692fc2b [GenTang] the improvement of exception handling 6adcf6d [GenTang] handling exceptions about adding tags to ec2 (cherry picked from commit 0f3a36071a44e986d97981032d5b192477b38bbd) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 06 February 2015, 21:29:10 UTC
09da688 Zhang, Liye 06 February 2015, 19:49:40 UTC [SPARK-4989][CORE] backport for branch-1.2 catch eventlog exception for wrong eventlog conf JIRA is [SPARK-4989](https://issues.apache.org/jira/browse/SPARK-4989) Author: Zhang, Liye <liye.zhang@intel.com> Closes #3969 from liyezhang556520/apache-branch-1.2 and squashes the following commits: 5c77e42 [Zhang, Liye] catch eventlog exception for wrong eventlog conf 06 February 2015, 19:49:40 UTC
f318af0 Liang-Chi Hsieh 04 February 2015, 08:52:41 UTC [Minor] Fix incorrect warning log The warning log looks incorrect. Just fix it. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #4360 from viirya/fixing_typo and squashes the following commits: 48fbe4f [Liang-Chi Hsieh] Fix incorrect warning log. (cherry picked from commit a74cbbf12fa59df37eb7172652138c78707d33d8) Signed-off-by: Tathagata Das <tdas@databricks.com> 04 February 2015, 08:53:06 UTC
3799763 Davies Liu 04 February 2015, 06:30:23 UTC [SPARK-4939] revive offers periodically in LocalBackend The locality timeout assume that the SchedulerBackend can revive offers periodically, but currently LocalBackend did do that, then some job with mixed locality levels in local mode will hang forever. This PR let LocalBackend revive offers periodically, just like in cluster mode. Author: Davies Liu <davies@databricks.com> Closes #4147 from davies/revive and squashes the following commits: 2acdf9d [Davies Liu] Update LocalBackend.scala 3c8ca7c [Davies Liu] Update LocalBackend.scala d1b60d2 [Davies Liu] address comments from Kay 33ac9bb [Davies Liu] fix build d0da0d5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into revive 6cf5972 [Davies Liu] fix thread-safety ed62a31 [Davies Liu] fix scala style df9008b [Davies Liu] fix typo bfc1396 [Davies Liu] revive offers periodically in LocalBackend 04 February 2015, 06:32:34 UTC
62c7587 Jesper Lundgren 03 February 2015, 22:53:39 UTC [STREAMING] SPARK-4986 Wait for receivers to deregister and receiver job to terminate A slow receiver might not have enough time to shutdown cleanly even when graceful shutdown is used. This PR extends graceful waiting to make sure all receivers have deregistered and that the receiver job has terminated. Author: Jesper Lundgren <jesper.lundgren@vpon.com> Closes #4338 from cleaton/stopreceivers and squashes the following commits: a9cf223 [Jesper Lundgren] remove cleaner.ttl config f969b6e [Jesper Lundgren] fix inversed logic in unit test 3d0bd35 [Jesper Lundgren] switch boleans to match running status instead of terminated 9a9ff88 [Jesper Lundgren] wait for receivers to shutdown and receiver job to terminate d179372 [Jesper Lundgren] Add graceful shutdown unit test covering slow receiver onStop (cherry picked from commit 1e8b5394b44a0d3b36f64f10576c3ae3b977810c) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com> 03 February 2015, 22:54:04 UTC
36c2994 Tathagata Das 03 February 2015, 21:46:02 UTC [SPARK-5153][Streaming][Test] Increased timeout to deal with flaky KafkaStreamSuite Timeout increased to allow overloaded Jenkins to cope with delay in topic creation. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4342 from tdas/SPARK-5153 and squashes the following commits: dc42762 [Tathagata Das] Increased timeout to deal with delays in overloaded Jenkins. (cherry picked from commit 681f9df47ff40f7b0d9175d835e9758d33a13a06) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com> 03 February 2015, 21:46:24 UTC
591cd83 Patrick Wendell 03 February 2015, 00:39:28 UTC Preparing development version 1.2.2-SNAPSHOT 03 February 2015, 00:39:28 UTC
b6eaf77 Patrick Wendell 03 February 2015, 00:39:27 UTC Preparing Spark release v1.2.1-rc3 03 February 2015, 00:39:27 UTC
a64c7a8 Patrick Wendell 03 February 2015, 00:38:44 UTC Revert "Preparing Spark release v1.2.1-rc2" This reverts commit b77f87673d1f9f03d4c83cf583158227c551359b. 03 February 2015, 00:38:44 UTC
d944c0b Patrick Wendell 03 February 2015, 00:38:42 UTC Revert "Preparing development version 1.2.2-SNAPSHOT" This reverts commit 0a16abadc59082b7d3a24d7f3625236658632813. 03 February 2015, 00:38:42 UTC
88e0f2d Patrick Wendell 03 February 2015, 00:33:46 UTC Revert "[SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name)" This reverts commit 54864403c4f132d9c1380c015122a849dd44dff8. 03 February 2015, 00:33:46 UTC
5486440 seayi 03 February 2015, 00:06:52 UTC [SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes #3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala 03 February 2015, 00:09:10 UTC
b978c9f Martin Weindel 02 February 2015, 21:46:18 UTC Disabling Utils.chmod700 for Windows This patch makes Spark 1.2.1rc2 work again on Windows. Without it you get following log output on creating a Spark context: INFO org.apache.spark.SparkEnv:59 - Registering BlockManagerMaster ERROR org.apache.spark.util.Utils:75 - Failed to create local root dir in .... Ignoring this directory. ERROR org.apache.spark.storage.DiskBlockManager:75 - Failed to create any local dir. Author: Martin Weindel <martin.weindel@gmail.com> Author: mweindel <m.weindel@usu-software.de> Closes #4299 from MartinWeindel/branch-1.2 and squashes the following commits: 535cb7f [Martin Weindel] fixed last commit f17072e [Martin Weindel] moved condition to caller to avoid confusion on chmod700() return value 4de5e91 [Martin Weindel] reverted to unix line ends fe2740b [mweindel] moved comment ac4749c [mweindel] fixed chmod700 for Windows 02 February 2015, 21:46:18 UTC
00746a5 Nicholas Chammas 02 February 2015, 20:33:49 UTC [Docs] Fix Building Spark link text Author: Nicholas Chammas <nicholas.chammas@gmail.com> Closes #4312 from nchammas/patch-2 and squashes the following commits: 9d943aa [Nicholas Chammas] [Docs] Fix Building Spark link text (cherry picked from commit 3f941b68a2336aa7876aeda99865e7c19b53bc5c) Signed-off-by: Andrew Or <andrew@databricks.com> 02 February 2015, 20:33:56 UTC
0a16aba Patrick Wendell 28 January 2015, 07:48:55 UTC Preparing development version 1.2.2-SNAPSHOT 28 January 2015, 07:48:55 UTC
b77f876 Patrick Wendell 28 January 2015, 07:48:55 UTC Preparing Spark release v1.2.1-rc2 28 January 2015, 07:48:55 UTC
4026bba Patrick Wendell 28 January 2015, 07:47:00 UTC Revert "Preparing Spark release v1.2.1-rc1" This reverts commit 3e2d7d310b76c293b9ac787f204e6880f508f6ec. 28 January 2015, 07:47:00 UTC
063a4c5 Patrick Wendell 28 January 2015, 07:46:57 UTC Revert "Preparing development version 1.2.2-SNAPSHOT" This reverts commit f53a4319ba5f0843c077e64ae5a41e2fac835a5b. 28 January 2015, 07:46:57 UTC
fea9b43 Davies Liu 27 January 2015, 23:33:01 UTC [MLlib] fix python example of ALS in guide fix python example of ALS in guide, use Rating instead of np.array. Author: Davies Liu <davies@databricks.com> Closes #4226 from davies/fix_als_guide and squashes the following commits: 1433d76 [Davies Liu] fix python example of als in guide (cherry picked from commit fdaad4eb0388cfe43b5b6600927eb7b9182646f9) Signed-off-by: Xiangrui Meng <meng@databricks.com> 27 January 2015, 23:33:15 UTC
8090448 Sean Owen 27 January 2015, 18:22:50 UTC SPARK-5308 [BUILD] MD5 / SHA1 hash format doesn't match standard Maven output Here's one way to make the hashes match what Maven's plugins would create. It takes a little extra footwork since OS X doesn't have the same command line tools. An alternative is just to make Maven output these of course - would that be better? I ask in case there is a reason I'm missing, like, we need to hash files that Maven doesn't build. Author: Sean Owen <sowen@cloudera.com> Closes #4161 from srowen/SPARK-5308 and squashes the following commits: 70d09d0 [Sean Owen] Use $(...) syntax e25eff8 [Sean Owen] Generate MD5, SHA1 hashes in a format like Maven's plugin (cherry picked from commit ff356e2a21e31998cda3062e560a276a3bfaa7ab) Signed-off-by: Patrick Wendell <patrick@databricks.com> 27 January 2015, 18:22:57 UTC
f53a431 Patrick Wendell 27 January 2015, 01:07:29 UTC Preparing development version 1.2.2-SNAPSHOT 27 January 2015, 01:07:29 UTC
3e2d7d3 Patrick Wendell 27 January 2015, 01:07:29 UTC Preparing Spark release v1.2.1-rc1 27 January 2015, 01:07:29 UTC
8c46100 Patrick Wendell 27 January 2015, 01:06:22 UTC Revert "Preparing Spark release v1.2.1-rc1" This reverts commit e87eb2b42f137c22194cfbca2abf06fecdf943da. 27 January 2015, 01:06:22 UTC
e8da342 Patrick Wendell 27 January 2015, 01:06:19 UTC Revert "Preparing development version 1.2.2-SNAPSHOT" This reverts commit adfed7086f10fa8db4eeac7996c84cf98f625e9a. 27 January 2015, 01:06:19 UTC
adfed70 Ubuntu 27 January 2015, 00:12:04 UTC Preparing development version 1.2.2-SNAPSHOT 27 January 2015, 00:12:04 UTC
e87eb2b Ubuntu 27 January 2015, 00:12:04 UTC Preparing Spark release v1.2.1-rc1 27 January 2015, 00:12:04 UTC
back to top