Skip to main content

Browse the archive

https://github.com/apache/spark
28 July 2024, 20:17:12 UTC
sort by:
Revision Author Date Message Commit Date
54b1121 Patrick Wendell 19 June 2016, 21:06:21 UTC Preparing Spark release v1.6.2-rc2 19 June 2016, 21:06:21 UTC
3d569d9 Davies Liu 19 June 2016, 16:30:59 UTC Revert "[SPARK-15613] [SQL] Fix incorrect days to millis conversion due to Daylight Saving Time" This reverts commit 41efd2091781b31118c6d37be59e4f0f4ae2bf66. 19 June 2016, 16:30:59 UTC
41efd20 Davies Liu 19 June 2016, 07:34:52 UTC [SPARK-15613] [SQL] Fix incorrect days to millis conversion due to Daylight Saving Time ## What changes were proposed in this pull request? Internally, we use Int to represent a date (the days since 1970-01-01), when we convert that into unix timestamp (milli-seconds since epoch in UTC), we get the offset of a timezone using local millis (the milli-seconds since 1970-01-01 in a timezone), but TimeZone.getOffset() expect unix timestamp, the result could be off by one hour (in Daylight Saving Time (DST) or not). This PR change to use best effort approximate of posix timestamp to lookup the offset. In the event of changing of DST, Some time is not defined (for example, 2016-03-13 02:00:00 PST), or could lead to multiple valid result in UTC (for example, 2016-11-06 01:00:00), this best effort approximate should be enough in practice. ## How was this patch tested? Added regression tests. Author: Davies Liu <davies@databricks.com> Closes #13652 from davies/fix_timezone. (cherry picked from commit 001a58960311b07fe80e2f01e473f4987948d06e) Signed-off-by: Davies Liu <davies.liu@gmail.com> 19 June 2016, 07:35:17 UTC
3f1d730 andreapasqua 18 June 2016, 05:41:05 UTC [SPARK-16035][PYSPARK] Fix SparseVector parser assertion for end parenthesis ## What changes were proposed in this pull request? The check on the end parenthesis of the expression to parse was using the wrong variable. I corrected that. ## How was this patch tested? Manual test Author: andreapasqua <andrea@radius.com> Closes #13750 from andreapasqua/sparse-vector-parser-assertion-fix. (cherry picked from commit 4c64e88d5ba4c36cbdbc903376492f0f43401e4e) Signed-off-by: Xiangrui Meng <meng@databricks.com> 18 June 2016, 05:41:28 UTC
fd05389 hyukjinkwon 18 June 2016, 04:04:24 UTC [SPARK-15892][ML] Backport correctly merging AFTAggregators to branch 1.6 ## What changes were proposed in this pull request? This PR backports https://github.com/apache/spark/pull/13619. The original test added in branch-2.0 was failed in branch-1.6. This seems because the behaviour was changed in https://github.com/apache/spark/commit/101663f1ae222a919fc40510aa4f2bad22d1be6f. This was failure while calculating Euler's number which ends up with a infinity regardless of this path. So, I brought the dataset from `AFTSurvivalRegressionExample` to make sure this is working and then wrote the test. I ran the test before/after creating empty partitions. `model.scale` becomes `1.0` with empty partitions and becames `1.547` without them. After this patch, this becomes always `1.547`. ## How was this patch tested? Unit test in `AFTSurvivalRegressionSuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #13725 from HyukjinKwon/SPARK-15892-1-6. 18 June 2016, 04:04:24 UTC
e530823 Shixiong Zhu 17 June 2016, 20:33:31 UTC Revert "[SPARK-15395][CORE] Use getHostString to create RpcAddress (backport for 1.6)" This reverts commit 7ad82b663092615b02bef3991fb1a21af77d2358. See SPARK-16017. 17 June 2016, 20:33:43 UTC
4621fe9 Patrick Wendell 16 June 2016, 23:40:26 UTC Preparing development version 1.6.3-SNAPSHOT 16 June 2016, 23:40:26 UTC
4168d9c Patrick Wendell 16 June 2016, 23:40:19 UTC Preparing Spark release v1.6.2-rc1 16 June 2016, 23:40:19 UTC
b8f380f Patrick Wendell 16 June 2016, 23:35:51 UTC Preparing development version 1.6.3-SNAPSHOT 16 June 2016, 23:35:51 UTC
f166493 Patrick Wendell 16 June 2016, 23:35:44 UTC Preparing Spark release v1.6.2 16 June 2016, 23:35:44 UTC
a4485c3 Reynold Xin 16 June 2016, 23:30:18 UTC Update branch-1.6 for 1.6.2 release. 16 June 2016, 23:30:18 UTC
0a8ada5 Josh Rosen 16 June 2016, 21:18:58 UTC [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests In the `dev/run-tests.py` script we check a `Popen.retcode` for success using `retcode > 0`, but this is subtlety wrong because Popen's return code will be negative if the child process was terminated by a signal: https://docs.python.org/2/library/subprocess.html#subprocess.Popen.returncode In order to properly handle signals, we should change this to check `retcode != 0` instead. Author: Josh Rosen <joshrosen@databricks.com> Closes #13692 from JoshRosen/dev-run-tests-return-code-handling. (cherry picked from commit acef843f67e770f0a2709fb3fbd1a53c200b2bc5) Signed-off-by: Andrew Or <andrew@databricks.com> 16 June 2016, 21:19:19 UTC
cffc080 Takuya UESHIN 15 June 2016, 17:05:19 UTC [SPARK-15915][SQL] Logical plans should use subqueries eliminated plan when override sameResult. ## What changes were proposed in this pull request? This pr is a backport of #13638 for `branch-1.6`. ## How was this patch tested? Added the same test as #13638 modified for `branch-1.6`. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #13668 from ueshin/issues/SPARK-15915_1.6. 15 June 2016, 17:05:19 UTC
2f3e327 Joseph K. Bradley 14 June 2016, 21:08:33 UTC Revert "[SPARK-15892][ML] Incorrectly merged AFTAggregator with zero total count" This reverts commit be3c41b2633215ff6f20885c04f288aab25a1712. 14 June 2016, 21:08:33 UTC
be3c41b hyukjinkwon 12 June 2016, 21:26:53 UTC [SPARK-15892][ML] Incorrectly merged AFTAggregator with zero total count ## What changes were proposed in this pull request? Currently, `AFTAggregator` is not being merged correctly. For example, if there is any single empty partition in the data, this creates an `AFTAggregator` with zero total count which causes the exception below: ``` IllegalArgumentException: u'requirement failed: The number of instances should be greater than 0.0, but got 0.' ``` Please see [AFTSurvivalRegression.scala#L573-L575](https://github.com/apache/spark/blob/6ecedf39b44c9acd58cdddf1a31cf11e8e24428c/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala#L573-L575) as well. Just to be clear, the python example `aft_survival_regression.py` seems using 5 rows. So, if there exist partitions more than 5, it throws the exception above since it contains empty partitions which results in an incorrectly merged `AFTAggregator`. Executing `bin/spark-submit examples/src/main/python/ml/aft_survival_regression.py` on a machine with CPUs more than 5 is being failed because it creates tasks with some empty partitions with defualt configurations (AFAIK, it sets the parallelism level to the number of CPU cores). ## How was this patch tested? An unit test in `AFTSurvivalRegressionSuite.scala` and manually tested by `bin/spark-submit examples/src/main/python/ml/aft_survival_regression.py`. Author: hyukjinkwon <gurwls223@gmail.com> Author: Hyukjin Kwon <gurwls223@gmail.com> Closes #13619 from HyukjinKwon/SPARK-15892. (cherry picked from commit e3554605b36bdce63ac180cc66dbdee5c1528ec7) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> 12 June 2016, 21:27:20 UTC
393f4ba Mortada Mehyar 10 June 2016, 07:23:34 UTC [DOCUMENTATION] fixed groupby aggregation example for pyspark ## What changes were proposed in this pull request? fixing documentation for the groupby/agg example in python ## How was this patch tested? the existing example in the documentation dose not contain valid syntax (missing parenthesis) and is not using `Column` in the expression for `agg()` after the fix here's how I tested it: ``` In [1]: from pyspark.sql import Row In [2]: import pyspark.sql.functions as func In [3]: %cpaste Pasting code; enter '--' alone on the line to stop or use Ctrl-D. :records = [{'age': 19, 'department': 1, 'expense': 100}, : {'age': 20, 'department': 1, 'expense': 200}, : {'age': 21, 'department': 2, 'expense': 300}, : {'age': 22, 'department': 2, 'expense': 300}, : {'age': 23, 'department': 3, 'expense': 300}] :-- In [4]: df = sqlContext.createDataFrame([Row(**d) for d in records]) In [5]: df.groupBy("department").agg(df["department"], func.max("age"), func.sum("expense")).show() +----------+----------+--------+------------+ |department|department|max(age)|sum(expense)| +----------+----------+--------+------------+ | 1| 1| 20| 300| | 2| 2| 22| 600| | 3| 3| 23| 300| +----------+----------+--------+------------+ Author: Mortada Mehyar <mortada.mehyar@gmail.com> Closes #13587 from mortada/groupby_agg_doc_fix. (cherry picked from commit 675a73715d3c8adb9d9a9dce5f76a2db5106790c) Signed-off-by: Reynold Xin <rxin@databricks.com> 10 June 2016, 07:23:49 UTC
739d992 Josh Rosen 09 June 2016, 18:04:08 UTC [SPARK-15827][BUILD] Publish Spark's forked sbt-pom-reader to Maven Central Spark's SBT build currently uses a fork of the sbt-pom-reader plugin but depends on that fork via a SBT subproject which is cloned from https://github.com/scrapcodes/sbt-pom-reader/tree/ignore_artifact_id. This unnecessarily slows down the initial build on fresh machines and is also risky because it risks a build breakage in case that GitHub repository ever changes or is deleted. In order to address these issues, I have published a pre-built binary of our forked sbt-pom-reader plugin to Maven Central under the `org.spark-project` namespace and have updated Spark's build to use that artifact. This published artifact was built from https://github.com/JoshRosen/sbt-pom-reader/tree/v1.0.0-spark, which contains the contents of ScrapCodes's branch plus an additional patch to configure the build for artifact publication. /cc srowen ScrapCodes for review. Author: Josh Rosen <joshrosen@databricks.com> Closes #13564 from JoshRosen/use-published-fork-of-pom-reader. (cherry picked from commit f74b77713e17960dddb7459eabfdc19f08f4024b) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 09 June 2016, 18:08:22 UTC
bb917fc Josh Rosen 09 June 2016, 07:51:24 UTC [SPARK-12712] Fix failure in ./dev/test-dependencies when run against empty .m2 cache This patch fixes a bug in `./dev/test-dependencies.sh` which caused spurious failures when the script was run on a machine with an empty `.m2` cache. The problem was that extra log output from the dependency download was conflicting with the grep / regex used to identify the classpath in the Maven output. This patch fixes this issue by adjusting the regex pattern. Tested manually with the following reproduction of the bug: ``` rm -rf ~/.m2/repository/org/apache/commons/ ./dev/test-dependencies.sh ``` Author: Josh Rosen <joshrosen@databricks.com> Closes #13568 from JoshRosen/SPARK-12712. (cherry picked from commit 921fa40b14082bfd1094fa49fb3b0c46a79c1aaa) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 09 June 2016, 07:52:43 UTC
5830828 Jason Lee 15 January 2016, 12:04:05 UTC [SPARK-12655][GRAPHX] GraphX does not unpersist RDDs Some VertexRDD and EdgeRDD are created during the intermediate step of g.connectedComponents() but unnecessarily left cached after the method is done. The fix is to unpersist these RDDs once they are no longer in use. A test case is added to confirm the fix for the reported bug. Author: Jason Lee <cjlee@us.ibm.com> Closes #10713 from jasoncl/SPARK-12655. (cherry picked from commit d0a5c32bd05841f411a342a80c5da9f73f30d69a) Signed-off-by: Sean Owen <sowen@cloudera.com> 07 June 2016, 08:25:04 UTC
6a9f19d Brett Randall 05 June 2016, 14:31:56 UTC [SPARK-15723] Fixed local-timezone-brittle test where short-timezone form "EST" is … ## What changes were proposed in this pull request? Stop using the abbreviated and ambiguous timezone "EST" in a test, since it is machine-local default timezone dependent, and fails in different timezones. Fixed [SPARK-15723](https://issues.apache.org/jira/browse/SPARK-15723). ## How was this patch tested? Note that to reproduce this problem in any locale/timezone, you can modify the scalatest-maven-plugin argLine to add a timezone: <argLine>-ea -Xmx3g -XX:MaxPermSize=${MaxPermGen} -XX:ReservedCodeCacheSize=${CodeCacheSize} -Duser.timezone="Australia/Sydney"</argLine> and run $ mvn test -DwildcardSuites=org.apache.spark.status.api.v1.SimpleDateParamSuite -Dtest=none. Equally this will fix it in an effected timezone: <argLine>-ea -Xmx3g -XX:MaxPermSize=${MaxPermGen} -XX:ReservedCodeCacheSize=${CodeCacheSize} -Duser.timezone="America/New_York"</argLine> To test the fix, apply the above change to `pom.xml` to set test TZ to `Australia/Sydney`, and confirm the test now passes. Author: Brett Randall <javabrett@gmail.com> Closes #13462 from javabrett/SPARK-15723-SimpleDateParamSuite. (cherry picked from commit 4e767d0f9042bfea6074c2637438859699ec4dc3) Signed-off-by: Sean Owen <sowen@cloudera.com> 05 June 2016, 15:12:49 UTC
a0cf7d0 Subroto Sanyal 03 June 2016, 23:50:00 UTC [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential. ## What changes were proposed in this pull request? The credentials are not added to the credentials of UserGroupInformation.getCurrentUser(). Further if the client has possibility to login using keytab then the updateDelegationToken thread is not started on client. ## How was this patch tested? ran dev/run-tests Author: Subroto Sanyal <ssanyal@datameer.com> Closes #13499 from subrotosanyal/SPARK-15754-save-ugi-from-changing. (cherry picked from commit 61d729abdaf9bdd54aea74a643828d570b036d87) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> 03 June 2016, 23:50:22 UTC
4259a28 Josh Rosen 03 June 2016, 00:47:31 UTC [SPARK-15736][CORE][BRANCH-1.6] Gracefully handle loss of DiskStore files If an RDD partition is cached on disk and the DiskStore file is lost, then reads of that cached partition will fail and the missing partition is supposed to be recomputed by a new task attempt. In the current BlockManager implementation, however, the missing file does not trigger any metadata updates / does not invalidate the cache, so subsequent task attempts will be scheduled on the same executor and the doomed read will be repeatedly retried, leading to repeated task failures and eventually a total job failure. In order to fix this problem, the executor with the missing file needs to properly mark the corresponding block as missing so that it stops advertising itself as a cache location for that block. This patch fixes this bug and adds an end-to-end regression test (in `FailureSuite`) and a set of unit tests (`in BlockManagerSuite`). This is a branch-1.6 backport of #13473. Author: Josh Rosen <joshrosen@databricks.com> Closes #13479 from JoshRosen/handle-missing-cache-files-branch-1.6. 03 June 2016, 00:47:31 UTC
0a13e4c Kevin McHale 02 June 2016, 16:17:33 UTC [SPARK-14204][SQL] register driverClass rather than user-specified class This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an `IllegalStateException`. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. Author: Kevin McHale <kevin@premise.com> Closes #12000 from mchalek/jdbc-driver-registration. 02 June 2016, 16:17:33 UTC
714f4d7 Tejas Patil 01 June 2016, 00:52:22 UTC [SPARK-15601][CORE] CircularBuffer's toString() to print only the contents written if buffer isn't full 1. The class allocated 4x space than needed as it was using `Int` to store the `Byte` values 2. If CircularBuffer isn't full, currently toString() will print some garbage chars along with the content written as is tries to print the entire array allocated for the buffer. The fix is to keep track of buffer getting full and don't print the tail of the buffer if it isn't full (suggestion by sameeragarwal over https://github.com/apache/spark/pull/12194#discussion_r64495331) 3. Simplified `toString()` Added new test case Author: Tejas Patil <tejasp@fb.com> Closes #13351 from tejasapatil/circular_buffer. (cherry picked from commit ac38bdc756c25632069e7887a657250fe2fd6d82) Signed-off-by: Sean Owen <sowen@cloudera.com> 01 June 2016, 00:58:17 UTC
ea84b33 Takeshi YAMAMURO 31 May 2016, 12:25:16 UTC [SPARK-15528][SQL] Fix race condition in NumberConverter ## What changes were proposed in this pull request? A local variable in NumberConverter is wrongly shared between threads. This pr fixes the race condition. ## How was this patch tested? Manually checked. Author: Takeshi YAMAMURO <linguin.m.s@gmail.com> Closes #13391 from maropu/SPARK-15528. (cherry picked from commit 95db8a44f3e2d79913cbe0d29297796b4c3b0d1b) Signed-off-by: Sean Owen <sowen@cloudera.com> 31 May 2016, 12:28:16 UTC
c53c83c Sameer Agarwal 27 May 2016, 01:49:54 UTC [BUILD][1.6] Fix compilation ## What changes were proposed in this pull request? Makes `UnsafeSortDataFormat` and `RecordPointerAndKeyPrefix` public. These are already public in 2.0 and are used in an `ExternalSorterSuite` test (see https://github.com/apache/spark/commit/0b8bdf793a98296fd1ac1fc499946929c6a5959d) ## How was this patch tested? Successfully builds locally Author: Sameer Agarwal <sameer@databricks.com> Closes #13339 from sameeragarwal/fix-compile. 27 May 2016, 01:49:54 UTC
0b8bdf7 Sameer Agarwal 26 May 2016, 22:49:16 UTC [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort This patch fixes a few integer overflows in `UnsafeSortDataFormat.copyRange()` and `ShuffleSortDataFormat copyRange()` that seems to be the most likely cause behind a number of `TimSort` contract violation errors seen in Spark 2.0 and Spark 1.6 while sorting large datasets. Added a test in `ExternalSorterSuite` that instantiates a large array of the form of [150000000, 150000001, 150000002, ...., 300000000, 0, 1, 2, ..., 149999999] that triggers a `copyRange` in `TimSort.mergeLo` or `TimSort.mergeHi`. Note that the input dataset should contain at least 268.43 million rows with a certain data distribution for an overflow to occur. Author: Sameer Agarwal <sameer@databricks.com> Closes #13336 from sameeragarwal/timsort-bug. (cherry picked from commit fe6de16f781ff659b34e0ddda427d371d3d94536) Signed-off-by: Reynold Xin <rxin@databricks.com> 26 May 2016, 22:50:50 UTC
5cc1e2c Simon Scott 26 May 2016, 13:13:28 UTC [SPARK-10722] RDDBlockId not found in driver-heartbeater ## What changes were proposed in this pull request? To ensure that the deserialization of TaskMetrics uses a ClassLoader that knows about RDDBlockIds. The problem occurs only very rarely since it depends on which thread of the thread pool is used for the heartbeat. I observe that the code in question has been largely rewritten for v2.0.0 of Spark and the problem no longer manifests. However it would seem reasonable to fix this for those users who need to continue with the 1.6 version for some time yet. Hence I have created a fix for the 1.6 code branch. ## How was this patch tested? Due to the nature of the problem a reproducible testcase is difficult to produce. This problem was causing our application's nightly integration tests to fail randomly. Since applying the fix the tests have not failed due to this problem, for nearly six weeks now. Author: Simon Scott <simon.scott@viavisolutions.com> Closes #13222 from simonjscott/fix-10722. 26 May 2016, 13:13:28 UTC
9a18115 Kousuke Saruta 20 May 2016, 22:50:06 UTC [SPARK-15165] [SPARK-15205] [SQL] Introduce place holder for comments in generated code (branch-1.6) ## What changes were proposed in this pull request? This PR introduce place holder for comment in generated code and the purpose is same for #12939 but much safer. Generated code to be compiled doesn't include actual comments but includes place holder instead. Place holders in generated code will be replaced with actual comments only at the time of logging. Also, this PR can resolve SPARK-15205. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Added new test cases. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #13230 from sarutak/SPARK-15165-branch-1.6. 20 May 2016, 22:50:06 UTC
7ad82b6 Shixiong Zhu 20 May 2016, 16:56:50 UTC [SPARK-15395][CORE] Use getHostString to create RpcAddress (backport for 1.6) ## What changes were proposed in this pull request? Backport #13185 to branch 1.6. ## How was this patch tested? Jenkins unit tests. Author: Shixiong Zhu <shixiong@databricks.com> Closes #13196 from zsxwing/host-string-1.6. 20 May 2016, 16:56:50 UTC
7200e6b Oleg Danilov 20 May 2016, 05:23:28 UTC [SPARK-14261][SQL] Memory leak in Spark Thrift Server Fixed memory leak (HiveConf in the CommandProcessorFactory) Author: Oleg Danilov <oleg.danilov@wandisco.com> Closes #12932 from dosoft/SPARK-14261. (cherry picked from commit e384c7fbb94cef3c18e8fa8d06159b76b88b5167) Signed-off-by: Reynold Xin <rxin@databricks.com> 20 May 2016, 05:25:25 UTC
fd2da7b Andrew Or 12 May 2016, 00:25:57 UTC [SPARK-15260] Atomically resize memory pools (branch 1.6) ## What changes were proposed in this pull request? (This is the branch-1.6 version of #13039) When we acquire execution memory, we do a lot of things between shrinking the storage memory pool and enlarging the execution memory pool. In particular, we call memoryStore.evictBlocksToFreeSpace, which may do a lot of I/O and can throw exceptions. If an exception is thrown, the pool sizes on that executor will be in a bad state. This patch minimizes the things we do between the two calls to make the resizing more atomic. ## How was this patch tested? Jenkins. Author: Andrew Or <andrew@databricks.com> Closes #13058 from andrewor14/safer-pool-1.6. 12 May 2016, 00:25:57 UTC
e2a43d0 Andrew Or 11 May 2016, 20:36:58 UTC [SPARK-15262] Synchronize block manager / scheduler executor state ## What changes were proposed in this pull request? If an executor is still alive even after the scheduler has removed its metadata, we may receive a heartbeat from that executor and tell its block manager to reregister itself. If that happens, the block manager master will know about the executor, but the scheduler will not. That is a dangerous situation, because when the executor does get disconnected later, the scheduler will not ask the block manager to also remove metadata for that executor. Later, when we try to clean up an RDD or a broadcast variable, we may try to send a message to that executor, triggering an exception. ## How was this patch tested? Jenkins. Author: Andrew Or <andrew@databricks.com> Closes #13055 from andrewor14/block-manager-remove. (cherry picked from commit 40a949aae9c3040019a52482d091912a85b0f4d4) Signed-off-by: Shixiong Zhu <shixiong@databricks.com> 11 May 2016, 20:37:16 UTC
ced71d3 Shixiong Zhu 29 February 2016, 19:52:11 UTC [SPARK-13522][CORE] Fix the exit log place for heartbeat ## What changes were proposed in this pull request? Just fixed the log place introduced by #11401 ## How was this patch tested? unit tests. Author: Shixiong Zhu <shixiong@databricks.com> Closes #11432 from zsxwing/SPARK-13522-follow-up. 11 May 2016, 18:29:10 UTC
86bf93e Shixiong Zhu 29 February 2016, 19:02:45 UTC [SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times ## What changes were proposed in this pull request? Sometimes, network disconnection event won't be triggered for other potential race conditions that we may not have thought of, then the executor will keep sending heartbeats to driver and won't exit. This PR adds a new configuration `spark.executor.heartbeat.maxFailures` to kill Executor when it's unable to heartbeat to the driver more than `spark.executor.heartbeat.maxFailures` times. ## How was this patch tested? unit tests Author: Shixiong Zhu <shixiong@databricks.com> Closes #11401 from zsxwing/SPARK-13522. 11 May 2016, 18:29:06 UTC
c433c0a Shixiong Zhu 26 February 2016, 23:11:57 UTC [SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state ## What changes were proposed in this pull request? When the driver removes an executor's state, the connection between the driver and the executor may be still alive so that the executor cannot exit automatically (E.g., Master will send RemoveExecutor when a work is lost but the executor is still alive), so the driver should try to tell the executor to stop itself. Otherwise, we will leak an executor. This PR modified the driver to send `StopExecutor` to the executor when it's removed. ## How was this patch tested? manual test: increase the worker heartbeat interval to force it's always timeout and the leak executors are gone. Author: Shixiong Zhu <shixiong@databricks.com> Closes #11399 from zsxwing/SPARK-13519. 11 May 2016, 18:29:01 UTC
d165486 xin Wu 11 May 2016, 08:30:45 UTC [SPARK-14495][SQL][1.6] fix resolution failure of having clause with distinct aggregate function #### Symptom: In the latest **branch 1.6**, when a `DISTINCT` aggregation function is used in the `HAVING` clause, Analyzer throws `AnalysisException` with a message like following: ``` resolved attribute(s) gid#558,id#559 missing from date#554,id#555 in operator !Expand [List(date#554, null, 0, if ((gid#558 = 1)) id#559 else null),List(date#554, id#555, 1, null)], [date#554,id#561,gid#560,if ((gid = 1)) id else null#562]; ``` #### Root cause: The problem is that the distinct aggregate in having condition are resolved by the rule `DistinctAggregationRewriter` twice, which messes up the resulted `EXPAND` operator. In a `ResolveAggregateFunctions` rule, when resolving ```Filter(havingCondition, _: Aggregate)```, the `havingCondition` is resolved as an `Aggregate` in a nested loop of analyzer rule execution (by invoking `RuleExecutor.execute`). At this nested level of analysis, the rule `DistinctAggregationRewriter` rewrites this distinct aggregate clause to an expanded two-layer aggregation, where the `aggregateExpresssions` of the final `Aggregate` contains the resolved `gid` and the aggregate expression attributes (In the above case, they are `gid#558, id#559`). After completion of the nested analyzer rule execution, the resulted `aggregateExpressions` in the `havingCondition` is pushed down into the underlying `Aggregate` operator. The `DistinctAggregationRewriter` rule is executed again. The `projections` field of `EXPAND` operator is populated with the `aggregateExpressions` of the `havingCondition` mentioned above. However, the attributes (In the above case, they are `gid#558, id#559`) in the projection list of `EXPAND` operator can not be found in the underlying relation. #### Solution: This PR retrofits part of [#11579](https://github.com/apache/spark/pull/11579) that moves the `DistinctAggregationRewriter` to the beginning of Optimizer, so that it guarantees that the rewrite only happens after all the aggregate functions are resolved first. Thus, it avoids resolution failure. #### How is the PR change tested New [test cases ](https://github.com/xwu0226/spark/blob/f73428f94746d6d074baf6702589545bdbd11cad/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala#L927-L988) are added to drive `DistinctAggregationRewriter` rewrites for multi-distinct aggregations , involving having clause. A following up PR will be submitted to add these test cases to master(2.0) branch. Author: xin Wu <xinwu@us.ibm.com> Closes #12974 from xwu0226/SPARK-14495_review. 11 May 2016, 08:30:45 UTC
1678bff Josh Rosen 09 May 2016, 23:21:32 UTC [SPARK-15209] Fix display of job descriptions with single quotes in web UI timeline ## What changes were proposed in this pull request? This patch fixes an escaping bug in the Web UI's event timeline that caused Javascript errors when displaying timeline entries whose descriptions include single quotes. The original bug can be reproduced by running ```scala sc.setJobDescription("double quote: \" ") sc.parallelize(1 to 10).count() sc.setJobDescription("single quote: ' ") sc.parallelize(1 to 10).count() ``` and then browsing to the driver UI. Previously, this resulted in an "Uncaught SyntaxError" because the single quote from the description was not escaped and ended up closing a Javascript string literal too early. The fix implemented here is to change the relevant Javascript to define its string literals using double-quotes. Our escaping logic already properly escapes double quotes in the description, so this is safe to do. ## How was this patch tested? Tested manually in `spark-shell` using the following cases: ```scala sc.setJobDescription("double quote: \" ") sc.parallelize(1 to 10).count() sc.setJobDescription("single quote: ' ") sc.parallelize(1 to 10).count() sc.setJobDescription("ampersand: &") sc.parallelize(1 to 10).count() sc.setJobDescription("newline: \n text after newline ") sc.parallelize(1 to 10).count() sc.setJobDescription("carriage return: \r text after return ") sc.parallelize(1 to 10).count() ``` /cc sarutak for review. Author: Josh Rosen <joshrosen@databricks.com> Closes #12995 from JoshRosen/SPARK-15209. (cherry picked from commit 3323d0f931ddd11f41abca11425b5e43a6538667) Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp> 09 May 2016, 23:22:32 UTC
518af07 Philipp Hoffmann 09 May 2016, 18:02:13 UTC [SPARK-15223][DOCS] fix wrongly named config reference ## What changes were proposed in this pull request? The configuration setting `spark.executor.logs.rolling.size.maxBytes` was changed to `spark.executor.logs.rolling.maxSize` in 1.4 or so. This commit fixes a remaining reference to the old name in the documentation. Also the description for `spark.executor.logs.rolling.maxSize` was edited to clearly state that the unit for the size is bytes. ## How was this patch tested? no tests Author: Philipp Hoffmann <mail@philipphoffmann.de> Closes #13001 from philipphoffmann/patch-3. 09 May 2016, 18:03:30 UTC
ab00652 cenyuhai 06 May 2016, 20:50:49 UTC [SPARK-13566][CORE] Avoid deadlock between BlockManager and Executor Thread Temp patch for branch 1.6， avoid deadlock between BlockManager and Executor Thread. Author: cenyuhai <cenyuhai@didichuxing.com> Closes #11546 from cenyuhai/SPARK-13566. 06 May 2016, 20:50:49 UTC
a3aa22a Sean Owen 06 May 2016, 11:21:25 UTC [SPARK-14915] Fix incorrect resolution of merge conflict in commit bf3c0608f1779b4dd837b8289ec1d4516e145aea ## What changes were proposed in this pull request? I botched the back-port of SPARK-14915 to branch-1.6 in https://github.com/apache/spark/commit/bf3c0608f1779b4dd837b8289ec1d4516e145aea resulting in a code block being added twice. This simply removes it, such that the net change is the intended one. ## How was this patch tested? Jenkins tests. (This in theory has already been tested.) Author: Sean Owen <sowen@cloudera.com> Closes #12950 from srowen/SPARK-14915.2. 06 May 2016, 11:21:25 UTC
bf3c060 Jason Moore 05 May 2016, 10:02:35 UTC [SPARK-14915][CORE] Don't re-queue a task if another attempt has already succeeded Don't re-queue a task if another attempt has already succeeded. This currently happens when a speculative task is denied from committing the result due to another copy of the task already having succeeded. I'm running a job which has a fair bit of skew in the processing time across the tasks for speculation to trigger in the last quarter (default settings), causing many commit denied exceptions to be thrown. Previously, these tasks were then being retried over and over again until the stage possibly completes (despite using compute resources on these superfluous tasks). With this change (applied to the 1.6 branch), they no longer retry and the stage completes successfully without these extra task attempts. Author: Jason Moore <jasonmoore2k@outlook.com> Closes #12751 from jasonmoore2k/SPARK-14915. (cherry picked from commit 77361a433adce109c2b752b11dda25b56eca0352) Signed-off-by: Sean Owen <sowen@cloudera.com> 05 May 2016, 19:14:35 UTC
2db19a3 Sean Owen 03 May 2016, 12:13:35 UTC [SPARK-14897][CORE] Upgrade Jetty to latest version of 8 Update Jetty 8.1 to the latest 2016/02 release, from a 2013/10 release, for security and bug fixes. This does not resolve the JIRA necessarily, as it's still worth considering an update to 9.3. Jenkins tests Author: Sean Owen <sowen@cloudera.com> Closes #12842 from srowen/SPARK-14897. (cherry picked from commit 57ac7c182465e1653e74a8ad6c826b2cf56a0ad8) Signed-off-by: Sean Owen <sowen@cloudera.com> 03 May 2016, 12:15:01 UTC
10d513f Reynold Xin 30 April 2016, 05:26:12 UTC [SPARK-14757] [SQL] Fix nullability bug in EqualNullSafe codegen This patch fixes a null handling bug in EqualNullSafe's code generation. Updated unit test so they would fail without the fix. Closes #12628. Author: Reynold Xin <rxin@databricks.com> Author: Arash Nabili <arash@levyx.com> Closes #12799 from rxin/equalnullsafe. 30 April 2016, 05:28:34 UTC
8ac0ce6 Gregory Hart 28 April 2016, 18:21:43 UTC [SPARK-14965][SQL] Indicate an exception is thrown for a missing struct field ## What changes were proposed in this pull request? Fix to ScalaDoc for StructType. ## How was this patch tested? Built locally. Author: Gregory Hart <greg.hart@thinkbiganalytics.com> Closes #12758 from freastro/hotfix/SPARK-14965. (cherry picked from commit 12c360c057f09d13a31c458ad277640b5f6de394) Signed-off-by: Reynold Xin <rxin@databricks.com> 28 April 2016, 18:22:11 UTC
f4af6a8 Yin Huai 27 April 2016, 23:33:30 UTC [SPARK-13023][PROJECT INFRA][BRANCH-1.6] Fix handling of root module in modules_to_test() This is a 1.6 branch backport of SPARK-13023 based on JoshRosen's https://github.com/apache/spark/commit/41f0c85f9be264103c066935e743f59caf0fe268. There's a minor bug in how we handle the `root` module in the `modules_to_test()` function in `dev/run-tests.py`: since `root` now depends on `build` (since every test needs to run on any build test), we now need to check for the presence of root in `modules_to_test` instead of `changed_modules`. Author: Yin Huai <yhuai@databricks.com> Closes #12743 from yhuai/1.6build. 27 April 2016, 23:33:30 UTC
5e53d4a Joseph K. Bradley 27 April 2016, 23:11:12 UTC [SPARK-14671][ML] Pipeline setStages should handle subclasses of PipelineStage Pipeline.setStages failed for some code examples which worked in 1.5 but fail in 1.6. This tends to occur when using a mix of transformers from ml.feature. It is because Java Arrays are non-covariant and the addition of MLWritable to some transformers means the stages0/1 arrays above are not of type Array[PipelineStage]. This PR modifies the following to accept subclasses of PipelineStage: * Pipeline.setStages() * Params.w() Unit test which fails to compile before this fix. Author: Joseph K. Bradley <joseph@databricks.com> Closes #12430 from jkbradley/pipeline-setstages. (cherry picked from commit f5ebb18c45ffdee2756a80f64239cb9158df1a11) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> Conflicts: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala 27 April 2016, 23:15:46 UTC
496496b Joseph K. Bradley 26 April 2016, 21:00:39 UTC [SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula - 1.6 backport Backport of [https://github.com/apache/spark/pull/11965] for branch-1.6. There were no merge conflicts. ## What changes were proposed in this pull request? StringIndexerModel.transform sets the output column metadata to use name inputCol. It should not. Fixing this causes a problem with the metadata produced by RFormula. Fix in RFormula: I added the StringIndexer columns to prefixesToRewrite, and I modified VectorAttributeRewriter to find and replace all "prefixes" since attributes collect multiple prefixes from StringIndexer + Interaction. Note that "prefixes" is no longer accurate since internal strings may be replaced. ## How was this patch tested? Unit test which failed before this fix. Author: Joseph K. Bradley <joseph@databricks.com> Closes #12595 from jkbradley/StringIndexer-fix-1.6. 26 April 2016, 21:00:39 UTC
c3ed950 hyukjinkwon 21 April 2016, 10:32:27 UTC [SPARK-14787][SQL] Upgrade Joda-Time library from 2.9 to 2.9.3 ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14787 The possible problems are described in the JIRA above. Please refer this if you are wondering the purpose of this PR. This PR upgrades Joda-Time library from 2.9 to 2.9.3. ## How was this patch tested? `sbt scalastyle` and Jenkins tests in this PR. closes #11847 Author: hyukjinkwon <gurwls223@gmail.com> Closes #12552 from HyukjinKwon/SPARK-14787. (cherry picked from commit ec2a276022568944e19a51d4d39305710cdc7c0f) Signed-off-by: Sean Owen <sowen@cloudera.com> 21 April 2016, 10:32:36 UTC
1cda10b Arash Parsa 21 April 2016, 10:29:24 UTC [SPARK-14739][PYSPARK] Fix Vectors parser bugs ## What changes were proposed in this pull request? The PySpark deserialization has a bug that shows while deserializing all zero sparse vectors. This fix filters out empty string tokens before casting, hence properly stringified SparseVectors successfully get parsed. ## How was this patch tested? Standard unit-tests similar to other methods. Author: Arash Parsa <arash@ip-192-168-50-106.ec2.internal> Author: Arash Parsa <arashpa@gmail.com> Author: Vishnu Prasad <vishnu667@gmail.com> Author: Vishnu Prasad S <vishnu667@gmail.com> Closes #12516 from arashpa/SPARK-14739. (cherry picked from commit 2b8906c43760591f2e2da99bf0e34fa1bb63bfd1) Signed-off-by: Sean Owen <sowen@cloudera.com> 21 April 2016, 10:29:36 UTC
17b1384 Ryan Blue 20 April 2016, 10:26:42 UTC [SPARK-14679][UI] Fix UI DAG visualization OOM. ## What changes were proposed in this pull request? The DAG visualization can cause an OOM when generating the DOT file. This happens because clusters are not correctly deduped by a contains check because they use the default equals implementation. This adds a working equals implementation. ## How was this patch tested? This adds a test suite that checks the new equals implementation. Author: Ryan Blue <blue@apache.org> Closes #12437 from rdblue/SPARK-14679-fix-ui-oom. (cherry picked from commit a3451119d951949f24f3a4c5e33a5daea615dfed) Signed-off-by: Sean Owen <sowen@cloudera.com> 20 April 2016, 10:27:16 UTC
cd81214 CodingCat 19 April 2016, 01:51:23 UTC [SPARK-13227] Risky apply() in OpenHashMap https://issues.apache.org/jira/browse/SPARK-13227 It might confuse the future developers when they use OpenHashMap.apply() with a numeric value type. null.asInstance[Int], null.asInstance[Long], null.asInstace[Float] and null.asInstance[Double] will return 0/0.0/0L, which might confuse the developer if the value set contains 0/0.0/0L with an existing key The current patch only adds the comments describing the issue, with the respect to apply the minimum changes to the code base The more direct, yet more aggressive, approach is use Option as the return type andrewor14 JoshRosen any thoughts about how to avoid the potential issue? Author: CodingCat <zhunansjtu@gmail.com> Closes #11107 from CodingCat/SPARK-13227. (cherry picked from commit 4b3d1294aeecc0001a7fa48c92796e6075d34540) Signed-off-by: Reynold Xin <rxin@databricks.com> 19 April 2016, 01:51:40 UTC
58dfba6 Joseph K. Bradley 15 April 2016, 18:50:21 UTC [SPARK-14665][ML][PYTHON] Fixed bug with StopWordsRemover default stopwords The default stopwords were a Java object. They are no longer. Unit test which failed before the fix Author: Joseph K. Bradley <joseph@databricks.com> Closes #12422 from jkbradley/pyspark-stopwords. (cherry picked from commit d6ae7d4637d23c57c4eeab79d1177216f380ec9c) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> Conflicts: python/pyspark/ml/feature.py python/pyspark/ml/tests.py 15 April 2016, 18:58:08 UTC
93c9a63 Joseph K. Bradley 14 April 2016, 19:44:59 UTC [SPARK-14618][ML][DOC] Updated RegressionEvaluator.metricName param doc ## What changes were proposed in this pull request? In Spark 1.4, we negated some metrics from RegressionEvaluator since CrossValidator always maximized metrics. This was fixed in 1.5, but the docs were not updated. This PR updates the docs. ## How was this patch tested? no tests Author: Joseph K. Bradley <joseph@databricks.com> Closes #12377 from jkbradley/regeval-doc. (cherry picked from commit bf65c87f706019d235d7093637341668a13b1be1) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> 14 April 2016, 19:45:11 UTC
413d060 Sital Kedia 12 April 2016, 23:10:07 UTC [SPARK-14363] Fix executor OOM due to memory leak in the Sorter Fix memory leak in the Sorter. When the UnsafeExternalSorter spills the data to disk, it does not free up the underlying pointer array. As a result, we see a lot of executor OOM and also memory under utilization. This is a regression partially introduced in PR https://github.com/apache/spark/pull/9241 Tested by running a job and observed around 30% speedup after this change. Author: Sital Kedia <skedia@fb.com> Closes #12285 from sitalkedia/executor_oom. (cherry picked from commit d187e7dea9540d26b7800de4eb79863ef5f574bf) Signed-off-by: Davies Liu <davies.liu@gmail.com> Conflicts: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 12 April 2016, 23:12:17 UTC
582ed8a Davies Liu 12 April 2016, 22:03:00 UTC [SPARK-14544] [SQL] improve performance of SQL UI tab ## What changes were proposed in this pull request? This PR improve the performance of SQL UI by: 1) remove the details column in all executions page (the first page in SQL tab). We can check the details by enter the execution page. 2) break-all is super slow in Chrome recently, so switch to break-word. 3) Using "display: none" to hide a block. 4) using one js closure for for all the executions, not one for each. 5) remove the height limitation of details, don't need to scroll it in the tiny window. ## How was this patch tested? Exists tests. ![ui](https://cloud.githubusercontent.com/assets/40902/14445712/68d7b258-0004-11e6-9b48-5d329b05d165.png) Author: Davies Liu <davies@databricks.com> Closes #12311 from davies/ui_perf. 12 April 2016, 22:05:14 UTC
2554c35 Xiangrui Meng 12 April 2016, 18:30:09 UTC [SPARK-14563][ML] use a random table name instead of __THIS__ in SQLTransformer ## What changes were proposed in this pull request? Use a random table name instead of `__THIS__` in SQLTransformer, and add a test for `transformSchema`. The problems of using `__THIS__` are: * It doesn't work under HiveContext (in Spark 1.6) * Race conditions ## How was this patch tested? * Manual test with HiveContext. * Added a unit test for `transformSchema` to improve coverage. cc: yhuai Author: Xiangrui Meng <meng@databricks.com> Closes #12330 from mengxr/SPARK-14563. (cherry picked from commit 1995c2e6482bf4af5a4be087bfc156311c1bec19) Signed-off-by: Xiangrui Meng <meng@databricks.com> 12 April 2016, 18:30:17 UTC
663a492 Zhang, Liye 01 April 2016, 03:17:52 UTC [SPARK-14242][CORE][NETWORK] avoid copy in compositeBuffer for frame decoder ## What changes were proposed in this pull request? In this patch, we set the initial `maxNumComponents` to `Integer.MAX_VALUE` instead of the default size ( which is 16) when allocating `compositeBuffer` in `TransportFrameDecoder` because `compositeBuffer` will introduce too many memory copies underlying if `compositeBuffer` is with default `maxNumComponents` when the frame size is large (which result in many transport messages). For details, please refer to [SPARK-14242](https://issues.apache.org/jira/browse/SPARK-14242). ## How was this patch tested? spark unit tests and manual tests. For manual tests, we can reproduce the performance issue with following code: `sc.parallelize(Array(1,2,3),3).mapPartitions(a=>Array(new Array[Double](1024 * 1024 * 50)).iterator).reduce((a,b)=> a).length` It's easy to see the performance gain, both from the running time and CPU usage. Author: Zhang, Liye <liye.zhang@intel.com> Closes #12038 from liyezhang556520/spark-14242. 12 April 2016, 02:25:10 UTC
05dbc28 Yanbo Liang 08 April 2016, 18:49:44 UTC [SPARK-14298][ML][MLLIB] LDA should support disable checkpoint ## What changes were proposed in this pull request? In the doc of [```checkpointInterval```](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala#L241), we told users that they can disable checkpoint by setting ```checkpointInterval = -1```. But we did not handle this situation for LDA actually, we should fix this bug. ## How was this patch tested? Existing tests. cc jkbradley Author: Yanbo Liang <ybliang8@gmail.com> Closes #12089 from yanboliang/spark-14298. (cherry picked from commit 56af8e85cca056096fe4e765d8d287e0f9efc0d2) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> 11 April 2016, 23:18:03 UTC
f4110cd Josh Rosen 11 April 2016, 22:14:35 UTC [BUILD][HOTFIX] Download Maven from regular mirror network rather than archive.apache.org [archive.apache.org](https://archive.apache.org/) is undergoing maintenance, breaking our `build/mvn` script: > We are in the process of relocating this service. To save on the immense bandwidth that this service outputs, we have put it in maintenance mode, disabling all downloads for the next few days. We expect the maintenance to be complete no later than the morning of Monday the 11th of April, 2016. This patch fixes this issue by updating the script to use the regular mirror network to download Maven. (This is a backport of #12262 to 1.6) Author: Josh Rosen <joshrosen@databricks.com> Closes #12307 from JoshRosen/fix-1.6-mvn-download. 11 April 2016, 22:14:35 UTC
c12db0d Sameer Agarwal 11 April 2016, 17:20:22 UTC [SPARK-14454] [1.6] Better exception handling while marking tasks as failed Backports https://github.com/apache/spark/pull/12234 to 1.6. Original description below: ## What changes were proposed in this pull request? This patch adds support for better handling of exceptions inside catch blocks if the code within the block throws an exception. For instance here is the code in a catch block before this change in `WriterContainer.scala`: ```scala logError("Aborting task.", cause) // call failure callbacks first, so we could have a chance to cleanup the writer. TaskContext.get().asInstanceOf[TaskContextImpl].markTaskFailed(cause) if (currentWriter != null) { currentWriter.close() } abortTask() throw new SparkException("Task failed while writing rows.", cause) ``` If `markTaskFailed` or `currentWriter.close` throws an exception, we currently lose the original cause. This PR fixes this problem by implementing a utility function `Utils.tryWithSafeCatch` that suppresses (`Throwable.addSuppressed`) the exception that are thrown within the catch block and rethrowing the original exception. ## How was this patch tested? No new functionality added Author: Sameer Agarwal <sameer@databricks.com> Closes #12272 from sameeragarwal/fix-exception-1.6. 11 April 2016, 17:20:22 UTC
baf2985 Zhang, Liye 11 April 2016, 17:06:57 UTC [SPARK-14290] [SPARK-13352] [CORE] [BACKPORT-1.6] avoid significant memory copy in Netty's tran… ## What changes were proposed in this pull request? When netty transfer data that is not `FileRegion`, data will be in format of `ByteBuf`, If the data is large, there will occur significant performance issue because there is memory copy underlying in `sun.nio.ch.IOUtil.write`, the CPU is 100% used, and network is very low. In this PR, if data size is large, we will split it into small chunks to call `WritableByteChannel.write()`, so that avoid wasting of memory copy. Because the data can't be written within a single write, and it will call `transferTo` multiple times. ## How was this patch tested? Spark unit test and manual test. Manual test: `sc.parallelize(Array(1,2,3),3).mapPartitions(a=>Array(new Array[Double](1024 * 1024 * 50)).iterator).reduce((a,b)=> a).length` For more details, please refer to [SPARK-14290](https://issues.apache.org/jira/browse/SPARK-14290) Author: Zhang, Liye <liye.zhang@intel.com> Closes #12296 from liyezhang556520/apache-branch-1.6-spark-14290. 11 April 2016, 17:06:57 UTC
7a02c44 Jason Moore 10 April 2016, 06:34:57 UTC [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception ## What changes were proposed in this pull request? When deciding whether a CommitDeniedException caused a task to fail, consider the root cause of the Exception. ## How was this patch tested? Added a test suite for the component that extracts the root cause of the error. Made a distribution after cherry-picking this commit to branch-1.6 and used to run our Spark application that would quite often fail due to the CommitDeniedException. Author: Jason Moore <jasonmoore2k@outlook.com> Closes #12228 from jasonmoore2k/SPARK-14357. (cherry picked from commit 22014e6fb919a35c31d852b7c2f5b7eb05751208) Signed-off-by: Andrew Or <andrew@databricks.com> 10 April 2016, 06:35:10 UTC
77ebae3 Andrew Or 08 April 2016, 00:49:39 UTC [SPARK-14468] Always enable OutputCommitCoordinator ## What changes were proposed in this pull request? `OutputCommitCoordinator` was introduced to deal with concurrent task attempts racing to write output, leading to data loss or corruption. For more detail, read the [JIRA description](https://issues.apache.org/jira/browse/SPARK-14468). Before: `OutputCommitCoordinator` is enabled only if speculation is enabled. After: `OutputCommitCoordinator` is always enabled. Users may still disable this through `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <andrew@databricks.com> Closes #12244 from andrewor14/always-occ. (cherry picked from commit 3e29e372ff518827bae9dcd26087946fde476843) Signed-off-by: Andrew Or <andrew@databricks.com> 08 April 2016, 00:49:50 UTC
8a94a59 Michael Gummelt 08 April 2016, 00:41:37 UTC [DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode. Docs change to remove the sentence about Mesos not supporting cluster mode. It was not. Author: Michael Gummelt <mgummelt@mesosphere.io> Closes #12249 from mgummelt/fix-mesos-cluster-docs. (cherry picked from commit 30e980ad8e6443dddd54f3c2d48b3904499545cf) Signed-off-by: Andrew Or <andrew@databricks.com> 08 April 2016, 00:42:18 UTC
dca0d9a Yuhao Yang 06 April 2016, 18:36:26 UTC [SPARK-14322][MLLIB] Use treeAggregate instead of reduce in OnlineLDAOptimizer ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-14322 OnlineLDAOptimizer uses RDD.reduce in two places where it could use treeAggregate. This can cause scalability issues. This should be an easy fix. This is also a bug since it modifies the first argument to reduce, so we should use aggregate or treeAggregate. See this line: https://github.com/apache/spark/blob/f12f11e578169b47e3f8b18b299948c0670ba585/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala#L452 and a few lines below it. ## How was this patch tested? unit tests Author: Yuhao Yang <hhbyyh@gmail.com> Closes #12106 from hhbyyh/ldaTreeReduce. (cherry picked from commit 8cffcb60deb82d04a5c6e144ec9927f6f7addc8b) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> 06 April 2016, 18:37:13 UTC
cfe9f02 jeanlyn 05 April 2016, 19:32:03 UTC [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks ## What changes were proposed in this pull request? This patch try to update the `updatedBlockStatuses ` when removing blocks, making sure `BlockManager` correctly updates `updatedBlockStatuses` ## How was this patch tested? test("updated block statuses") in BlockManagerSuite.scala Author: jeanlyn <jeanlyn92@gmail.com> Closes #12150 from jeanlyn/updataBlock1.6. 05 April 2016, 19:32:03 UTC
285cb9c Yong Tang 05 April 2016, 03:19:20 UTC [SPARK-14368][PYSPARK] Support python.spark.worker.memory with upper-case unit. ## What changes were proposed in this pull request? This fix tries to address the issue in PySpark where `spark.python.worker.memory` could only be configured with a lower case unit (`k`, `m`, `g`, `t`). This fix allows the upper case unit (`K`, `M`, `G`, `T`) to be used as well. This is to conform to the JVM memory string as is specified in the documentation . ## How was this patch tested? This fix adds additional test to cover the changes. Author: Yong Tang <yong.tang.github@outlook.com> Closes #12163 from yongtang/SPARK-14368. (cherry picked from commit 7db56244fa3dba92246bad6694f31bbf68ea47ec) Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp> 05 April 2016, 03:19:50 UTC
91530b0 Jo Voordeckers 04 April 2016, 20:29:48 UTC [SPARK-11327][MESOS] Backport dispatcher does not respect all args f… Backport for https://github.com/apache/spark/pull/10370 andrewor14 Author: Jo Voordeckers <jo.voordeckers@gmail.com> Closes #12101 from jayv/mesos_cluster_params_backport. 04 April 2016, 20:29:52 UTC
f12f11e Kazuaki Ishizaki 31 March 2016, 22:05:48 UTC [SPARK-14138] [SQL] Fix generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames ## What changes were proposed in this pull request? This PR reduces Java byte code size of method in ```SpecificColumnarIterator``` by using two approaches: 1. Generate and call ```getTYPEColumnAccessor()``` for each type, which is actually used, for instantiating accessors 2. Group a lot of method calls (more than 4000) into a method ## How was this patch tested? Added a new unit test to ```InMemoryColumnarQuerySuite``` Here is generate code ```java /* 033 */ private org.apache.spark.sql.execution.columnar.CachedBatch batch = null; /* 034 */ /* 035 */ private org.apache.spark.sql.execution.columnar.IntColumnAccessor accessor; /* 036 */ private org.apache.spark.sql.execution.columnar.IntColumnAccessor accessor1; /* 037 */ /* 038 */ public SpecificColumnarIterator() { /* 039 */ this.nativeOrder = ByteOrder.nativeOrder(); /* 030 */ this.mutableRow = new MutableUnsafeRow(rowWriter); /* 041 */ } /* 042 */ /* 043 */ public void initialize(Iterator input, DataType[] columnTypes, int[] columnIndexes, /* 044 */ boolean columnNullables[]) { /* 044 */ this.input = input; /* 046 */ this.columnTypes = columnTypes; /* 047 */ this.columnIndexes = columnIndexes; /* 048 */ } /* 049 */ /* 050 */ /* 051 */ private org.apache.spark.sql.execution.columnar.IntColumnAccessor getIntColumnAccessor(int idx) { /* 052 */ byte[] buffer = batch.buffers()[columnIndexes[idx]]; /* 053 */ return new org.apache.spark.sql.execution.columnar.IntColumnAccessor(ByteBuffer.wrap(buffer).order(nativeOrder)); /* 054 */ } /* 055 */ /* 056 */ /* 057 */ /* 058 */ /* 059 */ /* 060 */ /* 061 */ public boolean hasNext() { /* 062 */ if (currentRow < numRowsInBatch) { /* 063 */ return true; /* 064 */ } /* 065 */ if (!input.hasNext()) { /* 066 */ return false; /* 067 */ } /* 068 */ /* 069 */ batch = (org.apache.spark.sql.execution.columnar.CachedBatch) input.next(); /* 070 */ currentRow = 0; /* 071 */ numRowsInBatch = batch.numRows(); /* 072 */ accessor = getIntColumnAccessor(0); /* 073 */ accessor1 = getIntColumnAccessor(1); /* 074 */ /* 075 */ return hasNext(); /* 076 */ } /* 077 */ /* 078 */ public InternalRow next() { /* 079 */ currentRow += 1; /* 080 */ bufferHolder.reset(); /* 081 */ rowWriter.zeroOutNullBytes(); /* 082 */ accessor.extractTo(mutableRow, 0); /* 083 */ accessor1.extractTo(mutableRow, 1); /* 084 */ unsafeRow.setTotalSize(bufferHolder.totalSize()); /* 085 */ return unsafeRow; /* 086 */ } ``` (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #11984 from kiszk/SPARK-14138. 31 March 2016, 22:05:48 UTC
3cc3d85 Yuhao Yang 30 March 2016, 22:58:19 UTC [SPARK-11507][MLLIB] add compact in Matrices fromBreeze jira: https://issues.apache.org/jira/browse/SPARK-11507 "In certain situations when adding two block matrices, I get an error regarding colPtr and the operation fails. External issue URL includes full error and code for reproducing the problem." root cause: colPtr.last does NOT always equal to values.length in breeze SCSMatrix, which fails the require in SparseMatrix. easy step to repro: ``` val m1: BM[Double] = new CSCMatrix[Double] (Array (1.0, 1, 1), 3, 3, Array (0, 1, 2, 3), Array (0, 1, 2) ) val m2: BM[Double] = new CSCMatrix[Double] (Array (1.0, 2, 2, 4), 3, 3, Array (0, 0, 2, 4), Array (1, 2, 1, 2) ) val sum = m1 + m2 Matrices.fromBreeze(sum) ``` Solution: By checking the code in [CSCMatrix](https://github.com/scalanlp/breeze/blob/28000a7b901bc3cfbbbf5c0bce1d0a5dda8281b0/math/src/main/scala/breeze/linalg/CSCMatrix.scala), CSCMatrix in breeze can have extra zeros in the end of data array. Invoking compact will make sure it aligns with the require of SparseMatrix. This should add limited overhead as the actual compact operation is only performed when necessary. Author: Yuhao Yang <hhbyyh@gmail.com> Closes #9520 from hhbyyh/matricesFromBreeze. (cherry picked from commit ca458618d8ee659ffa9a081083cd475a440fa8ff) Signed-off-by: Joseph K. Bradley <joseph@databricks.com> Conflicts: mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala 30 March 2016, 23:03:35 UTC
84ad254 Carson Wang 29 March 2016, 18:07:58 UTC [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason. ## What changes were proposed in this pull request? The event timeline doesn't show on job page if an executor is removed with a multiple line reason. This PR replaces all new line characters in the reason string with spaces. ![timelineerror](https://cloud.githubusercontent.com/assets/9278199/14100211/5fd4cd30-f5be-11e5-9cea-f32651a4cd62.jpg) ## How was this patch tested? Verified on the Web UI. Author: Carson Wang <carson.wang@intel.com> Closes #12029 from carsonwang/eventTimeline. (cherry picked from commit 15c0b0006b3d04434b505210df541aeb28a51de8) Signed-off-by: Andrew Or <andrew@databricks.com> 29 March 2016, 18:08:12 UTC
c2ce247 jeanlyn 29 March 2016, 17:51:00 UTC [SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after few days, here is some driver heap dump before OOM ``` num #instances #bytes class name ---------------------------------------------- 1: 13845916 553836640 org.apache.spark.storage.BlockStatus 2: 14020324 336487776 org.apache.spark.storage.StreamBlockId 3: 13883881 333213144 scala.collection.mutable.DefaultEntry 4: 8907 89043952 [Lscala.collection.mutable.HashEntry; 5: 62360 65107352 [B 6: 163368 24453904 [Ljava.lang.Object; 7: 293651 20342664 [C ... ``` `BlockStatus` and `StreamBlockId` keep on growing, and the driver OOM in the end. After investigated, i found the `executorIdToStorageStatus` in `StorageStatusListener` seems never remove the blocks from `StorageStatus`. In order to fix the issue, i try to use `onBlockUpdated` replace `onTaskEnd ` , so we can update the block informations(add blocks, drop the block from memory to disk and delete the blocks) in time. ## How was this patch tested? Existing unit tests and manual tests Author: jeanlyn <jeanlyn92@gmail.com> Closes #12028 from jeanlyn/fixoom1.6. 29 March 2016, 17:51:00 UTC
a757944 Dongjoon Hyun 29 March 2016, 04:00:00 UTC [SPARK-14219][GRAPHX] Fix `pickRandomVertex` not to fall into infinit… ## What changes were proposed in this pull request? Currently, `GraphOps.pickRandomVertex()` falls into infinite loops for graphs having only one vertex. This PR fixes it by modifying the following termination-checking condition. ```scala - if (selectedVertices.count > 1) { + if (selectedVertices.count > 0) { ``` ## How was this patch tested? Pass the Jenkins tests (including new test case). Author: Dongjoon Hyun <dongjoon@apache.org> Closes #12021 from dongjoon-hyun/SPARK-14219-2. 29 March 2016, 04:00:00 UTC
504b992 nfraison 28 March 2016, 21:10:25 UTC [SPARK-13622][YARN] Issue creating level db for YARN shuffle service ## What changes were proposed in this pull request? This patch will ensure that we trim all path set in yarn.nodemanager.local-dirs and that the the scheme is well removed so the level db can be created. ## How was this patch tested? manual tests. Author: nfraison <nfraison@yahoo.fr> Closes #11475 from ashangit/level_db_creation_issue. (cherry picked from commit ff3bea38ed2ac8dac5832f0bf8eac70192a512ef) 28 March 2016, 21:50:18 UTC
546569e Chenliang Xu 28 March 2016, 15:33:37 UTC [SPARK-14187][MLLIB] Fix incorrect use of binarySearch in SparseMatrix ## What changes were proposed in this pull request? Fix incorrect use of binarySearch in SparseMatrix ## How was this patch tested? Unit test added. Author: Chenliang Xu <chexu@groupon.com> Closes #11992 from luckyrandom/SPARK-14187. (cherry picked from commit c8388297c436691a236520d2396deaf556aedb0e) Signed-off-by: Xiangrui Meng <meng@databricks.com> 28 March 2016, 15:34:49 UTC
fba84d1 Reynold Xin 25 March 2016, 08:17:23 UTC [SPARK-14149] Log exceptions in tryOrIOException ## What changes were proposed in this pull request? We ran into a problem today debugging some class loading problem during deserialization, and JVM was masking the underlying exception which made it very difficult to debug. We can however log the exceptions using try/catch ourselves in serialization/deserialization. The good thing is that all these methods are already using Utils.tryOrIOException, so we can just put the try catch and logging in a single place. ## How was this patch tested? A logging change with a manual test. Author: Reynold Xin <rxin@databricks.com> Closes #11951 from rxin/SPARK-14149. (cherry picked from commit 70a6f0bb57ca2248444157e2707fbcc3cb04e3bc) Signed-off-by: Reynold Xin <rxin@databricks.com> 25 March 2016, 08:17:31 UTC
70b5878 sethah 25 March 2016, 02:14:24 UTC [SPARK-14107][PYSPARK][ML] Add seed as named argument to GBTs in pyspark ## What changes were proposed in this pull request? GBTs in pyspark previously had seed parameters, but they could not be passed as keyword arguments through the class constructor. This patch adds seed as a keyword argument and also sets default value. ## How was this patch tested? Doc tests were updated to pass a random seed through the GBTClassifier and GBTRegressor constructors. Author: sethah <seth.hendrickson16@gmail.com> Closes #11944 from sethah/SPARK-14107. (cherry picked from commit 585097716c1979ea538ef733cf33225ef7be06f5) Signed-off-by: Xiangrui Meng <meng@databricks.com> 25 March 2016, 02:14:32 UTC
b75f970 Sun Rui 23 March 2016, 14:57:03 UTC [SPARK-14074][SPARKR] Specify commit sha1 ID when using install_github to install intr package. ## What changes were proposed in this pull request? In dev/lint-r.R, `install_github` makes our builds depend on a unstable source. This may cause un-expected test failures and then build break. This PR adds a specified commit sha1 ID to `install_github` to get a stable source. ## How was this patch tested? dev/lint-r Author: Sun Rui <rui.sun@intel.com> Closes #11913 from sun-rui/SPARK-14074. (cherry picked from commit 7d1175011c976756efcd4e4e4f70a8fd6f287026) Signed-off-by: Xiangrui Meng <meng@databricks.com> 23 March 2016, 14:57:14 UTC
5e9cefc jerryshao 23 March 2016, 14:14:29 UTC [SPARK-13642][YARN][1.6-BACKPORT] Properly handle signal kill in ApplicationMaster ## What changes were proposed in this pull request? This patch is fixing the race condition in ApplicationMaster when receiving a signal. In the current implementation, if signal is received and with no any exception, this application will be finished with successful state in Yarn, and there's no another attempt. Actually the application is killed by signal in the runtime, so another attempt is expected. This patch adds a signal handler to handle the signal things, if signal is received, marking this application finished with failure, rather than success. ## How was this patch tested? This patch is tested with following situations: Application is finished normally. Application is finished by calling System.exit(n). Application is killed by yarn command. ApplicationMaster is killed by "SIGTERM" send by kill pid command. ApplicationMaster is killed by NM with "SIGTERM" in case of NM failure. Author: jerryshao <sshao@hortonworks.com> Closes #11690 from jerryshao/SPARK-13642-1.6-backport. 23 March 2016, 14:14:29 UTC
179f6e3 Davies Liu 22 March 2016, 23:45:20 UTC [SPARK-13806] [SQL] fix rounding mode of negative float/double Round() in database usually round the number up (away from zero), it's different than Math.round() in Java. For example: ``` scala> java.lang.Math.round(-3.5) res3: Long = -3 ``` In Database, we should return -4.0 in this cases. This PR remove the buggy special case for scale=0. Add tests for negative values with tie. Author: Davies Liu <davies@databricks.com> Closes #11894 from davies/fix_round. (cherry picked from commit 4700adb98e4a37c2b0ef7123eca8a9a03bbdbe78) Signed-off-by: Davies Liu <davies.liu@gmail.com> Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala 22 March 2016, 23:47:36 UTC
3e21f7f Sun Rui 22 March 2016, 18:17:19 UTC [SPARK-14006][SPARKR] Fix SparkR lint-r test errors in branch-1.6. ## What changes were proposed in this pull request? A backport of #11652 for branch 1.6. It fixes all newly captured SparkR lint-r errors after the lintr package is updated from github. ## How was this patch tested? dev/lint-r SparkR unit tests Author: Sun Rui <rui.sun@intel.com> Closes #11884 from sun-rui/SPARK-14006. 22 March 2016, 18:17:19 UTC
3243d46 cenyuhai 22 March 2016, 12:53:11 UTC [SPARK-13772][SQL] Fix data type mismatch for decimal Fix data type mismatch for decimal, patch for branch-1.6. Author: cenyuhai <cenyuhai@didichuxing.com> Closes #11605 from cenyuhai/SPARK-13772. 22 March 2016, 12:53:18 UTC
f9221ad zero323 22 March 2016, 06:52:33 UTC [SPARK-14058][PYTHON] Incorrect docstring in Window.order ## What changes were proposed in this pull request? Replaces current docstring ("Creates a :class:`WindowSpec` with the partitioning defined.") with "Creates a :class:`WindowSpec` with the ordering defined." ## How was this patch tested? PySpark unit tests (no regression introduced). No changes to the code. Author: zero323 <matthew.szymkiewicz@gmail.com> Closes #11877 from zero323/order-by-description. (cherry picked from commit 8193a266b50460078f64f59c63eae53cdbceeace) Signed-off-by: Reynold Xin <rxin@databricks.com> 22 March 2016, 06:52:45 UTC
022e06d Sital Kedia 18 March 2016, 19:56:06 UTC [SPARK-13958] Executor OOM due to unbounded growth of pointer array in… ## What changes were proposed in this pull request? This change fixes the executor OOM which was recently introduced in PR apache/spark#11095 (Please fill in changes proposed in this fix) ## How was this patch tested? Tested by running a spark job on the cluster. (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) … Sorter Author: Sital Kedia <skedia@fb.com> Closes #11794 from sitalkedia/SPARK-13958. (cherry picked from commit 2e0c5284fd88ba89f53f93dcf1eb26bca2be49c5) Signed-off-by: Davies Liu <davies.liu@gmail.com> 18 March 2016, 19:56:16 UTC
1fcd17f trueyao 17 March 2016, 09:45:06 UTC [SPARK-13901][CORE] correct the logDebug information when jump to the next locality level JIRA Issue:https://issues.apache.org/jira/browse/SPARK-13901 In getAllowedLocalityLevel method of TaskSetManager,we get wrong logDebug information when jump to the next locality level.So we should fix it. Author: trueyao <501663994@qq.com> Closes #11719 from trueyao/logDebug-localityWait. (cherry picked from commit ea9ca6f04ce1fb83612f907d6d0996d6fb362bd2) Signed-off-by: Sean Owen <sowen@cloudera.com> 17 March 2016, 09:45:15 UTC
765307f CodingCat 15 March 2016, 10:10:23 UTC [SPARK-13803] restore the changes in SPARK-3411 ## What changes were proposed in this pull request? This patch contains the functionality to balance the load of the cluster-mode drivers among workers This patch restores the changes in https://github.com/apache/spark/pull/1106 which was erased due to the merging of https://github.com/apache/spark/pull/731 ## How was this patch tested? test with existing test cases Author: CodingCat <zhunansjtu@gmail.com> Closes #11702 from CodingCat/SPARK-13803. (cherry picked from commit bd5365bbe9ff6518cde9402ee8843ec1002fff5b) Signed-off-by: Sean Owen <sowen@cloudera.com> 15 March 2016, 10:10:33 UTC
6935b50 Yin Huai 15 March 2016, 06:42:05 UTC [SPARK-13207][SQL][BRANCH-1.6] Make partitioning discovery ignore _SUCCESS files. If a _SUCCESS appears in the inner partitioning dir, partition discovery will treat that _SUCCESS file as a data file. Then, partition discovery will fail because it finds that the dir structure is not valid. We should ignore those `_SUCCESS` files. In future, it is better to ignore all files/dirs starting with `_` or `.`. This PR does not make this change. I am thinking about making this change simple, so we can consider of getting it in branch 1.6. To ignore all files/dirs starting with `_` or `, the main change is to let ParquetRelation have another way to get metadata files. Right now, it relies on FileStatusCache's cachedLeafStatuses, which returns file statuses of both metadata files (e.g. metadata files used by parquet) and data files, which requires more changes. https://issues.apache.org/jira/browse/SPARK-13207 Author: Yin Huai <yhuai@databricks.com> Closes #11697 from yhuai/SPARK13207_branch16. 15 March 2016, 06:42:05 UTC
589d042 Bjorn Jonsson 14 March 2016, 19:27:49 UTC [MINOR][COMMON] Fix copy-paste oversight in variable naming ## What changes were proposed in this pull request? JavaUtils.java has methods to convert time and byte strings for internal use, this change renames a variable used in byteStringAs(), from timeError to byteError. Author: Bjorn Jonsson <bjornjon@gmail.com> Closes #11695 from bjornjon/master. (cherry picked from commit e06493cb7b790623a9106241a8d496ecea703328) Signed-off-by: Andrew Or <andrew@databricks.com> 14 March 2016, 19:27:56 UTC
13bd122 Daniel Santana 14 March 2016, 19:26:08 UTC [MINOR][DOCS] Added Missing back slashes ## What changes were proposed in this pull request? When studying spark many users just copy examples on the documentation and paste on their terminals and because of that the missing backlashes lead them run into some shell errors. The added backslashes avoid that problem for spark users with that behavior. ## How was this patch tested? I generated the documentation locally using jekyll and checked the generated pages Author: Daniel Santana <mestresan@gmail.com> Closes #11699 from danielsan/master. (cherry picked from commit 9f13f0fc1724e407d3be54152b82150a16395421) Signed-off-by: Andrew Or <andrew@databricks.com> 14 March 2016, 19:26:16 UTC
3519ce9 Jacky Li 14 March 2016, 01:44:02 UTC [SQL] fix typo in DataSourceRegister ## What changes were proposed in this pull request? fix typo in DataSourceRegister ## How was this patch tested? found when going through latest code Author: Jacky Li <jacky.likun@huawei.com> Closes #11686 from jackylk/patch-12. (cherry picked from commit f3daa099bf60edbd6ebf997c00e46db1e09f6dda) Signed-off-by: Reynold Xin <rxin@databricks.com> 14 March 2016, 01:44:12 UTC
5e08db3 Bjorn Jonsson 13 March 2016, 10:18:24 UTC [SPARK-13810][CORE] Add Port Configuration Suggestions on Bind Exceptions ## What changes were proposed in this pull request? Currently, when a java.net.BindException is thrown, it displays the following message: java.net.BindException: Address already in use: Service '$serviceName' failed after 16 retries! This change adds port configuration suggestions to the BindException, for example, for the UI, it now displays java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries! Consider explicitly setting the appropriate port for 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries. ## How was this patch tested? Manual tests Author: Bjorn Jonsson <bjornjon@gmail.com> Closes #11644 from bjornjon/master. (cherry picked from commit 515e4afbc7ec957609451ea75772d6ef1b914908) Signed-off-by: Sean Owen <sowen@cloudera.com> 13 March 2016, 10:18:43 UTC
db4795a Oscar D. Lara Yejas 11 March 2016, 01:10:23 UTC [SPARK-13327][SPARKR] Added parameter validations for colnames<- Author: Oscar D. Lara Yejas <odlaraye@oscars-mbp.attlocal.net> Author: Oscar D. Lara Yejas <odlaraye@oscars-mbp.usca.ibm.com> Closes #11220 from olarayej/SPARK-13312-3. (cherry picked from commit 416e71af4d26e67afb715ea1d625341cdea4873d) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu> 11 March 2016, 01:10:45 UTC
078c714 Dongjoon Hyun 11 March 2016, 01:07:18 UTC [MINOR][DOC] Fix supported hive version in doc ## What changes were proposed in this pull request? Today, Spark 1.6.1 and updated docs are release. Unfortunately, there is obsolete hive version information on docs: [Building Spark](http://spark.apache.org/docs/latest/building-spark.html#building-with-hive-and-jdbc-support). This PR fixes the following two lines. ``` -By default Spark will build with Hive 0.13.1 bindings. +By default Spark will build with Hive 1.2.1 bindings. -# Apache Hadoop 2.4.X with Hive 13 support +# Apache Hadoop 2.4.X with Hive 1.2.1 support ``` `sql/README.md` file also describe ## How was this patch tested? Manual. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Dongjoon Hyun <dongjoon@apache.org> Closes #11639 from dongjoon-hyun/fix_doc_hive_version. (cherry picked from commit 88fa866620b85522bae2f71601cebb744c09ba7e) Signed-off-by: Reynold Xin <rxin@databricks.com> 11 March 2016, 01:07:26 UTC
07ace27 Sean Owen 10 March 2016, 15:17:37 UTC [SPARK-13663][CORE] Upgrade Snappy Java to 1.1.2.1 Update snappy to 1.1.2.1 to pull in a single fix -- the OOM fix we already worked around. Supersedes https://github.com/apache/spark/pull/11524 Jenkins tests. Author: Sean Owen <sowen@cloudera.com> Closes #11631 from srowen/SPARK-13663. (cherry picked from commit 927e22eff894cb9b4fe8e5de3578517997292a8a) Signed-off-by: Sean Owen <sowen@cloudera.com> 10 March 2016, 15:27:08 UTC
60cb270 Yin Huai 10 March 2016, 02:41:38 UTC Revert "[SPARK-13760][SQL] Fix BigDecimal constructor for FloatType" This reverts commit 926e9c45a21c5b71ef0832d63b8dae7d4f3d8826. 10 March 2016, 02:43:54 UTC
8a1bd58 Sameer Agarwal 10 March 2016, 02:16:29 UTC [SPARK-13760][SQL] Fix BigDecimal constructor for FloatType ## What changes were proposed in this pull request? A very minor change for using `BigDecimal.decimal(f: Float)` instead of `BigDecimal(f: float)`. The latter is deprecated and can result in inconsistencies due to an implicit conversion to `Double`. ## How was this patch tested? N/A cc yhuai Author: Sameer Agarwal <sameer@databricks.com> Closes #11597 from sameeragarwal/bigdecimal. (cherry picked from commit 926e9c45a21c5b71ef0832d63b8dae7d4f3d8826) Signed-off-by: Yin Huai <yhuai@databricks.com> 10 March 2016, 02:17:16 UTC
bea91a9 Davies Liu 09 March 2016, 20:05:34 UTC [SPARK-13242] [SQL] codegen fallback in case-when if there many branches ## What changes were proposed in this pull request? If there are many branches in a CaseWhen expression, the generated code could go above the 64K limit for single java method, will fail to compile. This PR change it to fallback to interpret mode if there are more than 20 branches. ## How was this patch tested? Add tests Author: Davies Liu <davies@databricks.com> Closes #11606 from davies/fix_when_16. 09 March 2016, 20:05:34 UTC
95105b0 Andy Sloane 09 March 2016, 10:25:47 UTC [SPARK-13631][CORE] Thread-safe getLocationsWithLargestOutputs ## What changes were proposed in this pull request? If a job is being scheduled in one thread which has a dependency on an RDD currently executing a shuffle in another thread, Spark would throw a NullPointerException. This patch synchronizes access to `mapStatuses` and skips null status entries (which are in-progress shuffle tasks). ## How was this patch tested? Our client code unit test suite, which was reliably reproducing the race condition with 10 threads, shows that this fixes it. I have not found a minimal test case to add to Spark, but I will attempt to do so if desired. The same test case was tripping up on SPARK-4454, which was fixed by making other DAGScheduler code thread-safe. shivaram srowen Author: Andy Sloane <asloane@tetrationanalytics.com> Closes #11505 from a1k0n/SPARK-13631. (cherry picked from commit cbff2803ef117d7cffe6f05fc1bbd395a1e9c587) Signed-off-by: Sean Owen <sowen@cloudera.com> 09 March 2016, 10:26:00 UTC
8ec4f15 Josh Rosen 09 March 2016, 00:28:22 UTC [SPARK-13755] Escape quotes in SQL plan visualization node labels When generating Graphviz DOT files in the SQL query visualization we need to escape double-quotes inside node labels. This is a followup to #11309, which fixed a similar graph in Spark Core's DAG visualization. Author: Josh Rosen <joshrosen@databricks.com> Closes #11587 from JoshRosen/graphviz-escaping. (cherry picked from commit 81f54acc9cc0fb9d4ee552f6f56a26c78654a33b) Signed-off-by: Josh Rosen <joshrosen@databricks.com> 09 March 2016, 00:30:06 UTC
back to top