| b87d31d | Patrick Wendell | 10 December 2013, 22:35:56 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 10 December 2013, 22:35:56 UTC |
| 628ca85 | Patrick Wendell | 10 December 2013, 22:24:26 UTC | Small bug fix in YARN build patch | 10 December 2013, 22:24:26 UTC |
| 9415d2d | Patrick Wendell | 10 December 2013, 22:22:34 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit 909a9e4d11eccea03980a8ed7ba7f9f27c68e33a. | 10 December 2013, 22:22:34 UTC |
| d101dfe | Patrick Wendell | 10 December 2013, 22:22:31 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 7ca53692c821ead3a50520fe51e229d9fc1f856d. | 10 December 2013, 22:22:31 UTC |
| 7ca5369 | Patrick Wendell | 10 December 2013, 21:22:17 UTC | [maven-release-plugin] prepare for next development iteration | 10 December 2013, 21:22:17 UTC |
| 909a9e4 | Patrick Wendell | 10 December 2013, 21:22:11 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 10 December 2013, 21:22:11 UTC |
| b0e50f8 | Patrick Wendell | 10 December 2013, 21:01:26 UTC | Merge pull request #250 from pwendell/master README incorrectly suggests build sources spark-env.sh This is misleading because the build doesn't source that file. IMO it's better to force people to specify build environment variables on the command line always, like we do in every example, so I'm just removing this doc. (cherry picked from commit d2efe13574090e93c600adeacc7f6356bc196e6c) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 10 December 2013, 21:06:32 UTC |
| c45a267 | Patrick Wendell | 10 December 2013, 08:08:02 UTC | Small fix from prior commit | 10 December 2013, 08:08:02 UTC |
| 22fcb78 | Patrick Wendell | 10 December 2013, 08:02:27 UTC | Add missing dependencies | 10 December 2013, 08:05:44 UTC |
| 8129328 | Patrick Wendell | 10 December 2013, 07:13:15 UTC | Updating CHANGES and one fix from last merge | 10 December 2013, 07:13:15 UTC |
| d03589d | Patrick Wendell | 10 December 2013, 07:10:00 UTC | Merge pull request #248 from colorant/branch-0.8 Fix POM file for mvn assembly on hadoop 2.2 Yarn This is the fix for maven YARN build on hadoop 2.2 | 10 December 2013, 07:10:00 UTC |
| d614945 | Patrick Wendell | 10 December 2013, 07:08:44 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit 7e5564cc788c33a3048914d61e90eff88b1a3903. | 10 December 2013, 07:08:44 UTC |
| 52c0890 | Patrick Wendell | 10 December 2013, 07:08:33 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 216b473df0d6912f4905204402cfe04568e0560c. | 10 December 2013, 07:08:33 UTC |
| e468f81 | Raymond Liu | 10 December 2013, 05:30:57 UTC | Fix POM file for mvn assembly on hadoop 2.2 Yarn | 10 December 2013, 05:30:57 UTC |
| 216b473 | Patrick Wendell | 10 December 2013, 02:07:00 UTC | [maven-release-plugin] prepare for next development iteration | 10 December 2013, 02:07:00 UTC |
| 7e5564c | Patrick Wendell | 10 December 2013, 02:06:54 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 10 December 2013, 02:06:54 UTC |
| 4567295 | Patrick Wendell | 10 December 2013, 00:51:36 UTC | Merge pull request #246 from pwendell/master Add missing license headers I found this when doing further audits on the 0.8.1 release candidate. (cherry picked from commit 6169fe14a140146602fb07cfcd13eee6efad98f9) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 10 December 2013, 00:59:44 UTC |
| b6754ab | Patrick Wendell | 10 December 2013, 00:59:22 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit c88a9916a183e7a57b53537531620bbde6d8869a. | 10 December 2013, 00:59:44 UTC |
| 1e8b044 | Patrick Wendell | 10 December 2013, 00:59:20 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit af7498870a4eed0f6e4b9fe37cc69edc022d0d8c. | 10 December 2013, 00:59:44 UTC |
| af74988 | Patrick Wendell | 09 December 2013, 05:46:45 UTC | [maven-release-plugin] prepare for next development iteration | 09 December 2013, 05:46:45 UTC |
| c88a991 | Patrick Wendell | 09 December 2013, 05:46:33 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 09 December 2013, 05:46:33 UTC |
| 5ab8e04 | Patrick Wendell | 09 December 2013, 05:34:53 UTC | Updating CHANGES file | 09 December 2013, 05:36:53 UTC |
| 4b2769b | Patrick Wendell | 09 December 2013, 04:49:20 UTC | Merge pull request #195 from dhardy92/fix_DebScriptPackage [Deb] fix package of Spark classes adding org.apache prefix in scripts embeded in .deb (cherry picked from commit d992ec6d9be30e624c8edb2a50c193ac3cfbab7a) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 09 December 2013, 05:36:52 UTC |
| fde5347 | Patrick Wendell | 09 December 2013, 05:36:30 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit 00d5c734dd4b12e128295518e4bd620fdb13bed7. | 09 December 2013, 05:36:30 UTC |
| 7a72b60 | Patrick Wendell | 09 December 2013, 05:36:27 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 4c555328dd782efc2ab97ae35ea2f3a5b00cb450. | 09 December 2013, 05:36:27 UTC |
| 4c55532 | Patrick Wendell | 09 December 2013, 05:25:06 UTC | [maven-release-plugin] prepare for next development iteration | 09 December 2013, 05:25:06 UTC |
| 00d5c73 | Patrick Wendell | 09 December 2013, 05:24:49 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 09 December 2013, 05:24:49 UTC |
| 71d76a0 | Patrick Wendell | 09 December 2013, 04:50:52 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit bf23794a766d4f94076d2417f128f15465f25495. | 09 December 2013, 04:50:52 UTC |
| dcc678f | Patrick Wendell | 09 December 2013, 04:50:49 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 4ece27df4243a7b0ba2791c3c7bece5aed86d342. | 09 December 2013, 04:50:49 UTC |
| 4ece27d | Patrick Wendell | 08 December 2013, 19:37:12 UTC | [maven-release-plugin] prepare for next development iteration | 08 December 2013, 19:37:12 UTC |
| bf23794 | Patrick Wendell | 08 December 2013, 19:37:06 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 08 December 2013, 19:37:06 UTC |
| 1bc7259 | Patrick Wendell | 08 December 2013, 19:12:03 UTC | Minor documentation fixes | 08 December 2013, 19:12:03 UTC |
| c7058d1 | Patrick Wendell | 08 December 2013, 19:11:49 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit e88e6369d9ac55dff75c230ed5bc96c995b1d620. | 08 December 2013, 19:11:49 UTC |
| 408f50b | Patrick Wendell | 08 December 2013, 19:11:45 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 7f7ac64e2847b7cbdb9274fae75fed981601e7d7. | 08 December 2013, 19:11:45 UTC |
| 7f7ac64 | Patrick Wendell | 08 December 2013, 10:35:36 UTC | [maven-release-plugin] prepare for next development iteration | 08 December 2013, 10:35:36 UTC |
| e88e636 | Patrick Wendell | 08 December 2013, 10:35:29 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 08 December 2013, 10:35:29 UTC |
| 51cd2f0 | Patrick Wendell | 08 December 2013, 06:53:53 UTC | Merge pull request #243 from pwendell/branch-0.8 Improve CHANGES.txt file in branch 0.8 This makes the format consistent with the 0.8.0 release which was nicer. | 08 December 2013, 06:53:53 UTC |
| 871ab60 | Patrick Wendell | 08 December 2013, 06:49:47 UTC | Use consistent CHANGES.txt format | 08 December 2013, 06:51:35 UTC |
| c14f373 | Patrick Wendell | 08 December 2013, 06:34:34 UTC | Merge pull request #241 from pwendell/master Update broken links and add HDP 2.0 version string I ran a link checker on the UI and found several broken links. (cherry picked from commit 1f4a4bccf3cf7376c634bad2ebadfdd4c6f78195) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 08 December 2013, 06:35:31 UTC |
| 473cba2 | Patrick Wendell | 08 December 2013, 05:41:28 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit fba873857133fb87cd53dc4cb0501eea1bd7edbf. | 08 December 2013, 05:41:28 UTC |
| c761914 | Patrick Wendell | 08 December 2013, 05:41:24 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 91505f3f2ace8b26e4dae90d362563bf2faa8fbf. | 08 December 2013, 05:41:24 UTC |
| 91505f3 | Patrick Wendell | 07 December 2013, 21:05:11 UTC | [maven-release-plugin] prepare for next development iteration | 07 December 2013, 21:05:11 UTC |
| fba8738 | Patrick Wendell | 07 December 2013, 21:05:05 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 07 December 2013, 21:05:05 UTC |
| 1d3fa31 | Patrick Wendell | 07 December 2013, 20:54:05 UTC | Revert "[maven-release-plugin] prepare release v0.8.1-incubating" This reverts commit 0f059bd62d1c840713fac0d9c6ee6d9165682c72. | 07 December 2013, 20:54:05 UTC |
| a669605 | Patrick Wendell | 07 December 2013, 20:53:58 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit d0b9fce12d586c464306072fb210cb44b97dddd3. | 07 December 2013, 20:53:58 UTC |
| 9c9e71e | Patrick Wendell | 07 December 2013, 20:47:26 UTC | Merge pull request #241 from pwendell/branch-0.8 Fix race condition in JobLoggerSuite [0.8 branch] I found this when running the tests locally. It's similar to a race condition found when making the 0.8.0 release. | 07 December 2013, 20:47:26 UTC |
| 295734f | Patrick Wendell | 07 December 2013, 20:40:18 UTC | Fix race condition in JobLoggerSuite | 07 December 2013, 20:40:18 UTC |
| d0b9fce | Patrick Wendell | 07 December 2013, 20:31:33 UTC | [maven-release-plugin] prepare for next development iteration | 07 December 2013, 20:31:33 UTC |
| 0f059bd | Patrick Wendell | 07 December 2013, 20:31:25 UTC | [maven-release-plugin] prepare release v0.8.1-incubating | 07 December 2013, 20:31:25 UTC |
| 30bcd84 | Patrick Wendell | 07 December 2013, 20:06:08 UTC | Clean-up of changes file | 07 December 2013, 20:06:08 UTC |
| 92597c0 | Patrick Wendell | 07 December 2013, 19:56:16 UTC | Merge pull request #240 from pwendell/master SPARK-917 Improve API links in nav bar (cherry picked from commit 6494d62fe40ac408b14de3f0f3de8ec896a0ae6e) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 19:58:00 UTC |
| d6e5eab | Patrick Wendell | 07 December 2013, 08:15:41 UTC | typo fix | 07 December 2013, 09:15:20 UTC |
| cfca70e | Patrick Wendell | 07 December 2013, 04:16:15 UTC | Merge pull request #236 from pwendell/shuffle-docs Adding disclaimer for shuffle file consolidation (cherry picked from commit 1b38f5f2774982d524742e987b6cef26ccaae676) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 09:15:20 UTC |
| 80cc4ff | Patrick Wendell | 07 December 2013, 04:29:45 UTC | Merge pull request #237 from pwendell/formatting-fix Formatting fix This is a single-line change. The diff appears larger here due to github being out of sync. (cherry picked from commit 10c3c0c6524d0cf6c59b6f2227bf316cdeb7d06c) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 09:15:20 UTC |
| 4a6aae3 | Patrick Wendell | 07 December 2013, 04:14:56 UTC | Merge pull request #235 from pwendell/master Minor doc fixes and updating README (cherry picked from commit e5d5728b72e58046cc175ab06b5f1c7be4957711) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 09:15:20 UTC |
| 2642312 | Patrick Wendell | 07 December 2013, 01:29:03 UTC | Merge pull request #234 from alig/master Updated documentation about the YARN v2.2 build process (cherry picked from commit 241336add5be07fca5ff6c17eed368df7d0c3e3c) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 09:15:20 UTC |
| 07470d1 | Patrick Wendell | 07 December 2013, 08:15:00 UTC | Small fix for Harvey's patch | 07 December 2013, 09:15:20 UTC |
| 2d3eae2 | Matei Zaharia | 05 December 2013, 07:33:04 UTC | Merge pull request #199 from harveyfeng/yarn-2.2 Hadoop 2.2 migration Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches. Short description for each set of commits: a98f5a0 - "Misc style changes in the 'yarn' package" a67ebf4 - "A few more style fixes in the 'yarn' package" Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code. ab8652f - "Add a 'new-yarn' directory ... " Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here. 4f1c3fa - "Hadoop 2.2 YARN API migration ..." API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too. Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141. a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... " If Spark should be built against Hadoop 2.2, then: a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory. b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing. There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though... (cherry picked from commit 72b696156c8662cae2cef4b943520b4be86148ea) Conflicts: project/SparkBuild.scala streaming/pom.xml | 07 December 2013, 09:15:19 UTC |
| 1e9d084 | Matei Zaharia | 25 November 2013, 23:25:29 UTC | Merge pull request #101 from colorant/yarn-client-scheduler For SPARK-527, Support spark-shell when running on YARN sync to trunk and resubmit here In current YARN mode approaching, the application is run in the Application Master as a user program thus the whole spark context is on remote. This approaching won't support application that involve local interaction and need to be run on where it is launched. So In this pull request I have a YarnClientClusterScheduler and backend added. With this scheduler, the user application is launched locally,While the executor will be launched by YARN on remote nodes with a thin AM which only launch the executor and monitor the Driver Actor status, so that when client app is done, it can finish the YARN Application as well. This enables spark-shell to run upon YARN. This also enable other Spark applications to have the spark context to run locally with a master-url "yarn-client". Thus e.g. SparkPi could have the result output locally on console instead of output in the log of the remote machine where AM is running on. Docs also updated to show how to use this yarn-client mode. (cherry picked from commit eb4296c8f7561aaf8782479dd5cd7c9320b7fa6b) Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala | 07 December 2013, 09:15:19 UTC |
| 20d1f8b | Matei Zaharia | 20 November 2013, 18:36:10 UTC | Merge pull request #191 from hsaputra/removesemicolonscala Cleanup to remove semicolons (;) from Scala code -) The main reason for this PR is to remove semicolons from single statements of Scala code. -) Remove unused imports as I see them -) Fix ASF comment header from some of files (bad copy paste I suppose) (cherry picked from commit 4b895013cc965b37d44fd255656da470a3d2c222) Conflicts: examples/src/main/scala/org/apache/spark/streaming/examples/MQTTWordCount.scala Squash into 191 | 07 December 2013, 09:15:09 UTC |
| 2b76315 | Matei Zaharia | 16 November 2013, 19:44:10 UTC | Merge pull request #178 from hsaputra/simplecleanupcode Simple cleanup on Spark's Scala code Simple cleanup on Spark's Scala code while testing some modules: -) Remove some of unused imports as I found them -) Remove ";" in the imports statements -) Remove () at the end of method calls like size that does not have size effect. (cherry picked from commit 1b5b358309a5adfc12b75b0ebb4254ad8e69f5a0) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 08:02:14 UTC |
| ee22be0 | Matei Zaharia | 20 November 2013, 00:05:44 UTC | Merge pull request #189 from tgravescs/sparkYarnErrorHandling Impove Spark on Yarn Error handling Improve cli error handling and only allow a certain number of worker failures before failing the application. This will help prevent users from doing foolish things and their jobs running forever. For instance using 32 bit java but trying to allocate 8G containers. This loops forever without this change, now it errors out after a certain number of retries. The number of tries is configurable. Also increase the frequency we ping the RM to increase speed at which we get containers if they die. The Yarn MR app defaults to pinging the RM every 1 seconds, so the default of 5 seconds here is fine. But that is configurable as well in case people want to change it. I do want to make sure there aren't any cases that calling stopExecutors in CoarseGrainedSchedulerBackend would cause problems? I couldn't think of any and testing on standalone cluster as well as yarn. (cherry picked from commit aa638ed9c140174a47df082ed5631ffe8e624ee6) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 07 December 2013, 07:29:38 UTC |
| d77c337 | Reynold Xin | 06 December 2013, 07:29:42 UTC | Merge pull request #232 from markhamstra/FiniteWait jobWaiter.synchronized before jobWaiter.wait ...else ``IllegalMonitorStateException`` in ``SimpleFutureAction#ready``. (cherry picked from commit 078049877e123fe7e4c4553e36055de572cab7c4) Signed-off-by: Reynold Xin <rxin@apache.org> | 06 December 2013, 07:30:11 UTC |
| 17ca8a1 | Patrick Wendell | 05 December 2013, 22:32:01 UTC | Merge pull request #231 from pwendell/branch-0.8 Bumping version numbers for 0.8.1 release This bumps various version numbers for the release. Note that we don't bump any of the pom.xml files because they get automatically updated as part of the maven release plug-ins. | 05 December 2013, 22:32:01 UTC |
| d80a489 | Patrick Wendell | 05 December 2013, 21:10:19 UTC | Bumping version numbers for 0.8.1 release | 05 December 2013, 21:10:19 UTC |
| 47fce43 | Patrick Wendell | 05 December 2013, 20:31:24 UTC | Merge pull request #228 from pwendell/master Document missing configs and set shuffle consolidation to false. (cherry picked from commit 5d460253d6080d871cb71efb112ea17be0873771) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 05 December 2013, 20:33:02 UTC |
| 27212ad | Patrick Wendell | 05 December 2013, 01:26:36 UTC | Revert "[maven-release-plugin] prepare release spark-parent-0.8.1-incubating" This reverts commit 15c356c362529347ea87b95e7a6008e0391faceb. | 05 December 2013, 01:26:36 UTC |
| a35e186 | Patrick Wendell | 05 December 2013, 01:26:32 UTC | Revert "[maven-release-plugin] prepare for next development iteration" This reverts commit 2dd1e8f8613b7fab409ed8c72b70c48539d54904. | 05 December 2013, 01:26:32 UTC |
| 2dd1e8f | Patrick Wendell | 05 December 2013, 00:42:46 UTC | [maven-release-plugin] prepare for next development iteration | 05 December 2013, 00:42:46 UTC |
| 15c356c | Patrick Wendell | 05 December 2013, 00:41:55 UTC | [maven-release-plugin] prepare release spark-parent-0.8.1-incubating | 05 December 2013, 00:41:55 UTC |
| 03edfa5 | Patrick Wendell | 05 December 2013, 00:28:06 UTC | Change log for release 0.8.1-incubating | 05 December 2013, 00:28:06 UTC |
| cc33f9f | Patrick Wendell | 04 December 2013, 23:52:07 UTC | Merge pull request #227 from pwendell/master Fix small bug in web UI and minor clean-up. There was a bug where sorting order didn't work correctly for write time metrics. I also cleaned up some earlier code that fixed the same issue for read and write bytes. (cherry picked from commit 182f9baeed8e4cc62ca14ae04413394477a7ccfb) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 04 December 2013, 23:57:47 UTC |
| ba44f21 | Matei Zaharia | 04 December 2013, 18:28:50 UTC | Merge pull request #223 from rxin/transient Mark partitioner, name, and generator field in RDD as @transient. As part of the effort to reduce serialized task size. (cherry picked from commit d6e5473872f405a6f4e466705e33cf893af915c1) Signed-off-by: Patrick Wendell <pwendell@gmail.com> | 04 December 2013, 23:56:58 UTC |
| 31da065 | Reynold Xin | 22 October 2013, 03:33:29 UTC | Merge pull request #95 from aarondav/perftest Minor: Put StoragePerfTester in org/apache/ (cherry picked from commit a51359c917a9ebe379b32ebc53fd093c454ea195) Signed-off-by: Reynold Xin <rxin@apache.org> | 04 December 2013, 22:01:13 UTC |
| daaaee1 | Reynold Xin | 03 December 2013, 22:21:40 UTC | Merge pull request #218 from JoshRosen/spark-970-pyspark-unicode-error Fix UnicodeEncodeError in PySpark saveAsTextFile() (SPARK-970) This fixes [SPARK-970](https://spark-project.atlassian.net/browse/SPARK-970), an issue where PySpark's saveAsTextFile() could throw UnicodeEncodeError when called on an RDD of Unicode strings. Please merge this into master and branch-0.8. (cherry picked from commit 8a3475aed66617772f4e98e9f774b109756eb391) Signed-off-by: Reynold Xin <rxin@apache.org> | 03 December 2013, 22:22:05 UTC |
| 8b091fe | Matei Zaharia | 20 November 2013, 00:11:31 UTC | Merge pull request #181 from BlackNiuza/fix_tasks_number correct number of tasks in ExecutorsUI Index `a` is not `execId` here (cherry picked from commit f568912f85f58ae152db90f199c1f3a002f270c1) Signed-off-by: Reynold Xin <rxin@apache.org> | 03 December 2013, 05:28:13 UTC |
| d21266e | Reynold Xin | 01 December 2013, 20:46:58 UTC | Merge pull request #219 from sundeepn/schedulerexception Scheduler quits when newStage fails The current scheduler thread does not handle exceptions from newStage stage while launching new jobs. The thread fails on any exception that gets triggered at that level, leaving the cluster hanging with no schduler. (cherry picked from commit 740922f25d5f81617fbe02c7bcd1610d6426bbef) Signed-off-by: Reynold Xin <rxin@apache.org> | 01 December 2013, 20:47:30 UTC |
| be9c176 | Matei Zaharia | 26 November 2013, 02:50:18 UTC | Merge pull request #201 from rxin/mappartitions Use the proper partition index in mapPartitionsWIthIndex mapPartitionsWithIndex uses TaskContext.partitionId as the partition index. TaskContext.partitionId used to be identical to the partition index in a RDD. However, pull request #186 introduced a scenario (with partition pruning) that the two can be different. This pull request uses the right partition index in all mapPartitionsWithIndex related calls. Also removed the extra MapPartitionsWIthContextRDD and put all the mapPartitions related functionality in MapPartitionsRDD. (cherry picked from commit 14bb465bb3d65f5b1034ada85cfcad7460034073) Signed-off-by: Reynold Xin <rxin@apache.org> | 26 November 2013, 18:27:41 UTC |
| 9949561 | Reynold Xin | 24 November 2013, 23:50:46 UTC | Merge pull request #197 from aarondav/patrick-fix Fix 'timeWriting' stat for shuffle files Due to concurrent git branches, changes from shuffle file consolidation patch caused the shuffle write timing patch to no longer actually measure the time, since it requires time be measured after the stream has been closed. (cherry picked from commit 972171b9d93b07e8511a2da3a33f897ba033484b) Signed-off-by: Reynold Xin <rxin@apache.org> | 24 November 2013, 23:51:23 UTC |
| c59ce18 | Reynold Xin | 24 November 2013, 03:02:02 UTC | Merge pull request #200 from mateiz/hash-fix AppendOnlyMap fixes - Chose a more random reshuffling step for values returned by Object.hashCode to avoid some long chaining that was happening for consecutive integers (e.g. `sc.makeRDD(1 to 100000000, 100).map(t => (t, t)).reduceByKey(_ + _).count`) - Some other small optimizations throughout (see commit comments) (cherry picked from commit 718cc803f7e0600c9ab265022eb6027926a38010) Signed-off-by: Reynold Xin <rxin@apache.org> | 24 November 2013, 03:04:00 UTC |
| d7ab87e | Reynold Xin | 22 November 2013, 02:26:39 UTC | Merge pull request #193 from aoiwelle/patch-1 Fix Kryo Serializer buffer documentation inconsistency The documentation here is inconsistent with the coded default and other documentation. (cherry picked from commit 086b097e33a2ce622ec6352819bccc92106f43b7) Signed-off-by: Reynold Xin <rxin@apache.org> | 22 November 2013, 02:27:16 UTC |
| d7c6a00 | Reynold Xin | 22 November 2013, 02:12:13 UTC | Merge pull request #196 from pwendell/master TimeTrackingOutputStream should pass on calls to close() and flush(). Without this fix you get a huge number of open files when running shuffles. (cherry picked from commit f20093c3afa68439b1c9010de189d497df787c2a) Signed-off-by: Reynold Xin <rxin@apache.org> | 22 November 2013, 02:13:37 UTC |
| f678e10 | Reynold Xin | 20 November 2013, 23:15:55 UTC | Merge branch 'master' of github.com:tbfenet/incubator-spark PartitionPruningRDD is using index from parent I was getting a ArrayIndexOutOfBoundsException exception after doing union on pruned RDD. The index it was using on the partition was the index in the original RDD not the new pruned RDD. (cherry picked from commit 2fead510f74b962b293de4d724136c24a9825271) Signed-off-by: Reynold Xin <rxin@apache.org> | 20 November 2013, 23:17:28 UTC |
| 37126e8 | Matei Zaharia | 15 November 2013, 03:43:55 UTC | Merge pull request #174 from ahirreddy/master Write Spark UI url to driver file on HDFS This makes the SIMR code path simpler (cherry picked from commit ed25105fd9733acd631dab0993560ac66ffeae16) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:53:16 UTC |
| 8823057 | Matei Zaharia | 13 November 2013, 16:39:05 UTC | Merge pull request #166 from ahirreddy/simr-spark-ui SIMR Backend Scheduler will now write Spark UI URL to HDFS, which is to ... ...be retrieved by SIMR clients (cherry picked from commit 39af914b273e35ff431844951ee8dfadcbc0c400) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:53:04 UTC |
| e134ed5 | Matei Zaharia | 13 November 2013, 03:13:39 UTC | Merge pull request #137 from tgravescs/sparkYarnJarsHdfsRebase Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs. Allows you to specify the files on a different hdfs cluster and it will copy them over. It makes sure permissions are correct and makes sure to put things into public distributed cache so they can be reused amongst users if their permissions are appropriate. Also add a bit of error handling for missing arguments. (cherry picked from commit f49ea28d25728e19e56b140a2f374631c94153bc) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:51:19 UTC |
| af98fbc | Matei Zaharia | 14 November 2013, 00:48:44 UTC | Merge pull request #165 from NathanHowell/kerberos-master spark-assembly.jar fails to authenticate with YARN ResourceManager The META-INF/services/ sbt MergeStrategy was discarding support for Kerberos, among others. This pull request changes to a merge strategy similar to sbt-assembly's default. I've also included an update to sbt-assembly 0.9.2, a minor fix to it's zip file handling. (cherry picked from commit 9290e5bcd2c8e4d8bbf1d0ce1ac09bbf62ece4e0) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:50:23 UTC |
| a64397b | Matei Zaharia | 13 November 2013, 00:26:09 UTC | Merge pull request #153 from ankurdave/stop-spot-cluster Enable stopping and starting a spot cluster Clusters launched using `--spot-price` contain an on-demand master and spot slaves. Because EC2 does not support stopping spot instances, the spark-ec2 script previously could only destroy such clusters. This pull request makes it possible to stop and restart a spot cluster. * The `stop` command works as expected for a spot cluster: the master is stopped and the slaves are terminated. * To start a stopped spot cluster, the user must invoke `launch --use-existing-master`. This launches fresh spot slaves but resumes the existing master. (cherry picked from commit 87f2f4e5c2812351cdd1b2e35e2b12f62eeb3fdc) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:49:40 UTC |
| 6c60768 | Matei Zaharia | 13 November 2013, 00:19:50 UTC | Merge pull request #160 from xiajunluan/JIRA-923 Fix bug JIRA-923 Fix column sort issue in UI for JIRA-923. https://spark-project.atlassian.net/browse/SPARK-923 Signed-off-by: Reynold Xin <rxin@apache.org> Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala (cherry picked from commit b8bf04a085162478a64ca5d7be15d1af2f6a930e) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:46:40 UTC |
| f0d350a | Matei Zaharia | 15 November 2013, 03:44:50 UTC | Merge pull request #175 from kayousterhout/no_retry_not_serializable Don't retry tasks when they fail due to a NotSerializableException As with my previous pull request, this will be unit tested once the Cluster and Local schedulers get merged. (cherry picked from commit dfd40e9f6f87ff1f205944997cdbbb6bb7f0312c) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:44:49 UTC |
| 9d56371 | Matei Zaharia | 15 November 2013, 06:29:28 UTC | Merge pull request #173 from kayousterhout/scheduler_hang Fix bug where scheduler could hang after task failure. When a task fails, we need to call reviveOffers() so that the task can be rescheduled on a different machine. In the current code, the state in ClusterTaskSetManager indicating which tasks are pending may be updated after revive offers is called (there's a race condition here), so when revive offers is called, the task set manager does not yet realize that there are failed tasks that need to be relaunched. This isn't currently unit tested but will be once my pull request for merging the cluster and local schedulers goes in -- at which point many more of the unit tests will exercise the code paths through the cluster scheduler (currently the failure test suite uses the local scheduler, which is why we didn't see this bug before). (cherry picked from commit 96e0fb46309698b685c811a65bd8e1a691389994) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:43:59 UTC |
| 24e238b | Reynold Xin | 18 November 2013, 02:42:18 UTC | Merge pull request #182 from rxin/vector Slightly enhanced PrimitiveVector: 1. Added trim() method 2. Added size method. 3. Renamed getUnderlyingArray to array. 4. Minor documentation update. (cherry picked from commit e2ebc3a9d8bca83bf842b134f2f056c1af0ad2be) Signed-off-by: Reynold Xin <rxin@apache.org> | 18 November 2013, 02:42:41 UTC |
| e7927ad | Reynold Xin | 14 November 2013, 18:32:11 UTC | Merge pull request #169 from kayousterhout/mesos_fix Don't ignore spark.cores.max when using Mesos Coarse mode totalCoresAcquired is decremented but never incremented, causing Spark to effectively ignore spark.cores.max in coarse grained Mesos mode. (cherry picked from commit 1a4cfbea334c7b0dae287eab4c3131c8f4b8a992) Signed-off-by: Reynold Xin <rxin@apache.org> | 14 November 2013, 18:32:27 UTC |
| 41dc566 | Reynold Xin | 14 November 2013, 18:30:36 UTC | Merge pull request #170 from liancheng/hadooprdd-doc-typo Fixed a scaladoc typo in HadoopRDD.scala (cherry picked from commit 5a4f483652c3dc0730d1c908a6a46e2f77d270e1) Signed-off-by: Reynold Xin <rxin@apache.org> | 14 November 2013, 18:30:57 UTC |
| 333859f | Reynold Xin | 14 November 2013, 18:25:48 UTC | Merge pull request #171 from RIA-pierre-borckmans/master Fixed typos in the CDH4 distributions version codes. Nothing important, but annoying when doing a copy/paste... (cherry picked from commit d76f5203af1f714efea30106bef9c2a3a80fd56e) Signed-off-by: Reynold Xin <rxin@apache.org> | 14 November 2013, 18:26:27 UTC |
| c856651 | Matei Zaharia | 12 November 2013, 17:10:05 UTC | Merge pull request #164 from tdas/kafka-fix Made block generator thread safe to fix Kafka bug. This is a very important bug fix. Data can and was being lost in the kafka due to this. (cherry picked from commit dfd1ebc2d1e5c34a5979648e571302ae81a178f5) Signed-off-by: Reynold Xin <rxin@apache.org> | 12 November 2013, 18:22:18 UTC |
| 30786c6 | Matei Zaharia | 10 November 2013, 17:23:56 UTC | Merge pull request #157 from rxin/kryo 3 Kryo related changes. 1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting. 2. Register more internal classes (MapStatus, BlockManagerId). 3. Slightly refactored the internal class registration to allocate less memory. (cherry picked from commit 58d4f6c8a5d9739dc2a3f26f116528457336f0d3) Signed-off-by: Reynold Xin <rxin@apache.org> | 10 November 2013, 19:58:58 UTC |
| 5ce6c75 | Matei Zaharia | 10 November 2013, 01:53:49 UTC | Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker Add spark-tools assembly to spark-class'ss classpath This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with ./spark-class org.apache.spark.tools.JavaAPICompletenessChecker Previously, this tool was run through the `run` script. I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script. (cherry picked from commit 3efc0195625977335914f0a18cf32bd4e9b1d6d4) Signed-off-by: Reynold Xin <rxin@apache.org> | 10 November 2013, 06:31:59 UTC |
| 1d52b50 | Matei Zaharia | 09 November 2013, 21:48:00 UTC | Merge pull request #149 from tgravescs/fixSecureHdfsAccess Fix secure hdfs access for spark on yarn https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn. The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage. (cherry picked from commit 8af99f2356ed19fe43d722ada02f8802cbd46d40) Signed-off-by: Reynold Xin <rxin@apache.org> | 09 November 2013, 23:47:40 UTC |
| 32a0c4f | Reynold Xin | 09 November 2013, 23:40:29 UTC | Merge pull request #155 from rxin/jobgroup Don't reset job group when a new job description is set. (cherry picked from commit 83bf1920c85c33d8d725831b28bc5d11b53c27bd) Signed-off-by: Reynold Xin <rxin@apache.org> | 09 November 2013, 23:40:54 UTC |
