e9eba8c | Patrick Wendell | 13 September 2013, 21:06:45 UTC | Use different Hadoop version for YARN artifacts. This uses a seperate Hadoop version for YARN artifact. This means when people link against spark-yarn, things will resolve correctly. | 13 September 2013, 22:34:57 UTC |
45ec1cc | Patrick Wendell | 12 September 2013, 20:47:09 UTC | Add git scm url for publishing | 12 September 2013, 20:47:31 UTC |
a310de6 | Patrick Wendell | 12 September 2013, 02:36:11 UTC | Merge pull request #926 from kayousterhout/dynamic Changed localProperties to use ThreadLocal (not DynamicVariable). | 12 September 2013, 02:36:11 UTC |
58c7d8b | Matei Zaharia | 12 September 2013, 00:33:42 UTC | Merge pull request #927 from benh/mesos-docs Updated Spark on Mesos documentation. | 12 September 2013, 00:33:42 UTC |
8e2602d | Benjamin Hindman | 11 September 2013, 23:08:54 UTC | More updates to Spark on Mesos documentation. | 11 September 2013, 23:08:54 UTC |
a0f0c1b | Benjamin Hindman | 11 September 2013, 23:05:25 UTC | Updated Spark on Mesos documentation. | 11 September 2013, 23:05:25 UTC |
93c4253 | Kay Ousterhout | 11 September 2013, 19:12:20 UTC | Changed localProperties to use ThreadLocal (not DynamicVariable). The fact that DynamicVariable uses an InheritableThreadLocal can cause problems where the properties end up being shared across threads in certain circumstances. | 11 September 2013, 20:01:39 UTC |
91a59e6 | Patrick Wendell | 11 September 2013, 17:21:48 UTC | Merge pull request #919 from mateiz/jets3t Add explicit jets3t dependency, which is excluded in hadoop-client | 11 September 2013, 17:21:48 UTC |
b9128d3 | Patrick Wendell | 11 September 2013, 17:03:06 UTC | Merge pull request #922 from pwendell/port-change Change default port number from 3030 to 4030. | 11 September 2013, 17:03:06 UTC |
bddf135 | Patrick Wendell | 11 September 2013, 06:12:27 UTC | Change port from 3030 to 4040 | 11 September 2013, 17:01:38 UTC |
e07eef8 | Reynold Xin | 11 September 2013, 14:35:39 UTC | Merge pull request #925 from davidmccauley/master SPARK-894 - Not all WebUI fields delivered VIA JSON | 11 September 2013, 14:35:39 UTC |
5dd875c | David McCauley | 11 September 2013, 09:46:37 UTC | SPARK-894 - Not all WebUI fields delivered VIA JSON | 11 September 2013, 09:46:37 UTC |
8432f27 | Matei Zaharia | 11 September 2013, 06:19:53 UTC | Merge pull request #923 from haoyuan/master fix run-example script | 11 September 2013, 06:19:53 UTC |
45d964b | Haoyuan Li | 11 September 2013, 06:18:22 UTC | better expression | 11 September 2013, 06:18:22 UTC |
d40f140 | Patrick Wendell | 11 September 2013, 06:05:29 UTC | Merge pull request #921 from pwendell/master Fix HDFS access bug with assembly build. | 11 September 2013, 06:05:29 UTC |
56b9407 | Haoyuan Li | 11 September 2013, 06:03:09 UTC | fix run-example script | 11 September 2013, 06:03:09 UTC |
0c1985b | Patrick Wendell | 11 September 2013, 04:38:22 UTC | Fix HDFS access bug with assembly build. Due to this change in HDFS: https://issues.apache.org/jira/browse/HADOOP-7549 there is a bug when using the new assembly builds. The symptom is that any HDFS access results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge strategy in the assembly build which fixes the problem. | 11 September 2013, 05:05:13 UTC |
2425eb8 | Matei Zaharia | 10 September 2013, 18:12:59 UTC | Update Python API features | 10 September 2013, 18:12:59 UTC |
f117dc6 | Matei Zaharia | 10 September 2013, 06:39:25 UTC | Add explicit jets3t dependency, which is excluded in hadoop-client | 10 September 2013, 06:39:25 UTC |
8c14f4b | shivaram | 10 September 2013, 05:07:58 UTC | Merge pull request #917 from pwendell/master Document libgfortran dependency for MLBase | 10 September 2013, 05:07:58 UTC |
cefee1e | Patrick Wendell | 10 September 2013, 04:45:04 UTC | Document fortran dependency for MLBase | 10 September 2013, 04:45:04 UTC |
c81377b | Matei Zaharia | 10 September 2013, 03:16:19 UTC | Merge pull request #915 from ooyala/master Get rid of / improve ugly NPE when Utils.deleteRecursively() fails | 10 September 2013, 03:16:19 UTC |
61d2a01 | Matei Zaharia | 10 September 2013, 01:21:01 UTC | Merge pull request #916 from mateiz/mkdist-fix Fix copy issue in https://github.com/mesos/spark/pull/899 | 10 September 2013, 01:21:01 UTC |
fdb8b0e | Evan Chan | 09 September 2013, 21:29:32 UTC | Style fix: put body of if within curly braces | 09 September 2013, 21:29:32 UTC |
f5a8afa | Matei Zaharia | 09 September 2013, 20:47:56 UTC | Fix copy issue in https://github.com/mesos/spark/pull/899 | 09 September 2013, 20:47:56 UTC |
a85758c | Matei Zaharia | 09 September 2013, 20:45:40 UTC | Merge pull request #907 from stephenh/document_coalesce_shuffle Add better docs for coalesce. | 09 September 2013, 20:45:40 UTC |
2772607 | Evan Chan | 09 September 2013, 19:58:12 UTC | Print out more friendly error if listFiles() fails listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose. | 09 September 2013, 19:58:12 UTC |
084fc36 | Matei Zaharia | 09 September 2013, 19:01:35 UTC | Merge pull request #912 from tgravescs/ganglia-pom Add metrics-ganglia to core pom file | 09 September 2013, 19:01:35 UTC |
2186d93 | Y.CORP.YAHOO.COM\tgraves | 09 September 2013, 17:37:33 UTC | Add metrics-ganglia to core pom file | 09 September 2013, 17:37:33 UTC |
0456384 | Matei Zaharia | 09 September 2013, 16:57:54 UTC | Merge pull request #911 from pwendell/ganglia-sink Adding Manen dependency for Ganglia | 09 September 2013, 16:57:54 UTC |
59003d3 | Stephen Haberman | 09 September 2013, 16:45:03 UTC | Use a set since shuffle could change order. | 09 September 2013, 16:45:03 UTC |
6471bfe | Stephen Haberman | 09 September 2013, 16:44:15 UTC | Reword 'evenly distributed' to 'distributed with a hash partitioner. | 09 September 2013, 16:44:15 UTC |
528fdba | Patrick Wendell | 09 September 2013, 16:32:18 UTC | Adding Manen dependency | 09 September 2013, 16:32:18 UTC |
bf984e2 | Matei Zaharia | 09 September 2013, 06:50:24 UTC | Merge pull request #890 from mridulm/master Fix hash bug | 09 September 2013, 06:50:24 UTC |
e9d4f44 | Reynold Xin | 09 September 2013, 06:36:48 UTC | Merge pull request #909 from mateiz/exec-id-fix Fix an instance where full standalone mode executor IDs were passed to | 09 September 2013, 06:36:48 UTC |
2447b1c | Matei Zaharia | 09 September 2013, 05:27:49 UTC | Merge pull request #910 from mateiz/ml-doc-tweaks Small tweaks to MLlib docs | 09 September 2013, 05:27:49 UTC |
7a5c4b6 | Matei Zaharia | 09 September 2013, 04:47:24 UTC | Small tweaks to MLlib docs | 09 September 2013, 04:47:24 UTC |
7d3204b | Matei Zaharia | 09 September 2013, 04:39:12 UTC | Merge pull request #905 from mateiz/docs2 Job scheduling and cluster mode docs | 09 September 2013, 04:39:12 UTC |
f1f8371 | Matei Zaharia | 09 September 2013, 04:26:11 UTC | Merge pull request #896 from atalwalkar/master updated content | 09 September 2013, 04:26:11 UTC |
b458854 | Matei Zaharia | 09 September 2013, 04:25:49 UTC | Fix some review comments | 09 September 2013, 04:25:49 UTC |
81a8bd4 | Ameet Talwalkar | 09 September 2013, 02:21:30 UTC | respose to PR comments | 09 September 2013, 02:21:30 UTC |
bf280c8 | Ameet Talwalkar | 09 September 2013, 01:41:38 UTC | Merge remote-tracking branch 'upstream/master' | 09 September 2013, 01:41:38 UTC |
f68848d | Patrick Wendell | 09 September 2013, 01:32:16 UTC | Merge pull request #906 from pwendell/ganglia-sink Clean-up of Metrics Code/Docs and Add Ganglia Sink | 09 September 2013, 01:32:16 UTC |
f9b7f58 | Matei Zaharia | 09 September 2013, 01:27:50 UTC | Fix an instance where full standalone mode executor IDs were passed to StandaloneSchedulerBackend instead of the smaller IDs used within Spark (that lack the application name). This was reported by ClearStory in https://github.com/clearstorydata/spark/pull/9. Also fixed some messages that said slave instead of executor. | 09 September 2013, 01:27:50 UTC |
170b386 | Matei Zaharia | 09 September 2013, 00:51:27 UTC | Fix unit test failure due to changed default | 09 September 2013, 00:51:27 UTC |
5ac62db | Ameet Talwalkar | 09 September 2013, 00:39:08 UTC | updates based on comments to PR | 09 September 2013, 00:39:08 UTC |
b4e382c | Patrick Wendell | 08 September 2013, 23:06:49 UTC | Adding sc name in metrics source | 08 September 2013, 23:06:49 UTC |
8026537 | Patrick Wendell | 08 September 2013, 23:06:32 UTC | Fixing package name in template conf | 08 September 2013, 23:06:32 UTC |
0b95799 | Matei Zaharia | 08 September 2013, 22:30:16 UTC | Merge pull request #908 from pwendell/master Fix target JVM version in scala build | 08 September 2013, 22:30:16 UTC |
27bd74c | Patrick Wendell | 08 September 2013, 21:37:45 UTC | Fix target JVM version in scala build | 08 September 2013, 21:37:45 UTC |
5a587fb | Matei Zaharia | 08 September 2013, 20:51:57 UTC | Updated cluster diagram to show caches | 08 September 2013, 20:51:57 UTC |
c190b48 | Patrick Wendell | 08 September 2013, 17:47:45 UTC | Adding more docs and some code cleanup | 08 September 2013, 20:46:28 UTC |
df5fd35 | Stephen Haberman | 08 September 2013, 20:39:03 UTC | Add better docs for coalesce. Include the useful tip that if shuffle=true, coalesce can actually increase the number of partitions. This makes coalesce more like a generic `RDD.repartition` operation. (Ideally this `RDD.repartition` could automatically choose either a coalesce or a shuffle if numPartitions was either less than or greater than, respectively, the current number of partitions.) | 08 September 2013, 20:39:04 UTC |
af8ffdb | Matei Zaharia | 08 September 2013, 20:36:50 UTC | Review comments | 08 September 2013, 20:36:50 UTC |
04cfb3a | Matei Zaharia | 08 September 2013, 17:33:20 UTC | Merge pull request #898 from ilikerps/660 SPARK-660: Add StorageLevel support in Python | 08 September 2013, 17:33:20 UTC |
8de8ee5 | Patrick Wendell | 08 September 2013, 17:07:29 UTC | Ganglia sink | 08 September 2013, 17:08:18 UTC |
c0d3751 | Matei Zaharia | 08 September 2013, 07:44:41 UTC | Some tweaks to CDH/HDP doc | 08 September 2013, 07:44:41 UTC |
a386854 | Aaron Davidson | 08 September 2013, 07:30:47 UTC | Whoopsy daisy | 08 September 2013, 07:30:47 UTC |
f261d2a | Matei Zaharia | 08 September 2013, 04:41:18 UTC | Added cluster overview doc, made logo higher-resolution, and added more details on monitoring | 08 September 2013, 07:29:11 UTC |
651a96a | Matei Zaharia | 07 September 2013, 04:34:12 UTC | More fair scheduler docs and property names. Also changed uses of "job" terminology to "application" when they referred to an entire Spark program, to avoid confusion. | 08 September 2013, 07:29:11 UTC |
98fb698 | Matei Zaharia | 06 September 2013, 04:29:37 UTC | Work in progress: - Add job scheduling docs - Rename some fair scheduler properties - Organize intro page better - Link to Apache wiki for "contributing to Spark" | 08 September 2013, 07:29:11 UTC |
38488ac | Matei Zaharia | 08 September 2013, 07:28:53 UTC | Merge pull request #900 from pwendell/cdh-docs Provide docs to describe running on CDH/HDP cluster. | 08 September 2013, 07:28:53 UTC |
a8e376e | Patrick Wendell | 08 September 2013, 04:16:01 UTC | Merge pull request #904 from pwendell/master Adding Apache license to two files | 08 September 2013, 04:16:01 UTC |
6d21986 | Patrick Wendell | 08 September 2013, 03:46:58 UTC | Adding Apache license to two files | 08 September 2013, 03:46:58 UTC |
c1cc8c4 | Aaron Davidson | 07 September 2013, 21:41:31 UTC | Export StorageLevel and refactor | 07 September 2013, 21:41:31 UTC |
22b982d | Patrick Wendell | 07 September 2013, 21:37:54 UTC | File rename | 07 September 2013, 21:38:54 UTC |
cfde85e | Matei Zaharia | 07 September 2013, 20:53:08 UTC | Merge pull request #901 from ooyala/2013-09/0.8-doc-changes 0.8 Doc changes for make-distribution.sh | 07 September 2013, 20:53:08 UTC |
4a7813a | Matei Zaharia | 07 September 2013, 20:52:24 UTC | Merge pull request #903 from rxin/resulttask Fixed the bug that ResultTask was not properly deserializing outputId. | 07 September 2013, 20:52:24 UTC |
61c4762 | Patrick Wendell | 07 September 2013, 18:51:40 UTC | Changes based on feedback | 07 September 2013, 18:55:10 UTC |
8001687 | Aaron Davidson | 07 September 2013, 16:28:39 UTC | Remove reflection, hard-code StorageLevels The sc.StorageLevel -> StorageLevel pathway is a bit janky, but otherwise the shell would have to call a private method of SparkContext. Having StorageLevel available in sc also doesn't seem like the end of the world. There may be a better solution, though. As for creating the StorageLevel object itself, this seems to be the best way in Python 2 for creating singleton, enum-like objects: http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python | 07 September 2013, 16:34:07 UTC |
be1ee28 | Evan Chan | 07 September 2013, 15:56:24 UTC | CR feedback from Matei | 07 September 2013, 15:56:24 UTC |
afe46ba | Matei Zaharia | 07 September 2013, 14:28:51 UTC | Merge pull request #892 from jey/fix-yarn-assembly YARN build fixes | 07 September 2013, 14:28:51 UTC |
210eae2 | Reynold Xin | 07 September 2013, 13:59:47 UTC | Fixed the bug that ResultTask was not properly deserializing outputId. | 07 September 2013, 13:59:47 UTC |
b8a0b6e | Aaron Davidson | 06 September 2013, 22:36:04 UTC | Memoize StorageLevels read from JVM | 06 September 2013, 22:36:04 UTC |
2eebeff | Patrick Wendell | 06 September 2013, 22:25:22 UTC | Merge pull request #897 from pwendell/master Docs describing Spark monitoring and instrumentation | 06 September 2013, 22:25:22 UTC |
ff1dbf2 | Evan Chan | 06 September 2013, 21:20:44 UTC | Add references to make-distribution.sh | 06 September 2013, 21:20:44 UTC |
88d53f0 | Evan Chan | 06 September 2013, 21:03:44 UTC | "launch" scripts is more accurate terminology | 06 September 2013, 21:03:44 UTC |
5a18b85 | Evan Chan | 06 September 2013, 20:59:43 UTC | Easier way to start the master | 06 September 2013, 20:59:43 UTC |
76d5d2d | Evan Chan | 06 September 2013, 20:53:00 UTC | Add notes about starting spark-shell | 06 September 2013, 20:53:00 UTC |
a2a0cf9 | Patrick Wendell | 06 September 2013, 06:33:44 UTC | Docs describing Spark monitoring and instrumentation | 06 September 2013, 20:52:57 UTC |
e653a9d | Patrick Wendell | 06 September 2013, 19:15:49 UTC | Provide docs to describe running on CDH/HDP cluster. This doc consolidates information relevant to CDH/HDP users in a single place. | 06 September 2013, 20:49:57 UTC |
30a32c8 | Jey Kottalam | 06 September 2013, 17:35:15 UTC | Minor YARN build cleanups | 06 September 2013, 18:31:16 UTC |
7066124 | Jey Kottalam | 06 September 2013, 17:35:06 UTC | Fix YARN assembly generation under Maven | 06 September 2013, 18:31:16 UTC |
35ed09f | Jey Kottalam | 04 September 2013, 18:52:16 UTC | Clarify YARN example | 06 September 2013, 18:31:16 UTC |
1e15feb | Reynold Xin | 06 September 2013, 14:44:05 UTC | Hot fix to resolve the compilation error caused by SPARK-821. | 06 September 2013, 14:44:05 UTC |
ddcb9d3 | Patrick Wendell | 06 September 2013, 06:54:09 UTC | Merge pull request #895 from ilikerps/821 SPARK-821: Don't cache results when action run locally on driver | 06 September 2013, 06:54:09 UTC |
a63d4c7 | Aaron Davidson | 06 September 2013, 06:36:27 UTC | SPARK-660: Add StorageLevel support in Python It uses reflection... I am not proud of that fact, but it at least ensures compatibility (sans refactoring of the StorageLevel stuff). | 06 September 2013, 06:36:27 UTC |
3a04e76 | Aaron Davidson | 06 September 2013, 04:43:26 UTC | Reynold's second round of comments | 06 September 2013, 04:43:26 UTC |
d52edfa | Ameet Talwalkar | 06 September 2013, 04:06:50 UTC | updated content | 06 September 2013, 04:06:50 UTC |
699c331 | Matei Zaharia | 06 September 2013, 03:21:53 UTC | Merge pull request #891 from xiajunluan/SPARK-864 [SPARK-864]DAGScheduler Exception if we delete Worker and StandaloneExecutorBackend then add Worker | 06 September 2013, 03:21:53 UTC |
4f2236a | Aaron Davidson | 06 September 2013, 01:01:43 UTC | Add unit test and address comments | 06 September 2013, 01:06:30 UTC |
1418d18 | Aaron Davidson | 05 September 2013, 22:28:14 UTC | SPARK-821: Don't cache results when action run locally on driver Caching the results of local actions (e.g., rdd.first()) causes the driver to store entire partitions in its own memory, which may be highly constrained. This patch simply makes the CacheManager avoid caching the result of all locally-run computations. | 05 September 2013, 22:34:42 UTC |
7c15e3c | Andrew xia | 04 September 2013, 01:19:11 UTC | Fix bug SPARK-864 | 05 September 2013, 07:56:11 UTC |
5c7494d | Patrick Wendell | 05 September 2013, 05:47:03 UTC | Merge pull request #893 from ilikerps/master SPARK-884: Add unit test to validate Spark JSON output | 05 September 2013, 05:47:03 UTC |
714e7f9 | Aaron Davidson | 05 September 2013, 05:40:08 UTC | Fix line over 100 chars | 05 September 2013, 05:40:08 UTC |
37db141 | Aaron Davidson | 05 September 2013, 04:32:17 UTC | Address Patrick's comments | 05 September 2013, 04:34:20 UTC |
a547866 | Matei Zaharia | 05 September 2013, 04:11:56 UTC | Merge pull request #894 from c0s/master Updating assembly README to reflect recent changes in the build. | 05 September 2013, 04:11:56 UTC |
7c7c7e1 | Konstantin Boudnik | 05 September 2013, 03:47:45 UTC | Updating assembly README to reflect recent changes in the build. | 05 September 2013, 03:54:35 UTC |
9e6f2b6 | Aaron Davidson | 04 September 2013, 22:06:52 UTC | SPARK-884: Add unit test to validate Spark JSON output This unit test simply validates that the outputs of the JsonProtocol methods are syntactically valid JSON. | 04 September 2013, 22:26:46 UTC |
1e2474b | Mridul Muralidharan | 04 September 2013, 02:16:46 UTC | Address review comments - rename toHash to nonNegativeHash | 04 September 2013, 02:16:46 UTC |