https://github.com/apache/spark
Revision 5d10586a0065c6845e0e89afc5f22e09baa185b7 authored by Wenchen Fan on 20 September 2017, 16:00:43 UTC, committed by gatorsmile on 20 September 2017, 16:01:25 UTC
## What changes were proposed in this pull request?
Spark with Scala 2.10 fails with a group by cube:
```
spark.range(1).select($"id" as "a", $"id" as "b").write.partitionBy("a").mode("overwrite").saveAsTable("rollup_bug")
spark.sql("select 1 from rollup_bug group by rollup ()").show
```
It can be traced back to https://github.com/apache/spark/pull/15484 , which made `Expand.projections` a lazy `Stream` for group by cube.
In scala 2.10 `Stream` captures a lot of stuff, and in this case it captures the entire query plan which has some un-serializable parts.
This change is also good for master branch, to reduce the serialized size of `Expand.projections`.
## How was this patch tested?
manually verified with Spark with Scala 2.10.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #19289 from cloud-fan/bug.
(cherry picked from commit ce6a71e013c403d0a3690cf823934530ce0ea5ef)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
1 parent 6764408
Tip revision: 5d10586a0065c6845e0e89afc5f22e09baa185b7 authored by Wenchen Fan on 20 September 2017, 16:00:43 UTC
[SPARK-22076][SQL] Expand.projections should not be a Stream
[SPARK-22076][SQL] Expand.projections should not be a Stream
Tip revision: 5d10586
| File | Mode | Size |
|---|---|---|
| .github | ||
| R | ||
| assembly | ||
| bin | ||
| build | ||
| common | ||
| conf | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| external | ||
| graphx | ||
| launcher | ||
| licenses | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .gitattributes | -rw-r--r-- | 40 bytes |
| .gitignore | -rw-r--r-- | 1.2 KB |
| .travis.yml | -rw-r--r-- | 1.7 KB |
| CONTRIBUTING.md | -rw-r--r-- | 995 bytes |
| LICENSE | -rw-r--r-- | 17.5 KB |
| NOTICE | -rw-r--r-- | 24.1 KB |
| README.md | -rw-r--r-- | 3.7 KB |
| appveyor.yml | -rw-r--r-- | 1.9 KB |
| pom.xml | -rw-r--r-- | 94.8 KB |
| scalastyle-config.xml | -rw-r--r-- | 17.4 KB |
Computing file changes ...
