https://github.com/apache/spark
Revision bb0cce990c214d4ca9cf3828940a2ca5350acf79 authored by Yuming Wang on 14 June 2022, 07:43:20 UTC, committed by Dongjoon Hyun on 14 June 2022, 07:43:28 UTC
### What changes were proposed in this pull request?

This PR adds `ReplaceCTERefWithRepartition` into nonExcludableRules list.

### Why are the changes needed?

It will throw exception if user `set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ReplaceCTERefWithRepartition` before running this query:
```sql
SELECT
  (SELECT avg(id) FROM range(10)),
  (SELECT sum(id) FROM range(10)),
  (SELECT count(distinct id) FROM range(10))
```
Exception:
```
Caused by: java.lang.AssertionError: assertion failed: No plan for WithCTE
:- CTERelationDef 0, true
:  +- Project [named_struct(min(id), min(id)#223L, sum(id), sum(id)#226L, count(DISTINCT id), count(DISTINCT id)#229L) AS mergedValue#240]
:     +- Aggregate [min(id#221L) AS min(id)#223L, sum(id#221L) AS sum(id)#226L, count(distinct id#221L) AS count(DISTINCT id)#229L]
:        +- Range (0, 10, step=1, splits=None)
+- Project [scalar-subquery#218 [].min(id) AS scalarsubquery()#230L, scalar-subquery#219 [].sum(id) AS scalarsubquery()#231L, scalar-subquery#220 [].count(DISTINCT id) AS scalarsubquery()#232L]
   :  :- CTERelationRef 0, true, [mergedValue#240]
   :  :- CTERelationRef 0, true, [mergedValue#240]
   :  +- CTERelationRef 0, true, [mergedValue#240]
   +- OneRowRelation
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Unit test.

Closes #36847 from wangyum/SPARK-39448.

Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 0b785b3c77374fa7736f01bb55e87c796985ae14)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 47b8eee
History
Tip revision: bb0cce990c214d4ca9cf3828940a2ca5350acf79 authored by Yuming Wang on 14 June 2022, 07:43:20 UTC
[SPARK-39448][SQL] Add `ReplaceCTERefWithRepartition` into `nonExcludableRules` list
Tip revision: bb0cce9

README.md

back to top