Revision 6706c41cb42cbd270a6580385be67b2a2313df27 authored by Liang-Chi Hsieh on 01 August 2024, 19:24:43 UTC, committed by huaxingao on 01 August 2024, 19:24:43 UTC
### What changes were proposed in this pull request? This patch fixes `TransformWithStateExec` so when its `hasInitialState` is false, the `initialState` won't be rewritten by planner incorrectly to produce invalid query plan which will cause unexpected errors for extension rules that rely on the correctness of query plan. ### Why are the changes needed? [SPARK-47363](https://issues.apache.org/jira/browse/SPARK-47363) added the support for users to provide initial state for streaming query. Such query operators like `TransformWithStateExec` might have `hasInitialState` as false which means the initial state related parameters are not used. But when query planner applies rules on the query, it will still apply on the initial state query plan. When `hasInitialState` is false, some related parameters like `initialStateGroupingAttrs` are invalid and some rules will use these invalid parameters to transform the initial state query plan. For example, `EnsureRequirements` may apply invalid Sort and Exchange on the initial query plan. We encountered these invalid query plan in our extension rules. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test ### Was this patch authored or co-authored using generative AI tooling? No Closes #47546 from viirya/fix_initial_state. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: huaxingao <huaxin.gao11@gmail.com>
1 parent 08b1fb5
| File | Mode | Size |
|---|---|---|
| LICENSE-AnchorJS.txt | -rw-r--r-- | 1.1 KB |
| LICENSE-CC0.txt | -rw-r--r-- | 6.9 KB |
| LICENSE-bootstrap.txt | -rw-r--r-- | 1.1 KB |
| LICENSE-cloudpickle.txt | -rw-r--r-- | 1.6 KB |
| LICENSE-d3.min.js.txt | -rw-r--r-- | 731 bytes |
| LICENSE-dagre-d3.txt | -rw-r--r-- | 1.0 KB |
| LICENSE-datatables.txt | -rw-r--r-- | 1.3 KB |
| LICENSE-graphlib-dot.txt | -rw-r--r-- | 1.0 KB |
| LICENSE-jdom.txt | -rw-r--r-- | 2.5 KB |
| LICENSE-join.txt | -rw-r--r-- | 1.5 KB |
| LICENSE-jquery.txt | -rw-r--r-- | 1.1 KB |
| LICENSE-json-formatter.txt | -rw-r--r-- | 547 bytes |
| LICENSE-loose-version.txt | -rw-r--r-- | 13.6 KB |
| LICENSE-matchMedia-polyfill.txt | -rw-r--r-- | 149 bytes |
| LICENSE-modernizr.txt | -rw-r--r-- | 1.1 KB |
| LICENSE-mustache.txt | -rw-r--r-- | 1.2 KB |
| LICENSE-py4j.txt | -rw-r--r-- | 1.4 KB |
| LICENSE-respond.txt | -rw-r--r-- | 1.0 KB |
| LICENSE-sbt-launch-lib.txt | -rw-r--r-- | 1.5 KB |
| LICENSE-sorttable.js.txt | -rw-r--r-- | 937 bytes |
| LICENSE-vis-timeline.txt | -rw-r--r-- | 544 bytes |
Computing file changes ...
