https://github.com/apache/spark
Revision 968dd47d9ecb52b0627a0c9caa815b55715429ab authored by Jungtaek Lim on 09 March 2022, 02:03:57 UTC, committed by Jungtaek Lim on 09 March 2022, 02:22:22 UTC
### What changes were proposed in this pull request?

This PR fixes the StateSchemaCompatibilityChecker which mistakenly swapped `from` (should be provided schema) and `to` (should be existing schema).

### Why are the changes needed?

The bug mistakenly allows the case where it should not be allowed, and disallows the case where it should be allowed.

That allows nullable column to be stored into non-nullable column, which should be prohibited. This is less likely making runtime problem since state schema is conceptual one and row can be stored even not respecting the state schema.

The opposite case is worse, that disallows non-nullable column to be stored into nullable column, which should be allowed. Spark fails the query for this case.

### Does this PR introduce _any_ user-facing change?

Yes, after the fix, storing non-nullable column into nullable column for state will be allowed, which should have been allowed.

### How was this patch tested?

Modified UTs.

Closes #35731 from HeartSaVioR/SPARK-38412.

Authored-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
(cherry picked from commit 43c7824bba40ebfb64dcd50d8d0e84b5a4d3c8c7)
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
1 parent 0496173
History
Tip revision: 968dd47d9ecb52b0627a0c9caa815b55715429ab authored by Jungtaek Lim on 09 March 2022, 02:03:57 UTC
[SPARK-38412][SS] Fix the swapped sequence of from and to in StateSchemaCompatibilityChecker
Tip revision: 968dd47

README.md

back to top