https://github.com/apache/spark
Revision 968dd47d9ecb52b0627a0c9caa815b55715429ab authored by Jungtaek Lim on 09 March 2022, 02:03:57 UTC, committed by Jungtaek Lim on 09 March 2022, 02:22:22 UTC
### What changes were proposed in this pull request? This PR fixes the StateSchemaCompatibilityChecker which mistakenly swapped `from` (should be provided schema) and `to` (should be existing schema). ### Why are the changes needed? The bug mistakenly allows the case where it should not be allowed, and disallows the case where it should be allowed. That allows nullable column to be stored into non-nullable column, which should be prohibited. This is less likely making runtime problem since state schema is conceptual one and row can be stored even not respecting the state schema. The opposite case is worse, that disallows non-nullable column to be stored into nullable column, which should be allowed. Spark fails the query for this case. ### Does this PR introduce _any_ user-facing change? Yes, after the fix, storing non-nullable column into nullable column for state will be allowed, which should have been allowed. ### How was this patch tested? Modified UTs. Closes #35731 from HeartSaVioR/SPARK-38412. Authored-by: Jungtaek Lim <kabhwan.opensource@gmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com> (cherry picked from commit 43c7824bba40ebfb64dcd50d8d0e84b5a4d3c8c7) Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
1 parent 0496173
Tip revision: 968dd47d9ecb52b0627a0c9caa815b55715429ab authored by Jungtaek Lim on 09 March 2022, 02:03:57 UTC
[SPARK-38412][SS] Fix the swapped sequence of from and to in StateSchemaCompatibilityChecker
[SPARK-38412][SS] Fix the swapped sequence of from and to in StateSchemaCompatibilityChecker
Tip revision: 968dd47
CONTRIBUTING.md
## Contributing to Spark
*Before opening a pull request*, review the
[Contributing to Spark guide](https://spark.apache.org/contributing.html).
It lists steps that are required before creating a PR. In particular, consider:
- Is the change important and ready enough to ask the community to spend time reviewing?
- Have you searched for existing, related JIRAs and pull requests?
- Is this a new feature that can stand alone as a [third party project](https://spark.apache.org/third-party-projects.html) ?
- Is the change being proposed clearly explained and motivated?
When you contribute code, you affirm that the contribution is your original work and that you
license the work to the project under the project's open source license. Whether or not you
state this explicitly, by submitting any copyrighted material via pull request, email, or
other means you agree to license the material under the project's open source license and
warrant that you have the legal authority to do so.
Computing file changes ...
