https://github.com/apache/spark
Revision fd998c8a6783c0c8aceed8dcde4017cd479e42c8 authored by Bruce Robbins on 04 May 2022, 09:22:11 UTC, committed by Gengliang Wang on 04 May 2022, 09:22:24 UTC
### What changes were proposed in this pull request? In `DivideYMInterval#doGenCode` and `DivideDTInterval#doGenCode`, rely on the operand variable names provided by `nullSafeCodeGen` rather than calling `genCode` on the operands twice. ### Why are the changes needed? `DivideYMInterval#doGenCode` and `DivideDTInterval#doGenCode` call `genCode` on the operands twice (once directly, and once indirectly via `nullSafeCodeGen`). However, if you call `genCode` on an operand twice, you might not get back the same variable name for both calls (e.g., when the operand is not a `BoundReference` or if whole-stage codegen is turned off). When that happens, `nullSafeCodeGen` generates initialization code for one set of variables, but the divide expression generates usage code for another set of variables, resulting in compilation errors like this: ``` spark-sql> create or replace temp view v1 as > select * FROM VALUES > (interval '10' months, interval '10' day, 2) > as v1(period, duration, num); Time taken: 2.81 seconds spark-sql> cache table v1; Time taken: 2.184 seconds spark-sql> select period/(num + 3) from v1; 22/05/03 08:56:37 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 44: Expression "project_value_2" is not an rvalue ... 22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back to interpreter mode ... 0-2 Time taken: 0.149 seconds, Fetched 1 row(s) spark-sql> select duration/(num + 3) from v1; 22/05/03 08:57:29 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 54: Expression "project_value_2" is not an rvalue ... 22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back to interpreter mode ... 2 00:00:00.000000000 Time taken: 0.089 seconds, Fetched 1 row(s) ``` The error is not fatal (unless you have `spark.sql.codegen.fallback` set to `false`), but it muddies the log and can slow the query (since the expression is interpreted). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests (unit tests run with `spark.sql.codegen.fallback` set to `false`, so the new tests fail without the fix). Closes #36442 from bersprockets/interval_div_issue. Authored-by: Bruce Robbins <bersprockets@gmail.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit ca87bead23ca32a05c6a404a91cea47178f63e70) Signed-off-by: Gengliang Wang <gengliang@apache.org>
1 parent d3aadb4
Tip revision: fd998c8a6783c0c8aceed8dcde4017cd479e42c8 authored by Bruce Robbins on 04 May 2022, 09:22:11 UTC
[SPARK-39093][SQL] Avoid codegen compilation error when dividing year-month intervals or day-time intervals by an integral
[SPARK-39093][SQL] Avoid codegen compilation error when dividing year-month intervals or day-time intervals by an integral
Tip revision: fd998c8
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...