Revision 9389a2ccacce61cbbbc9bbb1b19b2825d932ba11 authored by Herman van Hovell on 05 September 2023, 13:35:12 UTC, committed by Herman van Hovell on 05 September 2023, 13:35:23 UTC
### What changes were proposed in this pull request? Ammonite places all user code inside Helper classes which are nested inside the class it creates for each command. This PR adds a custom code class wrapper for the Ammonite REPL. It makes sure the Helper classes generated by ammonite are always registered as an outer scope immediately. This way we can instantiate classes defined inside the Helper class, even when we execute Spark code as part of the Helper's constructor. ### Why are the changes needed? When you currently define a class and execute a Spark command using that class inside the same cell/line this will fail with an NullPointerException. The reason for that is that we cannot resolve the outer scope needed to instantiate the class. This PR fixes that issue. The following code will now execute successfully (include the curly braces): ```scala { case class Thing(val value: String) val r = (0 to 10).map( value => Thing(value.toString) ) spark.createDataFrame(r) } ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added more tests to the `ReplE2ESuite`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42807 from hvanhovell/SPARK-45072. Authored-by: Herman van Hovell <herman@databricks.com> Signed-off-by: Herman van Hovell <herman@databricks.com> (cherry picked from commit 40943c2748fdd28d970d017cb8ee86c294ee62df) Signed-off-by: Herman van Hovell <herman@databricks.com>
1 parent 0dea7db
File | Mode | Size |
---|---|---|
.github | ||
R | ||
assembly | ||
bin | ||
binder | ||
build | ||
common | ||
conf | ||
connector | ||
core | ||
data | ||
dev | ||
docs | ||
examples | ||
graphx | ||
hadoop-cloud | ||
launcher | ||
licenses | ||
licenses-binary | ||
mllib | ||
mllib-local | ||
project | ||
python | ||
repl | ||
resource-managers | ||
sbin | ||
sql | ||
streaming | ||
tools | ||
.asf.yaml | -rw-r--r-- | 1.3 KB |
.gitattributes | -rw-r--r-- | 130 bytes |
.gitignore | -rw-r--r-- | 1.8 KB |
CONTRIBUTING.md | -rw-r--r-- | 997 bytes |
LICENSE | -rw-r--r-- | 13.1 KB |
LICENSE-binary | -rw-r--r-- | 22.4 KB |
NOTICE | -rw-r--r-- | 2.0 KB |
NOTICE-binary | -rw-r--r-- | 56.5 KB |
README.md | -rw-r--r-- | 4.5 KB |
appveyor.yml | -rw-r--r-- | 2.8 KB |
pom.xml | -rw-r--r-- | 139.1 KB |
scalastyle-config.xml | -rw-r--r-- | 23.1 KB |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...