Skip to main content

Browse the archive

Revision bfa0d13f3f6b4b662ad0f355a8db00dd1244a698 authored by Herman van Hovell on 06 August 2024, 01:54:09 UTC, committed by Hyukjin Kwon on 06 August 2024, 01:54:09 UTC 
[SPARK-49083][CONNECT] Allow from_xml and from_json to natively work with json schemas
 
### What changes were proposed in this pull request?
We allow the `JsonToStructs` and `XmlToStructs` expressions to use a json schema.

### Why are the changes needed?
A couple of reasons:
- We want to use a reference to the `from_json` and `from_xml` methods in the Column API in order to make unification of the Classic and Connect Scala clients possible.
- Reduce the amount of duplication between the Function API and the SparkConnectPlanner.
- Make DataFrame and SQL API behave the same.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Existing tests.

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #47573 from hvanhovell/SPARK-49083.

Authored-by: Herman van Hovell <herman@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent da5912a
Raw File
spark-sql.cmd 
@echo off

rem
rem Licensed to the Apache Software Foundation (ASF) under one or more
rem contributor license agreements.  See the NOTICE file distributed with
rem this work for additional information regarding copyright ownership.
rem The ASF licenses this file to You under the Apache License, Version 2.0
rem (the "License"); you may not use this file except in compliance with
rem the License.  You may obtain a copy of the License at
rem
rem    http://www.apache.org/licenses/LICENSE-2.0
rem
rem Unless required by applicable law or agreed to in writing, software
rem distributed under the License is distributed on an "AS IS" BASIS,
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
rem See the License for the specific language governing permissions and
rem limitations under the License.
rem

rem This is the entry point for running SparkSQL. To avoid polluting the
rem environment, it just launches a new cmd to do the real work.

rem The outermost quotes are used to prevent Windows command line parse error
rem when there are some quotes in parameters, see SPARK-21877.
cmd /V /E /C ""%~dp0spark-sql2.cmd" %*"
back to top