Load Data with Spark SQL through the Spark Connector
The Spark Connector lets you interact with an Analytics Engine database using Apache Spark. A functional setup requires a working Spark installation and the Spark Connector jar file. Please contact Actian support for a copy of the Spark Connector jar and installation requirements for Spark.
In the following example, we assume that the user is connected to a spark-shell. Please ensure that the Spark Connector jar file is in the classpath. A new Spark session must be created in the following manner:
import org.apache.spark.sql._
import com.actian.spark_connector.extensions.SparkConnectorDataSourceV2Strategy
val spark = SparkSession.builder().withExtensions { extensions =>
extensions.injectPlannerStrategy(sp => new SparkConnectorDataSourceV2Strategy(sp))
}.getOrCreate()
Assuming that you have created a table in Analytics Engine as follows:
CREATE TABLE test(col1 int)
You can reference this table in Spark as follows:
spark.sql("""CREATE TEMPORARY VIEW AnalyticsEngine_table
USING com.actian.spark_connector.sql.AnalyticsEngineSource
OPTIONS (
host "localhost",
instance "VW",
database "testdb",
table "test",
user "actian",
password "actian"
)""")
You can load data into the table as shown below:
import org.apache.spark.sql.types.IntegerType
import org.apache.spark.sql.types.StructField
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.Row
import scala.jdk.CollectionConverters._
val values: Seq[Row] = Seq(Row(1))
val schema = StructType(Seq(StructField("col1", IntegerType, nullable = true)))
val valuesDF: DataFrame = spark.createDataFrame(values.asJava, schema)
valuesDF.createTempView("spark_table")
spark.sql("insert into analyticsengine_table select * from spark_table")
To view the inserted data:
val res = spark.sql("select * from analyticsengine_table")
res.show()
Last modified date: 09/11/2026