External Table Usage Notes
Note the following when using external tables:
• For writing to external tables, the mode is SaveMode.Append. Some data sources do not support this mode.
• CREATE EXTERNAL TABLE does not validate the supplied values for REFERENCE, FORMAT, or OPTIONS until the external table is used. So, although confusing at first, the following use case will result in an error if the target does not yet exist:
CREATE EXTERNAL TABLE test_table(col INT NOT NULL) USING SPARK
WITH REFERENCE='/opt/user_mount/test_table.json';
SELECT * FROM test_table; \g
Executing . . .
E_VW1213 External table provider reported an error 'java.io.IOException:
No input paths specified in job'.
However, as soon as the Analytics Engine user inserts some data, the external table is created at its original location and a subsequent SELECT statement will succeed:
INSERT INTO test_table VALUES (1);
SELECT * FROM test_table; \g
Executing . . .
(1 row)
┌─────────────┐
│col │
├─────────────┤
│ 1 │
└─────────────┘
(1 row)
IMPORTANT! It is not possible to insert into an external table, referencing a pre-existing file. This is because Spark creates or assumes a folder where it can also store some metadata along with the actual data. This implies, if your external table references a non existing path and you insert into this table, a folder with the given path will be created containing the inserted data and some additional metadata. If you need to add data to an existing file, a workaround is to put this file e.g. a csv file into a folder and use this folder path as external table reference string.
• When multiple parallel data source queries using a ACTIAN_DATASTREAMS type are executed concurrently, one or more Spark jobs may be delayed. If dependencies exist between concurrent Spark jobs, query execution may hang. This occurs because the Spark Connector uses all available CPU cores in the Spark Connector container when executing a single Spark job.
To reduce this risk, set the “spark.default.parallelism” parameter in the spark_provider.conf file to a value lower than the number of CPU cores available to the Spark Connector container. For more information, see
Configuration - Spark 4.0.1 Documentation.
• When using objects from a data source, Analytics Engine requests the schema of the remote object from the Spark Provider. The default timeout that the Analytics Engine waits to receive any schema information from the Spark Provider is 5 minutes. You can configure this timeout in seconds through the "dtsrc_socket_timeout" configuration parameter in the config.dat file.
For example, the following configures timeout to 4 hours:
ii.localhost.config.dtsrc_socket_timeout: 14400
Note: Resolving the schema may take longer for text-based formats like CSV. Schema discovery can be disabled by setting “inferSchema = false” in the data source options. It indicates Spark not to infer data types of the columns of CSV or JSON files and assume all columns as strings.
Last modified date: 09/11/2026