Delta Lake Catalog
Delta Lake Catalog is compatible with the Trino Connector, using the Delta Lake Connector to access Delta Lake tables.
This feature is supported starting from Doris version 3.0.1.
This is an experimental feature.
Instructions for Useβ
- Before accessing Delta Lake tables with the Trino Catalog, you must first compile the Trino Delta Lake plugin and place it in the specified directory. Refer to the following steps for specific instructions.
- The current Doris is compatible with the Trino 435 version plugin. Using a Trino plugin lower or higher than version 435 may cause compatibility issues.
- Time Travel feature of Delta Lake is not supported at the moment.
- Querying Delta Lake history information is not supported at the moment.
Compiling the Delta Lake Connector Pluginβ
JDK version 17 is required.
$ git clone https://github.com/apache/Doris-thirdparty.git
$ cd Doris-thirdparty
$ git checkout trino-435
$ cd plugin/trino-delta-lake
$ mvn clean install -DskipTest
After compiling, you will find the trino-delta-lake-435
directory under trino/plugin/trino-delta-lake/target/
.
You can also directly download the precompiled trino-delta-lake-435-20240724.tar.gz and extract it.
Deploying the Delta Lake Connectorβ
Place the trino-delta-lake-435/
directory in the connectors/
directory of all FE and BE deployment paths. (If it does not exist, you can create it manually).
βββ bin
βββ conf
βββ connectors
β βββ trino-delta-lake-435
...
After deployment, it is recommended to restart the FE and BE nodes to ensure the Connector is correctly loaded.
Creating the Delta Lake Catalogβ
create catalog delta_lake_catalog properties (
"type"="trino-connector",
"trino.connector.name"="delta_lake",
"trino.hive.metastore"="thrift",
"trino.hive.metastore.uri"= "thrift://ip:portrait ",
"trino.hive.config.resources"="/path/to/core-site.xml,/path/to/hdfs-site.xml"
);
Note: Properties prefixed with trino.
are native Trino properties. For more information about Delta Lake properties, refer to the Trino documentation.
Column Type Mappingβ
Delta Lake | Trino | Doris |
---|---|---|
BOOLEAN | BOOLEAN | BOOLEAN |
INTEGER | INTEGER | INT |
BYTE | TINYINT | TINYINT |
SHORT | SMALLINT | SMALLINT |
LONG | BIGINT | BIGINT |
FLOAT | REAL | FLOAT |
DOUBLE | DOUBLE | DOUBLE |
DECIMAL(p,s) | DECIMAL(p,s) | DECIMAL(p,s) |
STRING | VARCHAR | STRING |
BINARY | VARBINARY | STRING |
DATE | DATE | DATE |
TIMESTAMPNTZ (TIMESTAMP_NTZ) | TIMESTAMP(6) | DATETIME |
TIMESTAMP | TIMESTAMP(3) WITH TIME ZONE | DATETIME |
ARRAY | ARRAY | ARRAY |
MAP | MAP | MAP |
STRUCT | ROW | STRUCT |