seatunnel

2.3.13 Release Note

This is a great milestone for Apache SeaTunnel project! The release of version 2.3.13 with its so many new features and fixes will significantly boost the efficiency of data integration workflows for our users.

We’re really impressed by the dedication and hard work our contributors have put into this update, and we're confident this new version will be widely welcomed by the community.

Feature

• [Feature][Core][Checkpoint] Add checkpoint api (#10065)
• [Feature][Zeta] Support the min-pause configuration for checkpoints (#9804)
• [Feature][Flink] Add Flink version 1.20.1 support (#9576)
• [Feature][Flink] Support CDC source schema evolution on Flink engine (#9867)
• [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read (#10332)
• [Feature][connector-file-local] Localfile supports splitting and parallel reading of large files of csv, text, and json (#10142)
• [Feature][connectors-v2/connector-file] Support logical Parquet splits (#10239)
• [Feature][Connector-V2] Support sink connector for Apache HugeGraph (#10002)
• [Feature][connector-jdbc] Support source and sink of DuckDB (#10285)
• [Feature][Connector-V2] Support write data into lance dataset (#9894)
• [Feature][Connector-V2] Support sink connector for AWS DSQL #9726 (#9739)
• [Feature][Connector-V2] Support sync_mode=update for FtpFile/SftpFile/LocalFile source (binary) (#10437)
• [Feature][Connector-V2][Elasticsearch] Add Runtime Fields support (Elasticsearch 7.11+) (#10201)
• [Feature][Connector-v2]Resolve null first column in CSV Reader (#10383)
• [Feature][Connector-v2][Kafka] Add support for Schema Registry wire format in Protobuf deserialization (#10183)
• [Feature] [Connector-V2][HBase] Support time-range scan with min/max timestamp in HBaseSource (#10318)
• [Feature][Iceberg] Support schema.partition_keys and ${partition_keys} placeholder (#10389)
• [Feature][Connector-V2] Support hdfs file multi table source read (#9816)
• [Feature][Transform-V2] Support multimodal embeddings (#9673)
• [Feature][Transform-V2] Support RegexExtract Transform (#9829)
• [Feature][Connector-V2] Support `` for the SQL To Paimon converter (#10206)
• [Feature][Kafka source] Inject Kafka record timestamp as EventTime metadata (#9994)
• [Feature][Connector-V2] Postgresql support read TIMESTAMP_TZ (#10048)
• [Feature][Connector-V2] Support HDFS ViewFs Schema (#10117)
• [Feature][Core] Add REST API for viewing Pending queue details (#10078)
• [Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode (#9743)
• [Feature][Connector-V2] Support the dynamic options of the paimon table read (#9981)
• [Feature][Connector-V2][Milvus] Sink writer flush by interval (#9961)
• [Feature][Core] Add PendingJobs info in the response of GetOverviewOperation (#9902)
• [Feature][Connector-V2] Support row range boundaries for HBaseSource (#9983)
• [Feature][Hive] Kerberos renew for >24h Hive and HDFS writes (#9989)
• [Feature][Connector-redis] fix redis cluster bug and add cluster e2e (#9869)
• [Feature][Format] Improve maxwell_json,canal_json,debezium_json format support merge update_before and update_after (#9805)
• [Feature][File] Add markdown parser documentation (#9834)
• [Feature][File] Add markdown parser for RAG support #9714 (#9760)
• [Feature][Connectors-v2] Clean up temporary files for paimon sink (#9819)
• [Feature][Connector-V2] Support CDC mode for databend sink connector (#9661)
• [Feature][Engine]Metalake support for data source information storage and management (#9688)
• [Feature][Connector-V2] [IoTDB] Added source and sink connectors to support IoTDB 2.X (#9872)
• [Feature][Connector-File-Hadoop] Support sync_mode=update for HdfsFile source (binary) (#10268)
• [Feature][Connector-V2][Hive] Support multiple Hive metastore URIs for automatic failover (#10253)
• [Feature][Core][Metrics] Add sink committed metrics and calculate commit rate in job metrics summary (#10233)

2.3.12 Release Note

Docs

• [Doc] Added deepwiki to README.md #9354 (#9356)
• [Doc] Fix link error and DynamoDB parameter error (#9447)
• [Doc][Improve] update projection supported in Paimon (#9425)
• [Doc][Zeta] Add zeta tuning guide (#9539)
• [doc]The link in the document is 404 (#9561)
• [Docs] update jdbc related document (#9679)
• [Docs][Connector-Iceberg] Update Iceberg doc with support S3 Tables rest catalogs (#9686)
• [Fix][Doc] Fix rest api finished-jobs doc miss SAVEPOINT_DONE (#9676)
• [Fix][Doc] Fix transform-v2 TableFilter title (#9528)
• [Fix][Doc] Sink's plugin_input is incorrect configurations (#9492)
• [Fix][Doc] Update StarRocks doc change schema necessity to true (#9656)
• [Improve][Doc] Add GenericDialect support in jdbc doc (#9763)
• [Improve][Doc] Add multimodal support description to docs (#9652)
• [Improve][Doc] Add SeaTunnel tools into Readme (#9707)
• [Improve][Doc] Fix formatting of parameter types (#9753)
• [Improve][Doc] Update sql function result type (apache#9703) (#9711)

Bug Fix

• [Bug] [Connector-V2] Fix the issue of writing the ORC format Iceberg report "Illegal provider-class name" (#6754) (#9588)
• [Bug] [Connector-V2] Updates Iceberg version to 1.6.1 (#9387) (#9451)
• [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null (#9498)
• [Bug][connector-http] Fix paging request running infinitely (#9504)
• [Bug][Connector-V2] fix NPE when decimal type precision is incompatible for Paimon (#9452)
• [Bug][Connector-V2] NoSuchMethodError caused by Netty version conflict on Spark 3.3.0 (#9632)
• [bug][dist]fix the release packaging exception (#9436)
• [bugfix][Connector-V2] Fixed the load driver inaccurate situation (#9468)
• [Fix] [connector-jdbc] prevent precision loss in Float to BigDecimal conversion (#9670)
• [Fix] [Transform-V2] Fix embedding output columns vector dimension (#9646)
• [Fix][CI] Update iris docker version to fix ci (#9360)
• [Fix][Connecotr-kafka] Fix kafka IllegalArgumentException when offset is -1 (#9376)
• [Fix][Connector-clickhouse] Fix SeaTunnelRow tableId set error (#9585)
• [Fix][Connector-File] Fix parquet support user config schema (#9596)
• [Fix][Connector-HTTP] Add default content-type when user not set (#9497)
• [Fix][Connector-Http] fix Invalid mime type (#9363)
• [Fix][connector-http] fix parsing httpjson, the number of two fields is inconsistent with the import failure (#9103)
• [Fix][Connector-Iceberg] Fix Time Zone Issue for Iceberg Timestamp Type (#9460)
• [Fix][Connector-Jdbc] Supports reading and writing Postgres network dress types (#9618)
• [Fix][Connector-Jdbc]Fixed Vertica data source cannot upsert data. (#9607)
• [Fix][connector-rabbitmq] Set default value for durable, exclusive and auto-delete (#9631)
• [Fix][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer StreamMode (#9598)
• [Fix][Connector-V2] ArrowToSeatunnelRowReader convertSeatunnelRowValue add handle Second TIMESTAMP type (#9393)
• [Fix][Connector-V2] Correct typo in batch-size-per-scan option key (#9434)
• [Fix][Connector-V2] Fix misleading parameter name in DorisStreamLoad (#9685)
• [Fix][Connector-V2] Fix OceanBase Oracle create unsupported data type (#9383)
• [Fix][Connector-V2] Fix prometheus check time can not parse double value (#9311)
• [Fix][Connector-V2] Fix starrocks decimal column definition generation(#9470) (#9471)
• [Fix][Connector-V2] Optimize Paimon DECIMAL type check to prevent precision loss (#9480)
• [Fix][Connector-V2] Optimize start mode of kafka recovery job (#9736)
• [Fix][Connector-V2] Oracle cdc not update transaction commit when LOB enabled (#9412)
• [Fix][Connector-V2] set default tableid in SeaTunnelRow for Debezium JSON deserialization (#9431)
• [Fix][Connector-V2] Update catalog table schema of debezium json (#9525)
• [Fix][Connector-V2] Update file filter pattern compilation to remove unnecessary quoting (#9658)
• [F

2.3.11 Release Note

Docs

• [Doc][Improve] support chinese [docs/zh/connector-v2/source/MySQL-CDC.md] (#8884)
• Add MongoDB-CDC connector Chinese documentation (#9337)
• [Doc][zh-CN] Add HiveJdbc connector Chinese documentation (#8497) (#9329)
• [Doc][Connector-V2] Update save mode config for OssFileSink (#9303)
• [Doc][zh-CN] Add Jira connector Chinese documentation (#9236)
• [Doc][Improve] support chinese [docs/zh/connector-v2/source/Cloudberry.md] (#9211)
• [Fix][Doc] Delete space before _test and _sink (#9216)
• [Doc][Improve] support chinese [docs/zh/connector-v2/source/Github.md] (#9210)
• [Feature][doc] Repair md document format (#9177)
• [Fix][Doc] Adjust the description of the explode function (#9168)
• [Improve][Doc] Unify the header format and fix some documents with abnormal formats (#9159)
• [Improve][Doc] Adjust description of trim series sql function (#9143)
• [Improve][Doc] Update kerberos part of kafka doc (#9031)
• [Doc] Update changelog (#9071)
• [Doc] Fix classloader-cache-mode default value
• [Doc][Fix] DB2.md fix markdown formatting in Data Type Mapping (#9062)
• [Doc] fix deadlinks (#9053)
• [Improve][Doc] Improve changelog generate tool to ignore useless commit (#9035)
• [Fix][Doc] fix deadlinks (#9016)

Bug Fix

• [Bug][Zeta] SeaTunnelClient can not exit with error (#9281)
• [Improve][Oracle-CDC] Fix oracle rename ddl event missing column type (#9314)
• [Fix][Connector-jdbc] Fix postgresql sink trying to update unique key (#9293) (#9298)
• [Fix][Zeta] Fix losing checkpoint scheduling in extreme cases (#9246)
• [Fix][Connector-V2] Fix hive client thread unsafe (#9282)
• [Fix][Connector-V2] Fix oceanbase mysql jdbc sink create statement error (#9267)
• [Fix][CI] Fix update build status action error (#9300)
• [Fix][CI] Disable dead link check (#9296)
• [Fix][CI] Set continue-on-error to true for dead link job (#9290)
• [Fix][Connector-V2] Fix the problem that missing options configuration when building ClickHouse Nodes (#9277)
• [Fix][CI] Remove dead link check from sanity check (#9274)
• [fix][connector-mango] fix split with avgSize zero error (#9255)
• [Bugfix][Starrocks] Fix starrocks batch data exceeds the maximum limit (#9256)
• [Fix][Config] Fix invalid telemetry-log config in seatunnel.yaml (#9230)
• [Improve][connector-iceberg] fix schema change event (#9217)
• [Fix][Zeta] Fix premature closure of the checkpoint thread pool causes abnormal task status (#9228)
• [Bugfix][CI] Fix transform check (#9237)
• [Bugifx][kafka] Fix kafka enumerator assign split NPE (#9220)
• [Fix][CI] Release file resources in testcase (#9232)
• [Fix][Core]Fix config parse with "" error (#9225)
• [Fix][Connector-V2] Fix kafka database name (#9201)
• [Fix][Serialize] prevent scientific notation for decimal numbers in JSON output​ (#9186)
• [Fix][Zeta] Fix double-checked locking without volatile in HdfsStorage (#9200)
• [Fix][connector-file-base] fix parquet int32 convert error (#9142)
• [Fix][CI] Upgrade action job runner host version (#9185)
• [Fix][API] Fixed not invoke the SinkAggregatedCommitter's init method (#9070)
• [Fix][Connector-V2] Fix NullPointerException when column or tag contains null value in TDengine sink (#9158)
• [Bugfix][Csv] Fix csv format delimiter (#9066)
• [Fix][Zeta] Adjust the timing of invoking the enumerator open method (#9092)
• [Fix][Connector-V2] Fix postgres cdc with debezium_json format can not parse number without scale (#9052)
• [Fix][Connector-V2] Fix maxcompute write with multi parallelism (#9089)
• [Fix][MongoDB] The Long type cannot handle string values in scientific notation (#8783)
• [Bugfix][Elasticsearch] Fix add column event (#9069)
• [Fix][Connector-V2] Fix SqlServer create table when database with dot (#9007)
• [Fix] Fix DateUtils can not parse datetime string with local timezone (#9019)
• [Fix][Connector-V2] Fixed missing timestamp accuracy of starrocks connector (#9096)
• [Fix][Connector-V2][OceanBase]

2.3.10 Release Note

Feature

• [improve] update localfile connector config (#8765)
• [Improve][Zeta] Check closing log file handle every 1 minute (#9003)
• Revert " [improve] update localfile connector config" (#9018)
• [improve] update Web3j connector config option (#9005)
• [Improve] influxdb options (#8966)
• [improve] http connector options (#8969)
• [improve] iotdb options (#8965)
• [improve] email connector options (#8983)
• [Feature][Jdbc] Support sink ddl for sqlserver #8114 (#8936)
• [Improve] iceberg options (#8967)
• [Improve] Update label-scope-conf.yml for core label (#8979)
• [Improve][Core] Support parse quote as key (#8975)
• [Improve][deploy] helm chart config improve (#8954)
• [Feature][Connector-V2] Support between predicate pushdown in paimon (#8962)
• [Improve][Connector-V2] Ensure that the FTP connector behaves reliably during directory operation (#8959)
• [Improve] easysearch options (#8951)
• [Feature][elasticsearch-connector] support elasticsearch sql source (#8895)
• [Improve][Connector-V2] RocketMQ Source add message tag config (#8825)
• [improve] hudi options (#8952)
• [Improve][Shade] Unit the module name of Shade. (#8960)
• [Feature][Connector-V2] Suppor Time type in paimon connector (#8880)
• [improve] fake source options (#8950)
• [Improve] hbase options (#8923)
• [Improve][CDC] Filter ddl for snapshot phase (#8911)
• [Improve][Oracle-CDC] Support ReadOnlyLogWriterFlushStrategy (#8912)
• [improve] google sheets options (#8922)
• [Improve][Connector-V2] Random pick the starrocks fe address which can be connected (#8898)
• [Improve][E2E][Connector V2][CDC Oracle] OracleContainer support ARM architecture images (#8928)
• [Improve] filestore options (#8921)
• [improve] improve TaskLocation/TaskLocationGroup info (#8862)
• [Improve][CDC] Extract duplicate code (#8906)
• [Feature][Paimon] Customize the hadoop user (#8888)
• [Improve][connector-file-base] Improved multiple table file source allocation algorithm for subtasks (#8878)
• [Improve][connector-hive] Improved hive file allocation algorithm for subtasks (#8876)
• [Improve] sink mongodb schema is not required (#8887)
• [Feature][Connector-v2] Support multi starrocks source (#8789)
• [Feature][Kafka] Support native format read/write kafka record (#8724)
• [Improve][Redis] Optimized Redis connection params (#8841)
• [Improve][CDC] Filter heartbeat event (#8569)
• [Improve][Jdbc] Support upsert for opengauss (#8627)
• [Improve][Zeta] Split classloader in job master (#8622)
• [Feature][Connector-V2] Add parameter for read/write file (#8769)
• [Improve] doris options (#8745)
• [Feature][Transform-v2] Add support for Zhipu AI in Embedding and LLM module (#8790)
• [Improve][Jdbc] Remove useless utils. (#8793)
• [improve] update clickhouse connector config option (#8755)
• [Feature][Config] Support custom config keys for encrypt/decrypt (#8739)
• [Improve] rabbit mq options (#8740)
• [Improve] re-struct Zeta Engine config options (#8741)
• [Improve][Zeta] Disable restful api v1 by default (#8766)
• [improve] update kafka source default schema from content<ROW> to content (#8642)
• [Feature][Transforms-V2] Handling LLM non-standard format responses (#8551)
• [improve] console sink options (#8743)
• [Improve][Connector-V2] Improve orc read error message (#8751)
• [Improve][connector][activemq] Remove duplicate dependencies (#8753)
• [Improve][Jdbc] Improve catalog connection cache (#8626)
• [improve] datahub sink options (#8744)
• [improve] dingtalk sink options (#8742)
• [improve] Slack connector options (#8738)
• [Improve][Connector-V2] Add optional flag for rocketmq connector to skip parse errors instead of failing (#8737)
• [Improve][Connector-v2][Paimon]PaimonCatalog close error message update (#8640)
• [Feature][Jdbc] Support sink ddl for dameng (#8380)
• [Improve] restruct connector common options (#8634)
• [improve] add StarRocks options (#8639)