Architecture Overview
SoftClient4ES is a SQL gateway that translates standard SQL into Elasticsearch operations. It provides a unified GatewayApi that accepts any SQL statement and routes it to the appropriate executor.
High-Level Architecture
Statement Routing
The GatewayApi exposes a single entry point:
def run(sql: String): Future[ElasticResult[QueryResult]]SQL statements are automatically classified and routed:
| SQL Type | Executor | Examples |
|---|---|---|
| DQL | DqlExecutor | SELECT |
| DML | DmlExecutor | INSERT, UPDATE, DELETE, COPY INTO |
| DDL | DdlRouterExecutor | CREATE/ALTER/DROP TABLE, PIPELINE, WATCHER |
The API automatically normalizes SQL (removes comments, trims whitespace), splits multiple statements separated by ;, parses SQL into AST nodes, and dispatches to the correct executor.
Result Types
DQL Results
| Type | Description |
|---|---|
QueryRows | Materialized rows (maps) |
QueryStream | Streaming rows via scroll |
QueryStructured | Raw Elasticsearch response |
DML Results
DmlResult(inserted = N, updated = N, deleted = N, rejected = N)DDL Results
| Type | Returned by |
|---|---|
DdlResult(success) | CREATE, ALTER, DROP, TRUNCATE |
TableResult(table) | SHOW TABLE |
PipelineResult(pipeline) | SHOW PIPELINE |
SQLResult(sql) | SHOW CREATE TABLE/PIPELINE |
QueryRows | DESCRIBE TABLE/PIPELINE |
Module Structure
The project is organized into a multi-module ecosystem:
| Module | Purpose |
|---|---|
| Core (Apache 2.0) | SQL parser, AST, query translation, GatewayApi, REPL |
| Extensions (Elastic v2) | Materialized Views, SPI extensions |
| JDBC (Elastic v2) | JDBC Type 4 driver |
| Arrow Core (Elastic v2) | Arrow vector conversion, shared abstractions |
| ADBC Driver (Elastic v2) | In-process columnar access via ADBC API |
| Arrow Flight SQL (Elastic v2) | gRPC-based columnar server |
Client Access Methods
| Method | Process Model | Data Format | Protocol | Use Case |
|---|---|---|---|---|
| REPL | Standalone | Text (ASCII/JSON/CSV) | Direct | Ad-hoc queries |
| JDBC | In-process | Row-based (ResultSet) | JDBC API | Java apps, BI tools |
| ADBC | In-process | Columnar (Arrow) | ADBC API | Analytics, data engineering |
| Flight SQL | Separate server | Columnar (Arrow) | gRPC (HTTP/2) | Multi-client, networked |
Version Isolation
Each Elasticsearch major version has its own subproject tree (es6/, es7/, es8/, es9/). Version-agnostic code in core/ and sql/ never imports from version-specific modules. A bridge template pattern ensures code sharing across versions while maintaining isolation.
SPI Pattern
Client implementations are discovered via Java’s ServiceLoader mechanism (ElasticClientFactory), allowing clean separation between the SQL engine and Elasticsearch client libraries.