clp

A release that introduces a new coordinator for compression, support for running CLP on ARM machines, and several bug fixes and improvements.

compression-coordinator is a new component that leverages Spider to manage compression jobs. Together, they provide several benefits including improved failure handling, fairness, and horizontal scalability (see the docs for more details).

[!NOTE]

• compression-coordinator is only usable with clp-json.
• compression-coordinator has not yet reached feature-parity with compression-scheduler (see the docs for details). Until then, the two are meant to run side-by-side, with compression-scheduler handling any jobs that compression-coordinator doesn't yet support.

[!IMPORTANT]

• This release includes some changes that are incompatible with previous releases. If this affects you, reach out and we may be able to help with version migration. These changes are marked with "Breaking".
• Docker Compose 5.3.0 and 5.3.1 aren't supported due to an upstream regression in Docker Compose affecting disabled services.

This release includes two tars for CLP as a distributed system:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

This release also includes installable packages for CLP’s core.

Docs

The docs for this release are available here.

New features

• clp-json: Add the compression-coordinator component to manage compression jobs with the new Spider Huntsman (written in Rust); Purge the integration of the legacy Spider Wolf (written in C++). (#2401, #2402, #2403, #2404, #2405, #2406, #2408, #2411, #2415, #2416, #2417, #2418, #2420, #2421, #2434, #2435, #2436, #2438, #2419, #2452, #2455, #2456, #2457, #2460, #2462, #2479, #2481, #2492, #2493, #2494, #2495, #2501, #2507, #2535)
  • Breaking:
    • We now require users to set nameOverride and fullnameOverride globally so that they also apply to the Spider subchart. E.g., a user now needs to run helm install --set global.nameOverride rather than helm install --set nameOverride (likewise for fullnameOverride).
    • The C++-based Spider integration has been removed, so users can no longer use compression-scheduler with Spider.
    • The following settings have been removed from CLP's configuration file (etc/clp-config.yaml):
      • database.names.spider
      • compression_scheduler.type
      • spider_scheduler
    • The following settings have been removed from CLP's credentials file (etc/clp-credentials.yaml):
      • spider_password
      • spider_username
• clp-json/clp-text: Add support for running CLP on ARM64 (aarch64) machines. (#2357, #2426, #2515, #2518, #2527)
• clp-json: Add support for parsing RFC 2822/822 timestamp strings. (#2363)
• webui: Route queries from the WebUI through the API server, eliminating duplicated job submission and results retrieval work that they both used to do. (#2483, #2484)
  • Breaking:
    • The API server is now required for the WebUI to run.
    • The WebUI's /api/search/* routes have been removed. Users should now use the API server directly.
    • For CLP’s native query engine, support for explicitly clearing a query's search results has been removed. Users can rely on CLP's [retention periods](https://docs.yscope.com/clp/v0.14.0/user-docs/guides-retention

A release that adds OpenTelemetry metrics, compression and search performance improvements, compression and search throughput and responsiveness improvements, additional configuration knobs, and even more features and improvements.

[!NOTE] This release includes some changes that are incompatible with previous releases. If this affects you, reach out and we may be able to help with version migration. These changes are marked with "Breaking".

This release includes two tars for CLP as a distributed system:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

This release also includes installable packages for CLP’s core.

[!WARNING] Docker Compose 5.3.0 and 5.3.1 aren't supported due to an upstream regression in Docker Compose affecting disabled services.

Docs

The docs for this release are available here.

New features

• clp-json/clp-text: Add configurable collection of anonymized operational metrics and support for exporting metrics via OpenTelemetry. (#2267, #2278, #2286, #2288, #2309, #2335, #2351)
  • Breaking:
    • Anonymized telemetry is enabled by default.
      • On first start, when run interactively, users will be prompted for consent; non-interactive starts use the enabled default.
      • To disable telemetry, see the docs.
    • Instance IDs for new package instances now use a full UUIDv4 in var/log/instance-id instead of a four-character hexadecimal ID.
      • Existing instance IDs are retained, but tooling that assumes a four-character ID must be updated to accept UUIDv4 values.
• clp-json: Add support for ingesting general-purpose compressed files. (#2295)
• webui: Make the search result limit configurable, and warn when results from Presto are truncated. (#2289, #2350)
• clp-json: Add support for exporting Presto search results from the UI (resolves #2320). (#2331)
• clp-s: Add support for outputting count and count-by-time aggregation results to the command line and results-cache. (#2326, #2342)

Bug fixes & improvements

clp-json/clp-text

• clp-json: Improve query performance through column-based filtering for a subset of query shapes. (#2281, #2315)
• clp-json: Improve decompression performance by leveraging simdjson for JSON string-escaping where possible. (#2216)
• clp-json: Improve compression performance for unstructured logs by re-using the same log_surgeon::BufferParser to convert each file. (#2284)
• clp-json: Add support for running the Presto and native query engines simultaneously in the same Docker Compose deployment. (#2349)
  • Breaking: The package.query_engine configuration key has been replaced with the webui.query_engine key.
• clp-json/clp-text; Improve the throughput, responsiveness, and performance of the query-scheduler via caching and eliminating concurrency bottlenecks. (#2140, #2215, #2336)
• clp-json/clp-text: Add defensive timeouts to prevent scheduler hangs (fixes #2252). (#2271)
  • Breaking:
    • Compression and search tasks will now be terminated after the configured timeout.
      • In Docker Compose deployments, the default compression and query tasks default to a soft and hard limits of 600s and 1,200s respectively, and can be configured via task_soft_time_limit and task_time_limit.
      • In Helm deployments, both limits are 0 (disabled).
• clp-json: Improve timestamp-filter query translation in the UI to give Presto more query optimization opportunities. (#2328)
• log-in

A release that includes several bug fixes and improvements like improved query throughput.

[!NOTE] This release includes some changes that are incompatible with previous releases. If this affects you, reach out and we may be able to help with version migration. These changes are marked with "Breaking".

This release includes two tars for CLP as a distributed system:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

This release also includes installable packages for CLP’s core.

Docs

The docs for this release are available here.

New features

• api-server: Add an endpoint for retrieving usage statistics for compression jobs. (#2197)
• webui: Add export and copy buttons for search results. (#2185)

Bug fixes & improvements

• clp-json: Update timezone regex to only accept spaces when followed by an actual timezone (fixes #2187). (#2192)
• clp-json/clp-text: Dispatch batches of search tasks in parallel to improve query scheduler throughput. (#2184)
  • Breaking: By default, the query scheduler will now start a process pool with 4 processes, which can increase resource usage. To revert to the previous behavior, reduce query_scheduler.scheduler_concurrency to 1 in CLP’s config file.
• clp-json: Add support for compressing unstructured log events as large as 512MiB (fixes #2176). (#2193)
• api-server: Enforce max_num_results across all result output paths (fixes #1858). (#2180)
• clp-json/clp-text: Clean up temporary files on compression failure. (#2210)
• clp-json: Add retries with exponential backoff for transient S3 read failures before any data is read. (#2203)
• clp-json/clp-text: Increase database column sizes for compression & query job configs; Handle corrupted job configs gracefully (fixes #2151). (#2178)
  • Breaking: The maximum column size for some columns in the compression jobs, compression tasks, and query jobs metadata tables has been changed.

View the full changelog for more details.

Thanks to @gibber9809, @goynam, @jackluo923, @jonathan-imanu, and @junhaoliao for their contributions.

A release that includes several convenience features, bug fixes, and improvements like additional support for AWS authentication methods and minor improvements to query performance.

[!NOTE] This release includes some changes that are incompatible with previous releases. If this affects you, reach out and we may be able to help with version migration. These changes are marked with "Breaking".

This release includes two tars for CLP as a distributed system:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

This release also includes installable packages for CLP’s core.

Docs

The docs for this release are available here.

New features

• log-ingestor/helm: Add support for automatically retrieving AWS credentials via the default credentials provider chain. (#1962)
  • Breaking:
    • In CLP’s configuration file, aws_authentication.type: "ec2" has been renamed to "default" , broadening its scope from EC2 instance metadata to the full AWS SDK default credential provider chain.
    • The Helm chart renames the <clp-deployment-name>-job-watcher to <clp-deployment-name>-service-account.
    • The Helm chart now uses the ServiceAccount (<clp-deployment-name>-service-account) and exposes OIDC annotations in values.yaml for S3 access.
• log-ingestor: Switch buffer submission from soft timeout to hard timeout; Move buffer config from package config to job config. (#2150)
  • Breaking:
    • buffer_flush_timeout, buffer_flush_threshold, and channel_capacity have been removed from the package-level config (and Helm values). They are now per-job settings specified via the ingestion API's buffer_config field.
    • The buffer flush timer now uses a hard timeout (starts when the first entry is buffered and is not reset by subsequent entries), replacing the previous soft timeout (which reset on every new entry).
• helm: Add support for deploying Presto in CLP's Helm chart. (#2004)
  • Breaking:
    • The query_engine config has moved from clpConfig.package.query_engine to clpConfig.webui.query_engine.
    • The query-scheduler, query-worker, and reducer components are now conditionally deployed; they are skipped when their respective config sections are set to null.
• packaging: Add support for building deb, rpm, and apk packages containing CLP's core binaries. (#1996)

Bug fixes & improvements

• clp-json: Accept empty files as valid inputs (fixes #1993, #2063). (#2138)
• clp-json: Make timezone offset optional in timezone regex, allowing bare "UTC" and trailing "Z" formats (fixes #1952). (#2134)
• presto-clp: Fix AWS S3 endpoint URL format so that Presto queries for archives on S3 no longer return zero rows (fixes #2107, fixes #2108). (#2109)
• clp-json/clp-text: Improve performance for S3 ingestion jobs submitted through log-ingestor by eliminating redundant queries for S3 object metadata. (#2082, #2087)
• clp-json: Improve query performance when using the API server by not uploading empty query result files to S3. (#2135)
• deployment: Improve robustness of webui health checks by switching from TCP to HTTP to detect failures like trashing (fixes #2172). (#2186)
• helm!: Rename mcp-server port from "mcp" to "mcp-server" for consistency (fixes #2021). (#2022)
  • Breaking: Any name-based references (e.g., kubectl port-forward, Ingress or Gateway) to mcp will need to be updated to mcp-server. Usage of the port number is unaffected.
• clp-package: Only create a new dataset after paths to compress are validated in a compression job (fixes #2091). (#2092)
• deployment: Pin Presto container image