Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
tencent-cloud-clsplugin always uploads logs as an uncompressed protobufpayload, although the CLS structured log API accepts compressed uploads through
the
x-cls-compress-typerequest header (see上传结构化日志).
Compressing the payload reduces the log write traffic remarkably (about 21x on
realistic access logs), which is the main cost driver for high volume
deployments.
This PR adds optional zstd compression for the upload, mirroring the behaviour
of the official Go SDK (
zstdCompress+x-cls-compress-type: zstd).Changes:
apisix/utils/zstd.lua: a thin LuaJIT FFI binding over libzstd(
ZSTD_compressBound,ZSTD_compress,ZSTD_isError,ZSTD_getErrorName), with the default compression level 3(
ZSTD_CLEVEL_DEFAULT, the same aszstd.SpeedDefaultused by the Go SDK).libzstd is loaded lazily and the result - including the failure - is cached,
trying
libzstd.so.1,zstdandlibzstdin order so that environmentsshipping only the versioned soname work too.
available()reports whetherzstd is usable,
compress()returnsnil, errinstead of throwing.apisix/plugins/tencent-cloud-cls/cls-sdk.lua:_M.new()accepts an extraoptional
compress_typeargument (unknown values are rejected);send_cls_request()compresses the encoded protobuf whenzstdisconfigured and adds the
x-cls-compress-type: zstdheader.Content-Type,the signature and the request URL are unchanged. When compression fails, e.g.
because libzstd is missing at runtime, the log is uploaded uncompressed and
an error is logged - a missing native library never causes log loss.
apisix/plugins/tencent-cloud-cls.lua: newcompress_typeattribute(
"none"|"zstd", default"none"), passed through to the SDK.compress_typeand the libzstd requirement are documented in bothdocs/en/latest/plugins/tencent-cloud-cls.mdanddocs/zh/latest/plugins/tencent-cloud-cls.md.utils/install-dependencies.sh: installslibzstd-dev(apt) andlibzstd-devel(yum) so that dev/CI environments have libzstd available.Tests:
t/utils/zstd.t:available(), argument validation, the zstd frame magicnumber
28 b5 2f fd, the compressed size and compression levels 1/3/9 (thecompression cases are skipped when libzstd is not installed).
t/plugin/tencent-cloud-cls.tTEST 23-26: the schema rejects unsupportedcompress types, a route can be created with
compress_type: zstd, theuploaded body is a zstd frame and carries
x-cls-compress-type: zstd, andcompression failure falls back to an uncompressed upload that is still
processed successfully.
Notes for reviewers: this introduces a runtime dependency on libzstd (only when
compress_type: zstdis used). Images that do not shiplibzstd1should addit; without it the plugin keeps working and just uploads uncompressed payloads.
Which issue(s) this PR fixes:
Fixes #
Checklist