diff --git a/.github/workflows/cla.yml b/.github/workflows/cla.yml
index dfa7c8d0ea..4e7bcf38b9 100644
--- a/.github/workflows/cla.yml
+++ b/.github/workflows/cla.yml
@@ -12,7 +12,7 @@ jobs:
- name: "CLA Assistant"
if: (github.event.comment.body == 'recheck' || github.event.comment.body == 'I have read the CLA Document and I hereby sign the CLA') || github.event_name == 'pull_request_target'
# Beta Release
- uses: cla-assistant/github-action@v2.1.3-beta
+ uses: cla-assistant/github-action@v2.4.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# the below token should have repo scope and must be manually added by you in the repository's secret
diff --git a/.github/workflows/gh-pages.yml b/.github/workflows/gh-pages.yml
index 10d0cdb437..a246352abf 100644
--- a/.github/workflows/gh-pages.yml
+++ b/.github/workflows/gh-pages.yml
@@ -8,23 +8,23 @@ on:
jobs:
deploy:
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-22.04
steps:
- name: Git checkout
- uses: actions/checkout@v2
+ uses: actions/checkout@v4
with:
submodules: true # Fetch Hugo themes (true OR recursive)
fetch-depth: 0 # Fetch all history for .GitInfo and .Lastmod
ref: main
- name: Setup Hugo
- uses: peaceiris/actions-hugo@v2
+ uses: peaceiris/actions-hugo@v3
with:
- hugo-version: 'latest'
+ hugo-version: '0.128.2'
extended: true
- name: Cache Hugo modules
- uses: actions/cache@v2
+ uses: actions/cache@v4
with:
path: /tmp/hugo_cache
key: ${{ runner.os }}-hugomod-${{ hashFiles('**/go.sum') }}
@@ -32,12 +32,12 @@ jobs:
${{ runner.os }}-hugomod-
- name: Setup Node
- uses: actions/setup-node@v3
+ uses: actions/setup-node@v4
with:
- node-version: '14'
+ node-version: '20'
- name: Cache dependencies
- uses: actions/cache@v2
+ uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
@@ -51,7 +51,7 @@ jobs:
# run: hugo --gc
- name: Deploy
- uses: peaceiris/actions-gh-pages@v3
+ uses: peaceiris/actions-gh-pages@v4
if: github.ref == 'refs/heads/main'
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.gitignore b/.gitignore
index 26d948602c..ae663ecb0b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,13 @@
+docs/
*.DS_Store*
node_modules/
.vscode/*
yarn.lock
.hugo_build.lock
+/.idea/.gitignore
+/.idea/altinityknowledgebase.iml
+/resources/_gen/assets/scss/scss/main.scss_3f90599f3717b4a4920df16fdcadce3d.content
+/resources/_gen/assets/scss/scss/main.scss_3f90599f3717b4a4920df16fdcadce3d.json
+/.idea/modules.xml
+/.idea/vcs.xml
+/.idea/inspectionProfiles/Project_Default.xml
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000000..67e8c09fe0
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,236 @@
+# AGENTS.md
+
+This repository contains overall-facing ClickHouse knowledge base articles. Optimize for correctness, safety, and reviewability. Do not optimize for large rewrites, cosmetic churn, or generic prose improvement.
+
+## Mission
+
+When modifying an article, do all of the following:
+
+- preserve useful existing content;
+- fix concrete factual, safety, or version-compatibility problems;
+- add missing information only when it materially helps operators;
+- keep the diff narrow enough that a maintainer can review it quickly.
+
+Do **not** turn a focused article review into a full rewrite.
+
+## Default workflow
+
+1. Read the entire article before proposing any change.
+2. Read related issues, PRs, and review comments for the same article.
+3. Check upstream ClickHouse documentation and changelog for the version ranges that matter.
+4. Search upstream GitHub issues/PRs only for rare or specific edge cases that deserve extra explanation.
+5. Produce findings first, then a minimal patch.
+6. Do not open a public issue or PR before a human has read the proposal, unless explicitly instructed.
+
+If the article is mostly correct, prefer a precise issue note or a very small patch over a broad rewrite.
+
+## Non-negotiable rules
+
+- Prefer point fixes over rewrites.
+- Keep existing structure unless the structure itself is broken.
+- Do not remove useful operational detail unless it is wrong, obsolete, dangerous, or duplicate.
+- Do not replace product documentation. Extend it only where the KB adds operational value.
+- Every added paragraph, table, query, or command must answer a concrete operator question.
+- Every non-obvious factual claim must have a source.
+- Every behavior claim must have explicit version scope.
+- Every recipe must be copy-pasteable and tested, or it should stay out of the article.
+- Every destructive action must be preceded by an explicit warning.
+- No public-facing PR should be generated automatically from an unreviewed draft.
+
+## What a good KB edit looks like
+
+A good edit usually does one or more of the following:
+
+- corrects a factual statement;
+- adds exact version boundaries;
+- adds a tested, high-value recipe;
+- adds a warning before a dangerous action;
+- clarifies an edge case seen in upstream issues;
+- shortens or improves references without changing meaning.
+
+A bad edit usually does one or more of the following:
+
+- rewrites most of the article without necessity;
+- replaces specific operational detail with generic prose;
+- adds queries that are not tested;
+- adds queries that only work on newer versions without saying so;
+- introduces claims based on behavior that was later reverted or rolled back;
+- adds filler such as obvious prerequisites with no concrete payoff;
+- adds monitoring queries that do not provide materially new information;
+- adds recovery instructions without clearly stating corruption or duplication risks.
+
+## Source hierarchy
+
+Use sources in this order:
+
+1. Upstream ClickHouse documentation.
+2. Upstream ClickHouse changelog / release notes.
+3. Upstream code or comments when documentation is insufficient.
+4. Upstream GitHub issues and PRs for specific rare cases.
+5. Existing KB issues/PRs for repository context.
+
+Rules for using sources:
+
+- Use GitHub issues/PRs to explain rare cases, not to redefine normal behavior.
+- Treat issue comments as situational evidence unless corroborated.
+- If a behavior changed and later changed back, document the exact historical window instead of presenting it as current behavior.
+- If you cannot verify a claim, do not promote it into the article.
+
+## Versioning rules
+
+Every behavior claim must answer all three questions:
+
+1. Which versions does this apply to?
+2. Is this current behavior or historical behavior?
+3. Was the behavior later reverted, replaced, or removed?
+
+Version guidance:
+
+- State explicit version ranges whenever compatibility matters.
+- If a query relies on columns or functions added later, annotate the minimum supported version or provide a fallback query.
+- If compatibility is messy and not worth documenting inline, keep the article on the main supported behavior and move the edge case to a note.
+- Never silently mix examples from incompatible versions.
+
+## Query and command rules
+
+Only add a query or command if it satisfies all of the following:
+
+- it solves a concrete operator task;
+- it has a short explanation of what it detects or helps decide;
+- it was tested on a representative ClickHouse version, or compatibility is explicitly documented;
+- it does not rely on unsupported assumptions about schema, functions, or output columns.
+
+For every recipe, include enough context to make it usable:
+
+- purpose;
+- version scope if needed;
+- how to interpret the output;
+- safety note if destructive or risky;
+- fallback or limitation if known.
+
+### Hard constraints for SQL examples
+
+- Do not add a query that uses version-specific columns without noting that fact.
+- Do not add a query that uses helper functions unavailable on common target versions.
+- Do not add a helper query without explaining what it detects and why a reader would need it.
+- Do not add a second query that merely repeats information already shown by a prior query unless it is explicitly framed as monitoring, alerting, or dashboard input.
+- Prefer generating statements for review first; execution should be a separate, deliberate step.
+
+### Compatibility policy
+
+If a query is useful but not portable:
+
+- add a minimum-version note; or
+- provide an alternate query for older versions; or
+- omit it from the article and keep it in review notes instead.
+
+Untested queries do not belong in the KB.
+
+## Safety rules for destructive operations
+
+Destructive operations require extra care. This includes SQL commands such as `DROP DETACHED`, filesystem deletion, and recovery/attach flows that can duplicate or corrupt data when misused.
+
+Rules:
+
+- Put a warning block immediately before destructive or potentially destructive commands.
+- State the concrete risk: data loss, duplication, replica divergence, or unsafe cleanup.
+- Tell the reader to review generated commands before executing them.
+- Prefer documented SQL workflows over ad-hoc filesystem deletion when the product supports them.
+- If filesystem operations are still needed, present them as last-resort steps and explain why.
+- Do not describe a risky recovery path as "safe" or "recovered" unless the preconditions are explicit.
+
+Use this Hugo warning pattern:
+
+```md
+{{% alert title="Warning" color="warning" %}}
+Review generated commands carefully before executing them. Destructive actions can cause data loss, duplication, or replica inconsistency if used incorrectly. Ensure you have a valid backup and understand why each target object is safe to remove or attach.
+{{% /alert %}}
+```
+
+## Article-writing rules
+
+- Keep the original article voice and purpose.
+- Prefer dense, operational wording over "better sounding" prose.
+- Remove empty words.
+- Do not add text that only restates the obvious.
+- If you add a prerequisite, explain the exact failure mode it prevents.
+- Use tables only when they compress real information.
+- Keep appendix/reference links readable; avoid dumping long raw URLs into the body when a shorter reference style works.
+
+## Specific guidlines from other reviews (human add here)
+
+These are repository-specific lessons and should guide similar edits:
+
+- Do not turn a useful but imperfect article into a prettier, less useful rewrite.
+- Do not add "problems" just because the prompt asked to find them; only report real defects.
+- Do not add historical behavior as present-day guidance without exact version boundaries.
+- Do not add queries that depend on later-added fields or functions unless the article is version-scoped.
+- Do not add a query unless you can explain exactly what it detects.
+- Do not add asynchronous-metric queries as if they add new investigative detail when they only repeat inventory information; they are valid mainly for dashboards and alerting.
+- Do not include obvious prerequisites unless the article explains their concrete operational impact.
+- Do not present `ATTACH` / recovery recipes in a way that could lead to duplicate data without a strong warning and explicit preconditions.
+- Do not merge or publish a draft that the human owner has not read.
+- Never remove ® from ClickHouse® unless it is technically impossible in the target format or an established repository convention explicitly omits it.
+
+## Output format for article reviews
+
+When asked to review or modify an article, return work in this order:
+
+### 1. Findings
+
+A compact list or table with:
+
+- section / heading;
+- exact problem;
+- why it matters;
+- source;
+- proposed minimal fix.
+
+### 2. Patch plan
+
+Describe the smallest useful set of edits.
+
+### 3. Patch
+
+Provide a ready-to-apply Markdown diff or replacement blocks.
+
+### 4. Validation notes
+
+State explicitly:
+
+- versions checked;
+- sources used;
+- queries tested;
+- any assumptions not fully verified.
+
+### 5. Risk notes
+
+Call out anything destructive, compatibility-sensitive, or potentially misleading.
+
+## Decision rules: issue vs patch vs no change
+
+Choose the lightest valid outcome.
+
+- **No change**: the article is already correct enough and proposed edits are cosmetic.
+- **Issue / review notes**: there are factual questions or useful ideas, but they are not fully validated.
+- **Small patch**: the change is narrow, source-backed, and tested.
+- **Larger patch**: only when the article has real structural problems and the rewrite is justified with a reviewable rationale.
+
+Default to smaller.
+
+## Final checklist
+
+Before finishing, verify all of the following:
+
+- The full article was read.
+- Related repo discussion was read.
+- Version-sensitive claims are scoped.
+- Historical behavior is labeled as historical.
+- Added queries were tested or explicitly version-gated.
+- Every query has a reason to exist.
+- Destructive actions have warnings.
+- Useful existing detail was not accidentally removed.
+- The diff is reviewable.
+- A human can verify each important claim quickly.
+
+If any item above fails, reduce scope or stop at findings instead of forcing a patch.
diff --git a/README.md b/README.md
index 1339a1c617..c857183f29 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
Welcome to the Altinity Knowledgebase Repository! This Knowledgebase was established for Altinity Engineers and ClickHouse community members to work together to find common solutions.
-Submissions and merges to this repository are distrubuted at https://kb.altinity.com .
+Submissions and merges to this repository are distributed at https://kb.altinity.com .
This knowledgebase is licensed under Apache 2.0. Contributors who submit to the Altinity Knowledgebase agree to the Altinity Contribution License Agreement.
diff --git a/assets/icons/logo.svg b/assets/icons/logo.svg
index 063493b0af..192e1a9eb5 100644
--- a/assets/icons/logo.svg
+++ b/assets/icons/logo.svg
@@ -1 +1,46 @@
-
\ No newline at end of file
+
diff --git a/assets/scss/_content.scss b/assets/scss/_content.scss
index 7fea32581b..8afa573e93 100755
--- a/assets/scss/_content.scss
+++ b/assets/scss/_content.scss
@@ -4,12 +4,10 @@ Styles to override the theme.
*/
-.td-navbar {
- background: #333;
-}
-.td-navbar .navbar-brand svg {
- margin: -8px 10px 0;
+
+footer.bg-dark {
+ background: #132f48 !important;
}
img {
@@ -70,11 +68,11 @@ th {
tr:nth-child(odd) {
background: $td-sidebar-bg-color;
-}
+}
tr:nth-child(even) {
background-color: rgba(233, 236, 240, 0.5);
-}
+}
.feedback--title {
@@ -85,3 +83,92 @@ tr:nth-child(even) {
.feedback--answer {
width: 4em;
}
+
+
+// LEFT SIDEBAR
+
+@media (min-width: 768px){
+ .td-sidebar-nav {
+ min-height: 100%;
+ }}
+
+#m-upgrade_ebook,
+#m-join_slack,
+#m-maintenance_ebook,
+#m-clickhouse_training {
+ font-weight: bold;
+ color: #189DD0;
+ padding-left: 20px !important;
+ font-size: 15px;
+}
+
+#m-upgrade_ebook:hover span,
+#m-join_slack:hover span,
+#m-maintenance_ebook:hover span,
+#m-clickhouse_training:hover span {
+ text-decoration: underline;
+}
+
+#m-clickhouse_training {
+ background:url('data:image/svg+xml,') left 3px no-repeat transparent;
+ background-size: 17px;
+}
+
+#m-contact_us {
+ background:url('data:image/svg+xml,') left 3px no-repeat transparent;
+ background-size: 17px;
+}
+
+#m-join_slack {
+ background:url('data:image/svg+xml,') left 3px no-repeat transparent;
+ background-size: 17px;
+}
+
+#m-maintenance_ebook {
+ background:url('data:image/svg+xml,') left 3px no-repeat transparent;
+ background-size: 17px;
+}
+
+#m-upgrade_ebook {
+ background:url('data:image/svg+xml,') left 3px no-repeat transparent;
+ background-size: 17px;
+}
+
+#m-join_slack-li,
+#m-upgrade_ebook-li {
+ padding-top:20px;
+ border-top: 1px #189DD0 solid;
+ margin-top:20px;
+}
+
+
+
+footer {
+ min-height: auto !important;
+ color: #fff;
+}
+footer a, footer a:hover, footer a:active {
+ color: #fff;
+}
+
+footer .nav li {
+ font-size: 14px;
+ line-height: 1.8;
+}
+
+// Twitter icon fix
+
+footer i.fab.fa-twitter:before{
+ content: ' ';
+ width: 24px;
+ height: 24px;
+ display:inline-block;
+ background: url('data:image/svg+xml,') center bottom no-repeat transparent;
+ background-size: contain;
+ vertical-align: -3px;
+}
+
+footer .footer-inner {
+ max-width: 1280px;
+ margin: 0 auto;
+}
diff --git a/assets/scss/_nav.scss b/assets/scss/_nav.scss
new file mode 100755
index 0000000000..add3b5c29c
--- /dev/null
+++ b/assets/scss/_nav.scss
@@ -0,0 +1,374 @@
+//
+// Main navbar
+//
+
+.navbar {
+ max-width: 1280px;
+ background: #020a2b;
+ color: #fff;
+ margin: 0 auto;
+ padding: .5rem 1rem;
+}
+
+li.nav-item {
+ margin-bottom: 0;
+}
+
+.navbar-dark .navbar-nav .nav-link,
+.navbar-dark .navbar-nav .nav-link:hover,
+.navbar-dark .navbar-nav .nav-link:focus {
+ color: #fff;
+ font-size: 17px;
+ font-weight: 700;
+ line-height: 65px;
+ margin-bottom: 0;
+ padding: 0 10px;
+}
+
+.dropdown-toggle::after {
+ display: inline-block !important;
+ border: 0 none;
+ margin: 0 0 2px 6px;
+ width: 15px !important;
+ height: 9px;
+ background: url('data:image/svg+xml,') center center no-repeat transparent !important;
+ background-size: contain !important;
+ vertical-align:middle;
+}
+
+.dropdown-menu {
+ top: 60px;
+ width: 100%;
+ z-index: 1000;
+ display: none;
+ min-width: 16em;
+ padding: 0 0 1rem;
+ margin: 0 auto;
+ font-size: 1rem;
+ color: #fff;
+ text-align: left;
+ list-style-type: none;
+ background: transparent;
+ border-radius: 0 0 8px 8px;
+ box-shadow: none;
+ background-color: #020a2b;
+ max-width: 1280px;
+ left: 50%;
+ transform: translateX(-50%);
+}
+
+header .dropdown:hover > .dropdown-menu {
+ display: block;
+ margin-top: 0; /* Aligns the menu right under the button without a gap */
+}
+
+.mega-sub-menu {
+ list-style-type: none;
+}
+
+.dropdown-menu > .container {
+ margin: 0 auto;
+ max-width: 1280px;
+}
+
+
+.dropdown-item,
+.dropdown-item:hover,
+.dropdown-item:focus {
+ display: block;
+ width: 100%;
+ padding: 0.5rem 1.5rem;
+ clear: both;
+ font-weight: 400;
+ color: #fff;
+ text-align: inherit;
+ white-space: nowrap;
+ background-color: transparent;
+ border: 0;
+}
+
+.dropdown-item:hover,
+.dropdown-item:focus {
+ text-decoration:underline;
+ background: transparent;
+}
+
+.navbar-button, .navbar-button:hover {
+ margin: 0 5px;
+ display:inline-block;
+ border-radius: 15px;
+ padding: 4px 13px;
+ background-color: #1e9dcf;
+ color: white;
+ font-size: 13px;
+ line-height: 17px;
+ font-weight: 700;
+ text-decoration:none;
+}
+
+.header-social-wrap {
+ display: flex;
+ margin: auto 0 auto auto !important;
+ align-items: center;
+}
+
+
+.header-social-item {
+ display:inline-flex;
+ align-items: center;
+ justify-content: center;
+ color: white;
+ opacity: 0.8;
+ width: 44px;
+ height: 44px;
+ padding: 0 11px;
+ margin: auto 0;
+}
+
+.header-social-item:hover {
+ color: white;
+ opacity: 1;
+}
+
+.header-social-item svg {
+ max-width:22px;
+}
+
+@media (min-width: 1200px) {
+ body>header {
+ position: fixed;
+ top: 0;
+ width: 100%;
+ background: #020a2b;
+ z-index:1000;
+ min-height: 65px;
+ backdrop-filter: blur(4px);
+ }
+ .navbar {
+ position:relative;
+ margin: 0 auto;
+ padding-left: 5px;
+ padding-right: 5px;
+ background:transparent;
+ min-height: 65px;
+ padding-top:0;
+ padding-bottom:0;
+ }
+
+ .navbar .navbar-brand {
+ padding: 0;
+ margin: 0;
+ width: 150px;
+ flex-shrink: 0;
+ margin-right:20px
+ }
+
+ .td-sidebar {
+ padding-top: 75px;
+ background-color: #e9ecf0;
+ padding-right: 1rem;
+ border-right: 1px solid #dee2e6;
+ }
+
+ .td-sidebar-toc {
+ border-left: 1px solid $border-color;
+
+ @supports (position: sticky) {
+ position: sticky;
+ top: 75px;
+ height: calc(100vh - 85px);
+ overflow-y: auto;
+ }
+ order: 2;
+ padding-top: 5px;
+ padding-bottom: 1.5rem;
+ vertical-align: top;
+ }
+
+ #mobile_navbar {
+ display: none !important;
+ }
+
+
+}
+
+#mobile_navbar {
+ position:fixed;
+ inset: 60px 0 0 32px;
+ background: #020a2b;
+ z-index: 100000;
+ overflow-y: auto;
+ padding-bottom: 100px;
+}
+
+
+#mobile_navbar ul{
+ list-stye-type: none;
+ display:block;
+ padding-left: 14px;
+ margin-left: 0;
+ position:relative;
+}
+
+#mobile_navbar .nav-item {
+ list-stye-type: none;
+ display:block;
+ padding-left: 0;
+ margin-left: 0;
+ position:relative;
+}
+
+#mobile_navbar a {
+ font-size: 14px;
+ border-bottom: 1px solid rgba(255, 255, 255, 0.2);
+ display: block;
+ padding: 14px 7px;
+ color: white;
+ opacity: 0.8;
+}
+
+#mobile_navbar a:hover {
+ opacity: 1;
+}
+
+#mobile_navbar .has-collapse-sub-nav .nav-item-has-children>.sub-menu {
+ display: none;
+}
+
+#mobile_navbar .has-collapse-sub-nav .nav-item-has-children.active>.sub-menu {
+ display: block;
+}
+
+.mobile-drop-toggle {
+display: block;
+position: absolute;
+right: 0;
+top: 0;
+width: 50px;
+height: 50px;
+background: url('data:image/svg+xml,') center center no-repeat transparent;
+background-size: 15px;
+border-left: 1px solid rgba(255,255,255,0.1);
+cursor:pointer;
+}
+
+.nav-item-has-children.active >.drawer-nav-drop-wrap .mobile-drop-toggle {
+ transform: rotate(-180deg);
+}
+
+
+// MEGA MENUS
+
+.mega-menu {
+ letter-spacing: 0.5px !important;
+ color: #fff;
+ line-height: 1.3;
+
+ h2,
+ h3,
+ h4,
+ h5 {
+ color: #fff;
+ padding-bottom: 0.5rem;
+ font-weight: 700;
+ }
+
+ p {
+ margin-top: 0;
+ }
+
+ a {
+ display: inline-block;
+ color: #fff;
+ text-decoration: none;
+ font-weight: 600;
+ font-size: 17px;
+ line-height: 150%;
+ }
+
+ a:hover {
+ color: #fff;
+ text-decoration: underline;
+ }
+
+ p.col-header {
+ margin: 0 -20px 15px -30px;
+ padding: 0 20px 10px 30px;
+ border-bottom: 1px solid rgba(255, 255, 255, 0.3);
+ font-size: 1.2rem;
+ }
+
+ .desc,
+ .desc a {
+ font-size: 14px;
+ font-weight: 300;
+ line-height: 1.3;
+ }
+
+ .desc {
+ opacity: 0.8;
+ padding-bottom: 5px;
+ }
+
+ .sub_links {
+ margin-top: 10px;
+ }
+
+ .small-links a {
+ font-size: 15px;
+ }
+
+ .mega-bg-blue {
+ background: rgba(75, 156, 193, .35) !important;
+ }
+
+ .grid-row {
+ display: grid;
+ grid-auto-columns: minmax(0, 1fr);
+ grid-auto-flow: column;
+ }
+
+ .grid-row>div {
+ padding: 20px 20px 20px 30px;
+ }
+
+ .navicon-item {
+ margin-bottom: 20px;
+ padding-left: 55px;
+ background-repeat: no-repeat;
+ background-position: 0 2px;
+ }
+
+ .icon-item:last-child {
+ margin-bottom: 0;
+ }
+
+ .sub-section-header {
+ margin-left: -10px;
+ padding: 5px 5px 5px 10px;
+ margin-bottom: 10px;
+ background: rgba(24, 157, 208, 0.15);
+ border-radius: 8px;
+ font-size: 20px;
+ color: #fff;
+ }
+
+ .med-link {
+ font-size: 16px;
+ }
+
+ .mega-menu-column .button {
+ font-size: 16px;
+ font-weight: 400;
+ min-width: 150px;
+ display: inline-block;
+ text-align: center;
+ }
+
+ .slack-button {
+ background: #7B00A0 !important;
+ }
+
+}
+
+
diff --git a/config.toml b/config.toml
index 8b05c360ee..366ba6710b 100644
--- a/config.toml
+++ b/config.toml
@@ -1,6 +1,6 @@
baseURL = "http://kb.altinity.com/"
languageCode = "en-us"
-title = "Altinity Knowledge Base"
+title = "Altinity® Knowledge Base for ClickHouse®"
# theme = ["docsy"]
publishDir = "docs"
enableRobotsTXT = true
@@ -54,8 +54,8 @@ anchor = "smart"
[languages]
[languages.en]
-title = "Altinity Knowledge Base"
-description = "Altinity Knowledge Base"
+title = "Altinity® Knowledge Base for ClickHouse®"
+description = "Altinity® Knowledge Base for ClickHouse®"
languageName = "English"
# Weight used for sorting.
weight = 1
@@ -76,7 +76,7 @@ time_format_blog = "2006.01.02"
[params]
# copyright = " Altinity Inc."
-copyright = " Altinity Inc. Altinity® and Altinity.Cloud® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc."
+copyright = " Altinity Inc. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc. Kafka, Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners."
privacy_policy = "https://altinity.com/privacy-policy/"
favicon = "/favicon.ico"
@@ -158,28 +158,64 @@ enable = false
[params.links]
# End user relevant links. These will show up on left side of footer and in the community page if you have one.
-[[params.links.user]]
- name ="Twitter"
+[[params.links.developer]]
+ name ="Slack"
+ url = "https://altinity.com/slack"
+ icon = "fab fa-slack"
+ desc = "Join our Slack Community"
+[[params.links.developer]]
+ name ="X"
url = "https://twitter.com/AltinityDB"
icon = "fab fa-twitter"
- desc = "Follow us on Twitter to get the latest news!"
-[[params.links.user]]
+ desc = "Follow us on X to get the latest news!"
+[[params.links.developer]]
+ name = "LinkedIn"
+ url = "https://www.linkedin.com/company/altinity/"
+ icon = "fab fa-linkedin"
+ desc = "Partner with us on LinkedIn."
+[[params.links.developer]]
name = "Youtube"
url = "https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA"
icon = "fab fa-youtube"
desc = "Watch our videos."
-[[params.links.user]]
- name = "LinkedIn"
- url = "https://www.linkedin.com/company/altinity/"
- icon = "fab fa-linkedin"
- desc = "Partner with us on LinkedIn."
# Developer relevant links. These will show up on right side of footer and in the community page if you have one.
[[params.links.developer]]
name = "GitHub"
- url = "https://github.com/orgs/Altinity/"
+ url = "https://github.com/Altinity/altinityknowledgebase"
icon = "fab fa-github"
- desc = "Development takes place here!"
-
+ desc = "Development takes place here!"
+[[params.links.developer]]
+ name = "Reddit"
+ url = "https://www.reddit.com/r/Clickhouse/"
+ icon = "fab fa-reddit"
+ desc = "Altinity on Reddit"
+[params.llms]
+ # Set to false to disable /llms.txt generation
+ generate_llms_txt = true
+
+ # Set to false to disable /llms-full.txt generation
+ generate_llms_full_txt = true
+
+ # Include only specific pages or directories.
+ # If empty, all pages are included by default.
+ # If populated, ONLY paths matching the list will be generated.
+ # Both llms.txt and llms-full.txt respect this setting.
+ # Examples:
+ # "/about/" → include strictly the /about/ page
+ # "/blog/*" → include immediate children of /blog/ (e.g. /blog/post-1/)
+ # "/blog/**" → include /blog/ and all nested pages and directories
+ include = []
+
+ # Exclude specific pages or directories.
+ # Follows the same wildcard formats as include.
+ # Conflict handling: If the EXACT same path string is put in both include and exclude,
+ # the include rule has higher priority (the exclude rule is ignored).
+ # Both llms.txt and llms-full.txt respect this setting.
+ # Examples:
+ # "/about/" → exclude exactly /about/
+ # "/blog/**" → exclude all pages under /blog/
+ # "/blog/post-1/" → exclude a specific page
+ exclude = []
[outputFormats]
[outputFormats.PRINT]
baseName = "index"
@@ -187,10 +223,22 @@ isHTML = true
mediaType = "text/html"
path = "printview"
permalinkable = false
+[outputFormats.llms]
+ baseName = "llms"
+ isPlainText = true
+ mediaType = "text/plain"
+ rel = "alternate"
+ root = true
+[outputFormats.llmsfull]
+ baseName = "llms-full"
+ mediaType = "text/plain"
+ isPlainText = true
+ notAlternative = true
+ root = true
# Controls the ability to print.
[outputs]
-#home = ["HTML", "print"]
+home = ["HTML", "print", "llms","llmsfull", "md"]
section = [ "HTML", "print"]
#page = ["HTML", "print"]
diff --git a/content/AGENTS.md b/content/AGENTS.md
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/content/en/_index.md b/content/en/_index.md
index 92ba47dae1..37d1a48613 100755
--- a/content/en/_index.md
+++ b/content/en/_index.md
@@ -1,7 +1,7 @@
---
-title: "Altinity Knowledge Base"
-linkTitle: "Altinity Knowledge Base"
-description: "Up-to-date ClickHouse knowledge base for every ClickHouse user."
+title: "Altinity® Knowledge Base for ClickHouse®"
+linkTitle: "Altinity® Knowledge Base for ClickHouse®"
+description: "Up-to-date ClickHouse® knowledge base for every ClickHouse user."
keywords:
- ClickHouse Knowledge Base
- Altinity Knowledge Base
@@ -12,20 +12,26 @@ cascade:
_target:
path: "/**"
---
-## Welcome to the Altinity ClickHouse Knowledge Base (KB)
+## Welcome to the Altinity® Knowledge Base (KB) for ClickHouse®
This knowledge base is supported by [Altinity](http://altinity.com/) engineers to provide quick answers to common questions and issues involving ClickHouse.
-The [Altinity Knowledge Base is licensed under Apache 2.0](https://github.com/Altinity/altinityknowledgebase/blob/main/LICENSE), and available to all ClickHouse users. The information and code samples are available freely and distributed under the Apache 2.0 license.
+The [Altinity Knowledge Base is licensed under Apache 2.0,](https://github.com/Altinity/altinityknowledgebase/blob/main/LICENSE) and is available to all ClickHouse users. The information and code samples are available freely and distributed under the Apache 2.0 license.
For more detailed information about Altinity services support, see the following:
* [Altinity](https://altinity.com/): Providers of Altinity.Cloud, providing SOC-2 certified support for ClickHouse.
-* [Altinity ClickHouse Documentation](https://docs.altinity.com): Detailed guides on installing and connecting ClickHouse to other services.
-* [Altinity Resources](https://altinity.com/resources/): News, blog posts, and webinars about ClickHouse and Altinity services.
+* [Altinity.com Documentation](https://docs.altinity.com): Detailed guides on working with:
+ * [Altinity.Cloud](https://docs.altinity.com/altinitycloud/)
+ * [Altinity.Cloud Anywhere](https://docs.altinity.com/altinitycloudanywhere/)
+ * [The Altinity Cloud Manager](https://docs.altinity.com/altinitycloud/quickstartguide/clusterviewexplore/)
+ * [The Altinity Kubernetes Operator for ClickHouse](https://docs.altinity.com/releasenotes/altinity-kubernetes-operator-release-notes/)
+ * [The Altinity Sink Connector for ClickHouse](https://docs.altinity.com/releasenotes/altinity-sink-connector-release-notes/) and
+ * [Altinity Backup for ClickHouse](https://docs.altinity.com/releasenotes/altinity-backup-release-notes/)
+* [Altinity Blog](https://altinity.com/blog/): Blog posts about ClickHouse the database and Altinity services.
The following sites are also useful references regarding ClickHouse:
-* [ClickHouse.tech documentation](https://clickhouse.tech/docs/en/): From Yandex, the creators of ClickHouse
+* [ClickHouse.com documentation](https://clickhouse.com/docs/en/): Official documentation from ClickHouse Inc.
* [ClickHouse at Stackoverflow](https://stackoverflow.com/questions/tagged/clickhouse): Community driven responses to questions regarding ClickHouse
* [Google groups (Usenet) yes we remember it](https://groups.google.com/g/clickhouse): The grandparent of all modern discussion boards.
diff --git a/content/en/altinity-kb-dictionaries/_index.md b/content/en/altinity-kb-dictionaries/_index.md
index 15cc00ca4e..64bcc3214e 100644
--- a/content/en/altinity-kb-dictionaries/_index.md
+++ b/content/en/altinity-kb-dictionaries/_index.md
@@ -6,11 +6,11 @@ keywords:
- clickhouse arrays
- postgresql dictionary
description: >
- All you need to know about creating and using ClickHouse dictionaries.
+ All you need to know about creating and using ClickHouse® dictionaries.
weight: 11
---
-For more information on ClickHouse Dictionaries, see
+For more information on ClickHouse® Dictionaries, see
the presentation [https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup34/clickhouse_integration.pdf](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup34/clickhouse_integration.pdf), slides 82-95, video https://youtu.be/728Yywcd5ys?t=10642
@@ -20,7 +20,3 @@ https://altinity.com/blog/2020/5/19/clickhouse-dictionaries-reloaded
And some videos:
https://www.youtube.com/watch?v=FsVrFbcyb84
-
-Also there 3rd party articles on the same subj.
-https://prog.world/how-to-create-and-use-dictionaries-in-clickhouse/
-
diff --git a/content/en/altinity-kb-dictionaries/altinity-kb-sparse_hashed-vs-hashed.md b/content/en/altinity-kb-dictionaries/altinity-kb-sparse_hashed-vs-hashed.md
index bf27340627..5a2f5ce070 100644
--- a/content/en/altinity-kb-dictionaries/altinity-kb-sparse_hashed-vs-hashed.md
+++ b/content/en/altinity-kb-dictionaries/altinity-kb-sparse_hashed-vs-hashed.md
@@ -1,10 +1,10 @@
---
-title: "SPARSE_HASHED VS HASHED"
-linkTitle: "SPARSE_HASHED VS HASHED"
+title: "SPARSE_HASHED VS HASHED vs HASHED_ARRAY"
+linkTitle: "SPARSE_HASHED VS HASHED vs HASHED_ARRAY"
description: >
- SPARSE_HASHED VS HASHED
+ SPARSE_HASHED VS HASHED VS HASHED_ARRAY
---
-Sparse_hashed layout is supposed to save memory but has some downsides. We can test how much slower SPARSE_HASHED than HASHED is with the following:
+Sparse_hashed and hashed_array layouts are supposed to save memory but has some downsides. We can test it with the following:
```sql
create table orders(id UInt64, price Float64)
@@ -22,6 +22,11 @@ PRIMARY KEY id SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000
TABLE orders DB 'default' USER 'default'))
LIFETIME(MIN 0 MAX 0) LAYOUT(SPARSE_HASHED());
+CREATE DICTIONARY orders_hashed_array (id UInt64, price Float64)
+PRIMARY KEY id SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000
+TABLE orders DB 'default' USER 'default'))
+LIFETIME(MIN 0 MAX 0) LAYOUT(HASHED_ARRAY());
+
SELECT
name,
type,
@@ -30,26 +35,32 @@ SELECT
formatReadableSize(bytes_allocated) AS RAM
FROM system.dictionaries
WHERE name LIKE 'orders%'
-┌─name──────────┬─type─────────┬─status─┬─element_count─┬─RAM────────┐
-│ orders_sparse │ SparseHashed │ LOADED │ 5000000 │ 84.29 MiB │
-│ orders_hashed │ Hashed │ LOADED │ 5000000 │ 256.00 MiB │
-└───────────────┴──────────────┴────────┴───────────────┴────────────┘
+┌─name────────────────┬─type─────────┬─status─┬─element_count─┬─RAM────────┐
+│ orders_hashed_array │ HashedArray │ LOADED │ 5000000 │ 68.77 MiB │
+│ orders_sparse │ SparseHashed │ LOADED │ 5000000 │ 76.30 MiB │
+│ orders_hashed │ Hashed │ LOADED │ 5000000 │ 256.00 MiB │
+└─────────────────────┴──────────────┴────────┴───────────────┴────────────┘
SELECT sum(dictGet('default.orders_hashed', 'price', toUInt64(number))) AS res
FROM numbers(10000000)
┌─res─┐
│ 0 │
└─────┘
-1 rows in set. Elapsed: 0.279 sec. Processed 10.02 million rows ...
+1 rows in set. Elapsed: 0.546 sec. Processed 10.01 million rows ...
SELECT sum(dictGet('default.orders_sparse', 'price', toUInt64(number))) AS res
FROM numbers(10000000)
┌─res─┐
│ 0 │
└─────┘
-1 rows in set. Elapsed: 1.085 sec. Processed 10.02 million rows ...
-```
+1 rows in set. Elapsed: 1.422 sec. Processed 10.01 million rows ...
-As you can see **SPARSE_HASHED** is memory efficient and use about 3 times less memory (!!!) but is almost 4 times slower. But this is the ultimate case because this test does not read data from the disk (no MergeTree table involved).
+SELECT sum(dictGet('default.orders_hashed_array', 'price', toUInt64(number))) AS res
+FROM numbers(10000000)
+┌─res─┐
+│ 0 │
+└─────┘
+1 rows in set. Elapsed: 0.558 sec. Processed 10.01 million rows ...
+```
-We encourage you to test **SPARSE_HASHED** against your real queries, because it able to save a lot of memory and have larger (in rows) external dictionaries.
+As you can see **SPARSE_HASHED** is memory efficient and use about 3 times less memory (!!!) but is almost 3 times slower as well. On the other side **HASHED_ARRAY** is even more efficient in terms of memory usage and maintains almost the same performance as **HASHED** layout.
diff --git a/content/en/altinity-kb-dictionaries/dictionaries-and-arrays.md b/content/en/altinity-kb-dictionaries/dictionaries-and-arrays.md
index db0352e2f2..8cfa4b0027 100644
--- a/content/en/altinity-kb-dictionaries/dictionaries-and-arrays.md
+++ b/content/en/altinity-kb-dictionaries/dictionaries-and-arrays.md
@@ -4,7 +4,7 @@ linkTitle: "Dictionaries & arrays"
description: >
Dictionaries & arrays
---
-## Dictionary with Clickhouse table as a source
+## Dictionary with ClickHouse® table as a source
### Test data
diff --git a/content/en/altinity-kb-dictionaries/dictionary-on-top-tables.md b/content/en/altinity-kb-dictionaries/dictionary-on-top-tables.md
index 8dc6c35ea4..a7fa9c0cf5 100644
--- a/content/en/altinity-kb-dictionaries/dictionary-on-top-tables.md
+++ b/content/en/altinity-kb-dictionaries/dictionary-on-top-tables.md
@@ -1,8 +1,8 @@
---
-title: "Dictionary on the top of the several tables using VIEW"
-linkTitle: "Dictionary on the top of the several tables using VIEW"
+title: "Dictionary on the top of several tables using VIEW"
+linkTitle: "Dictionary on the top of several tables using VIEW"
description: >
- Dictionary on the top of the several tables using VIEW
+ Dictionary on the top of several tables using VIEW
---
```sql
diff --git a/content/en/altinity-kb-dictionaries/dimension_table_desing.md b/content/en/altinity-kb-dictionaries/dimension_table_desing.md
new file mode 100644
index 0000000000..21c5f4bcb8
--- /dev/null
+++ b/content/en/altinity-kb-dictionaries/dimension_table_desing.md
@@ -0,0 +1,161 @@
+---
+title: "Dimension table design "
+linkTitle: "Dimension table design "
+description: >
+ Dimension table design
+---
+## Dimension table design considerations
+
+### Choosing storage Engine
+
+To optimize the performance of reporting queries, dimensional tables should be loaded into RAM as ClickHouse Dictionaries whenever feasible. It's becoming increasingly common to allocate 100-200GB of RAM per server specifically for these Dictionaries. Implementing sharding by tenant can further reduce the size of these dimension tables, enabling a greater portion of them to be stored in RAM and thus enhancing query speed.
+
+Different Dictionary Layouts can take more or less RAM (in trade for speed).
+
+- The cached dictionary layout is ideal for minimizing the amount of RAM required to store dimensional data when the hit ratio is high. This layout allows frequently accessed data to be kept in RAM while less frequently accessed data is stored on disk, thereby optimizing memory usage without sacrificing performance.
+- HASHED_ARRAY or SPARSE_HASHED dictionary layouts take less RAM than HASHED. See tests [here](https://kb.altinity.com/altinity-kb-dictionaries/altinity-kb-sparse_hashed-vs-hashed/).
+- Normalization techniques can be used to lower RAM usage (see below)
+
+If the amount of data is so high that it does not fit in the RAM even after suitable sharding, a disk-based table with an appropriate engine and its parameters can be used for accessing dimensional data in report queries.
+
+MergeTree engines (including Replacing or Aggregating) are not tuned by default for point queries due to the high index granularity (8192) and the necessity of using FINAL (or GROUP BY) when accessing mutated data.
+
+When using the MergeTree engine for Dimensions, the table’s index granularity should be lowered to 256. More RAM will be used for PK, but it’s a reasonable price for reading less data from the disk and making report queries faster, and that amount can be lowered by lightweight PK design (see below).
+
+The `EmbeddedRocksDB` engine could be used as an alternative. It performs much better than ReplacingMergeTree for highly mutated data, as it is tuned by design for random point queries and high-frequency updates. However, EmbeddedRocksDB does not support Replication, so INSERTing data to such tables should be done over a Distributed table with `internal_replication` set to false, which is vulnerable to different desync problems. Some “sync” procedures should be designed, developed, and applied after serious data ingesting incidents (like ETL crashes).
+
+When the Dimension table is built on several incoming event streams, `AggregatingMergeTree` is preferable to `ReplacingMergeTree`, as it allows putting data from different event streams without external ETL processes:
+
+```sql
+CREATE TABLE table_C (
+ id UInt64,
+ colA SimpleAggregatingFunction(any,Nullable(UInt32)),
+ colB SimpleAggregatingFunction(max, String)
+) ENGINE = AggregatingMergeTree()
+PARTITION BY intDiv(id, 0x800000000000000) /* 32 bucket*/
+ORDER BY id;
+
+CREATE MATERIALIZED VIEW mv_A TO table_C AS SELECT id,colA FROM Kafka_A;
+CREATE MATERIALIZED VIEW mv_B TO table_C AS SELECT id,colB FROM Kafka_B;
+```
+
+EmbeddedRocksDB natively supports UPDATEs without any complications with AggregatingFunctions.
+
+For dimensions where some “start date” column is used in filtering, the [Range_Hashed](https://kb.altinity.com/altinity-kb-dictionaries/altinity-kb-range_hashed-example-open-intervals/) dictionary layout can be used if it is acceptable for RAM usage. For MergeTree variants, ASOF JOIN in queries is needed. Such types of dimensions are the first candidates for placement into RAM.
+
+EmbeddedRocksDB is not suitable here.
+
+### Primary Key
+
+To increase query performance, I recommend using a single UInt64 (not String) column for PK, where the upper 32 bits are reserved for tenant_id (shop_id) and the lower 32 bits for actual object_id (like customer_id, product_id, etc.)
+
+That benefits both EmbeddedRocksDB Engine (it can have only one Primary Key column) and ReplacingMergeTree, as FINAL processing will work much faster with a light ORDER BY column of a single UInt64 value.
+
+### Direct Dictionary and UDFs
+
+To make the SQL code of report queries more readable and manageable, I recommend always using Dictionaries to access dimensions. A `direct dictionary layout` should be used for disk-stored dimensions (EmbeddedRocksDB or *MergeTree).
+
+When Clickhouse builds a query to Direct Dictionary, it automatically creates a filter with a list of all needed ID values. There is no need to write code to filter necessary dimension rows to reduce the hash table for the right join table.
+
+Another trick for code manageability is creating an interface function for every dimension to place here all the complexity of managing IDs by packing several values into a single PK value:
+
+```sql
+create or replace function getCustomer as (shop, id, attr) ->
+ dictGetOrNull('dict_Customers', attr, bitOr((bitShiftLeft(toUInt64(shop),32)),id));
+```
+
+It also allows the flexibility of changing dictionary names when testing different types of Engines or can be used to spread dimensional data to several dictionaries. F.e. most active tenants can be served by expensive in-RAM dictionary, while others (not active) tenants will be served from disk.
+
+```sql
+create or replace function getCustomer as (shop, id, attr) ->
+ dictGetOrDefault('dict_Customers_RAM', attr, bitOr((bitShiftLeft(toUInt64(shop),32)),id) as key,
+ dictGetOrNull('dict_Customers_MT', attr, key));
+```
+
+We always recommended DENORMALIZATION for Fact tables. However, NORMALIZATION is still a usable approach for taking less RAM for Dimension data stored as dictionaries.
+
+Example of storing a long company name (String) in a separate dictionary:
+
+```sql
+create or replace function getCustomer as (shop, id, attr) ->
+ if(attr='company_name',
+ dictGetOrDefault('dict_Company_name', 'name',
+ dictGetOrNull('dict_Customers', 'company_id',
+ bitOr((bitShiftLeft(toUInt64(shop),32)),id)) as key),
+ dictGetOrNull('dict_Customers', attr, key)
+ );
+```
+
+Example of combining Hash and Direct Dictionaries. Allows to increase lifetime without losing consistency.
+
+```sql
+CREATE OR REPLACE FUNCTION getProduct AS (product_id, attr) ->
+ dictGetOrDefault('hashed_dictionary', attr,(shop_id, product_id),
+ dictGet('direct_dictionary',attr,(shop_id, product_id) )
+ );
+```
+
+### Tests/Examples
+
+EmbeddedRocksDB
+
+```sql
+CREATE TABLE Dim_Customers (
+ id UInt64,
+ name String,
+ new_or_returning bool
+) ENGINE = EmbeddedRocksDB()
+PRIMARY KEY (id);
+
+INSERT INTO Dim_Customers
+SELECT bitShiftLeft(3648061509::UInt64,32)+number,
+ ['Customer A', 'Customer B', 'Customer C', 'Customer D', 'Customer E'][number % 5 + 1],
+ number % 2 = 0
+FROM numbers(100);
+
+CREATE DICTIONARY dict_Customers
+(
+ id UInt64,
+ name String,
+ new_or_returning bool
+)
+PRIMARY KEY id
+LAYOUT(DIRECT())
+SOURCE(CLICKHOUSE(TABLE 'Dim_Customers'));
+
+select dictGetOrNull('dict_Customers', 'name',
+ bitOr((bitShiftLeft(toUInt64(shop_id),32)),customer_id));
+```
+
+ReplacingMergeTree
+
+```sql
+CREATE TABLE Dim_Customers (
+ id UInt64,
+ name String,
+ new_or_returning bool
+) ENGINE = ReplacingMergeTree()
+ORDER BY id
+PARTITION BY intDiv(id, 0x800000000000000) /* 32 buckets by shop_id */
+settings index_granularity=256;
+
+CREATE DICTIONARY dict_Customers
+(
+ id UInt64,
+ name String,
+ new_or_returning bool
+)
+PRIMARY KEY id
+LAYOUT(DIRECT())
+SOURCE(CLICKHOUSE(query 'select * from Dim_Customers FINAL'));
+
+set do_not_merge_across_partitions_select_final=1; -- or place it to profile
+select dictGet('dict_Customers','name',bitShiftLeft(3648061509::UInt64,32)+1);
+```
+
+Tests 1M random reads over 10M entries per shop_id in the Dimension table
+
+- [EmbeddedRocksDB](https://fiddle.clickhouse.com/c304d0cc-f1c2-4323-bd65-ab82165aecb6) - 0.003s
+- [ReplacingMergeTree](https://fiddle.clickhouse.com/093fc133-0685-4c97-aa90-d38200f93f9f)- 0.003s
+
+There is no difference in SELECT on that synthetic test with all MergeTree optimizations applied. The test must be rerun on actual data with the expected update volume. The difference could be seen on a table with high-volume real-time updates.
diff --git a/content/en/altinity-kb-dictionaries/mysql8-source-for-dictionaries.md b/content/en/altinity-kb-dictionaries/mysql8-source-for-dictionaries.md
index 3554bcda00..650519dff6 100644
--- a/content/en/altinity-kb-dictionaries/mysql8-source-for-dictionaries.md
+++ b/content/en/altinity-kb-dictionaries/mysql8-source-for-dictionaries.md
@@ -6,7 +6,7 @@ description: >
---
#### Authorization
-MySQL8 used default authorization plugin `caching_sha2_password`. Unfortunately, `libmysql` which currently used (21.4-) in clickhouse is not.
+MySQL8 used default authorization plugin `caching_sha2_password`. Unfortunately, `libmysql` which currently used (21.4-) in ClickHouse® is not.
You can fix it during create custom user with `mysql_native_password` authentication plugin.
diff --git a/content/en/altinity-kb-dictionaries/partial-updates.md b/content/en/altinity-kb-dictionaries/partial-updates.md
index 088e562e84..8efd312c3a 100644
--- a/content/en/altinity-kb-dictionaries/partial-updates.md
+++ b/content/en/altinity-kb-dictionaries/partial-updates.md
@@ -4,7 +4,7 @@ linkTitle: "Partial updates"
description: >
Partial updates
---
-Clickhouse is able to fetch from a source only updated rows. You need to define `update_field` section.
+ClickHouse® is able to fetch from a source only updated rows. You need to define `update_field` section.
As an example, We have a table in an external source MySQL, PG, HTTP, ... defined with the following code sample:
@@ -36,4 +36,4 @@ LIFETIME(MIN 30 MAX 30)
A dictionary with **update_field** `updated_at` will fetch only updated rows. A dictionary saves the current time (now) time of the last successful update and queries the source `where updated_at >= previous_update - 1` (shift = 1 sec.).
-In case of HTTP source Clickhouse will send get requests with **update_field** as an URL parameter `&updated_at=2020-01-01%2000:01:01`
+In case of HTTP source ClickHouse will send get requests with **update_field** as an URL parameter `&updated_at=2020-01-01%2000:01:01`
diff --git a/content/en/altinity-kb-dictionaries/security-named-collections.md b/content/en/altinity-kb-dictionaries/security-named-collections.md
new file mode 100644
index 0000000000..5c34dd11d8
--- /dev/null
+++ b/content/en/altinity-kb-dictionaries/security-named-collections.md
@@ -0,0 +1,47 @@
+---
+title: "Security named collections"
+linkTitle: "Security named collections"
+description: >
+ Security named collections
+---
+
+
+## Dictionary with ClickHouse® table as a source with named collections
+
+### Data for connecting to external sources can be stored in named collections
+
+```xml
+
+
+
+ localhost
+ 9000
+ default
+ ch_dict
+ mypass
+
+
+
+```
+
+### Dictionary
+
+```sql
+DROP DICTIONARY IF EXISTS named_coll_dict;
+CREATE DICTIONARY named_coll_dict
+(
+ key UInt64,
+ val String
+)
+PRIMARY KEY key
+SOURCE(CLICKHOUSE(NAME local_host TABLE my_table DB default))
+LIFETIME(MIN 1 MAX 2)
+LAYOUT(HASHED());
+
+INSERT INTO my_table(key, val) VALUES(1, 'first row');
+
+SELECT dictGet('named_coll_dict', 'b', 1);
+┌─dictGet('named_coll_dict', 'b', 1)─┐
+│ first row │
+└────────────────────────────────────┘
+```
diff --git a/content/en/altinity-kb-functions/array-like-memory-usage.md b/content/en/altinity-kb-functions/array-like-memory-usage.md
index a88ceec5a1..69dbedf0f5 100644
--- a/content/en/altinity-kb-functions/array-like-memory-usage.md
+++ b/content/en/altinity-kb-functions/array-like-memory-usage.md
@@ -2,12 +2,12 @@
title: "arrayMap, arrayJoin or ARRAY JOIN memory usage"
linkTitle: "arrayMap, arrayJoin or ARRAY JOIN memory usage"
description: >
- Why arrayMap, arrayFilter, arrayJoin use so much memory?
+ Why do arrayMap, arrayFilter, and arrayJoin use so much memory?
---
## arrayMap-like functions memory usage calculation.
-In order to calculate arrayMap or similar array* functions ClickHouse temporarily does arrayJoin-like operation, which in certain conditions can lead to huge memory usage for big arrays.
+In order to calculate arrayMap or similar array* functions ClickHouse® temporarily does arrayJoin-like operation, which in certain conditions can lead to huge memory usage for big arrays.
So for example, you have 2 columns:
diff --git a/content/en/altinity-kb-functions/arrayfold.md b/content/en/altinity-kb-functions/arrayfold.md
new file mode 100644
index 0000000000..3a66ce055d
--- /dev/null
+++ b/content/en/altinity-kb-functions/arrayfold.md
@@ -0,0 +1,17 @@
+---
+title: "arrayFold"
+linkTitle: "arrayFold"
+---
+
+## EWMA example
+
+```sql
+WITH
+ [40, 45, 43, 31, 20] AS data,
+ 0.3 AS alpha
+SELECT arrayFold((acc, x) -> arrayPushBack(acc, (alpha * x) + ((1 - alpha) * (acc[-1]))), arrayPopFront(data), [CAST(data[1], 'Float64')]) as ewma
+
+┌─ewma─────────────────────────────────────────────────────────────┐
+│ [40,41.5,41.949999999999996,38.66499999999999,33.06549999999999] │
+└──────────────────────────────────────────────────────────────────┘
+```
diff --git a/content/en/altinity-kb-functions/assumenotnull-and-friends.md b/content/en/altinity-kb-functions/assumenotnull-and-friends.md
index 7ba6ca1ee5..760d2afbbf 100644
--- a/content/en/altinity-kb-functions/assumenotnull-and-friends.md
+++ b/content/en/altinity-kb-functions/assumenotnull-and-friends.md
@@ -89,7 +89,7 @@ Code: 36, e.displayText() = DB::Exception: Unexpected value 0 in enum, Stack tra
```
{{% alert title="Info" color="info" %}}
-Null values in ClickHouse are stored in a separate dictionary: is this value Null. And for faster dispatch of functions there is no check on Null value while function execution, so functions like plus can modify internal column value (which has default value). In normal conditions it’s not a problem because on read attempt, ClickHouse first would check the Null dictionary and return value from column itself for non-Nulls only. And `assumeNotNull` function just ignores this Null dictionary. So it would return only column values, and in certain cases it’s possible to have unexpected results.
+Null values in ClickHouse® are stored in a separate dictionary: is this value Null. And for faster dispatch of functions there is no check on Null value while function execution, so functions like plus can modify internal column value (which has default value). In normal conditions it’s not a problem because on read attempt, ClickHouse first would check the Null dictionary and return value from column itself for non-Nulls only. And `assumeNotNull` function just ignores this Null dictionary. So it would return only column values, and in certain cases it’s possible to have unexpected results.
{{% /alert %}}
If it's possible to have Null values, it's better to use `ifNull` function instead.
diff --git a/content/en/altinity-kb-functions/how-to-encode-decode-quantiletdigest-state.md b/content/en/altinity-kb-functions/how-to-encode-decode-quantiletdigest-state.md
new file mode 100644
index 0000000000..766d864fec
--- /dev/null
+++ b/content/en/altinity-kb-functions/how-to-encode-decode-quantiletdigest-state.md
@@ -0,0 +1,85 @@
+---
+title: "How to encode/decode quantileTDigest states from/to list of centroids"
+linkTitle: "Encoding and Decoding of quantileTDigest states"
+weight: 100
+description: >-
+ A way to export or import quantileTDigest states from/into ClickHouse®
+---
+
+## quantileTDigestState
+
+quantileTDigestState is stored in two parts: a count of centroids in LEB128 format + list of centroids without a delimiter. Each centroid is represented as two Float32 values: Mean & Count.
+
+```sql
+SELECT
+ hex(quantileTDigestState(1)),
+ hex(toFloat32(1))
+
+┌─hex(quantileTDigestState(1))─┬─hex(toFloat32(1))─┐
+│ 010000803F0000803F │ 0000803F │
+└──────────────────────────────┴───────────────────┘
+ 01 0000803F 0000803F
+ ^ ^ ^
+ LEB128 Float32 Mean Float32 Count
+```
+
+We need to make two helper `UDF` functions:
+
+```xml
+cat /etc/clickhouse-server/decodeTDigestState_function.xml
+
+
+ executable
+ 0
+ decodeTDigestState
+ Array(Tuple(mean Float32, count Float32))
+
+ AggregateFunction(quantileTDigest, UInt32)
+
+ RowBinary
+ cat
+ 0
+
+
+
+cat /etc/clickhouse-server/encodeTDigestState_function.xml
+
+
+ executable
+ 0
+ encodeTDigestState
+ AggregateFunction(quantileTDigest, UInt32)
+
+ Array(Tuple(mean Float32, count Float32))
+
+ RowBinary
+ cat
+ 0
+
+
+```
+
+Those UDF – `(encode/decode)TDigestState` converts `TDigestState` to the `Array(Tuple(Float32, Float32))` and back.
+
+```sql
+SELECT quantileTDigest(CAST(number, 'UInt32')) AS result
+FROM numbers(10)
+
+┌─result─┐
+│ 4 │
+└────────┘
+
+SELECT decodeTDigestState(quantileTDigestState(CAST(number, 'UInt32'))) AS state
+FROM numbers(10)
+
+┌─state─────────────────────────────────────────────────────────┐
+│ [(0,1),(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1)] │
+└───────────────────────────────────────────────────────────────┘
+
+SELECT finalizeAggregation(encodeTDigestState(CAST('[(0,1),(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1)]', 'Array(Tuple(Float32, Float32))'))) AS result
+
+┌─result─┐
+│ 4 │
+└────────┘
+```
+
diff --git a/content/en/altinity-kb-functions/kurt_skew_statistics.md b/content/en/altinity-kb-functions/kurt_skew_statistics.md
new file mode 100644
index 0000000000..84cd3b4950
--- /dev/null
+++ b/content/en/altinity-kb-functions/kurt_skew_statistics.md
@@ -0,0 +1,76 @@
+---
+title: "kurt & skew statistical functions in ClickHouse®
+"
+linkTitle: "kurt & skew"
+weight: 100
+description: >-
+ How to make them return the same result like python scipy
+---
+
+```python
+from scipy.stats import skew, kurtosis
+
+# Creating a dataset
+
+dataset = [10,17,71,6,55,38,27,61,48,46,21,38,2,67,35,77,29,31,27,67,81,82,75,81,31,38,68,95,37,34,65,59,81,28,82,80,35,3,97,42,66,28,85,98,45,15,41,61,24,53,97,86,5,65,84,18,9,32,46,52,69,44,78,98,61,64,26,11,3,19,0,90,28,72,47,8,0,74,38,63,88,43,81,61,34,24,37,53,79,72,5,77,58,3,61,56,1,3,5,61]
+
+print(skew(dataset, axis=0, bias=True), skew(dataset))
+
+# -0.05785361619432152 -0.05785361619432152
+```
+```sql
+WITH arrayJoin([10,17,71,6,55,38,27,61,48,46,21,38,2,67,35,77,29,31,27,67,81,82,75,81,31,38,68,95,37,34,65,59,81,28,82,80,35,3,97,42,66,28,85,98,45,15,41,61,24,53,97,86,5,65,84,18,9,32,46,52,69,44,78,98,61,64,26,11,3,19,0,90,28,72,47,8,0,74,38,63,88,43,81,61,34,24,37,53,79,72,5,77,58,3,61,56,1,3,5,61]) AS value
+SELECT skewPop(value) AS ex_1
+
+┌──────────────────ex_1─┐
+│ -0.057853616194321014 │
+└───────────────────────┘
+```
+```python
+print(skew(dataset, bias=False))
+
+# -0.05873838908626328
+```
+```sql
+WITH arrayJoin([10, 17, 71, 6, 55, 38, 27, 61, 48, 46, 21, 38, 2, 67, 35, 77, 29, 31, 27, 67, 81, 82, 75, 81, 31, 38, 68, 95, 37, 34, 65, 59, 81, 28, 82, 80, 35, 3, 97, 42, 66, 28, 85, 98, 45, 15, 41, 61, 24, 53, 97, 86, 5, 65, 84, 18, 9, 32, 46, 52, 69, 44, 78, 98, 61, 64, 26, 11, 3, 19, 0, 90, 28, 72, 47, 8, 0, 74, 38, 63, 88, 43, 81, 61, 34, 24, 37, 53, 79, 72, 5, 77, 58, 3, 61, 56, 1, 3, 5, 61]) AS value
+SELECT
+ skewSamp(value) AS ex_1,
+ (pow(count(), 2) * ex_1) / ((count() - 1) * (count() - 2)) AS G
+
+┌─────────────────ex_1─┬────────────────────G─┐
+│ -0.05698798509149213 │ -0.05873838908626276 │
+└──────────────────────┴──────────────────────┘
+```
+```python
+print(kurtosis(dataset, bias=True, fisher=False), kurtosis(dataset, bias=True, fisher=True), kurtosis(dataset))
+
+# 1.9020275610791184 -1.0979724389208816 -1.0979724389208816
+```
+```sql
+WITH arrayJoin([10, 17, 71, 6, 55, 38, 27, 61, 48, 46, 21, 38, 2, 67, 35, 77, 29, 31, 27, 67, 81, 82, 75, 81, 31, 38, 68, 95, 37, 34, 65, 59, 81, 28, 82, 80, 35, 3, 97, 42, 66, 28, 85, 98, 45, 15, 41, 61, 24, 53, 97, 86, 5, 65, 84, 18, 9, 32, 46, 52, 69, 44, 78, 98, 61, 64, 26, 11, 3, 19, 0, 90, 28, 72, 47, 8, 0, 74, 38, 63, 88, 43, 81, 61, 34, 24, 37, 53, 79, 72, 5, 77, 58, 3, 61, 56, 1, 3, 5, 61]) AS value
+SELECT
+ kurtPop(value) AS pearson,
+ pearson - 3 AS fisher
+
+┌────────────pearson─┬──────────────fisher─┐
+│ 1.9020275610791124 │ -1.0979724389208876 │
+└────────────────────┴─────────────────────┘
+```
+```python
+print(kurtosis(dataset, bias=False))
+
+# -1.0924286152713967
+```
+```sql
+WITH arrayJoin([10, 17, 71, 6, 55, 38, 27, 61, 48, 46, 21, 38, 2, 67, 35, 77, 29, 31, 27, 67, 81, 82, 75, 81, 31, 38, 68, 95, 37, 34, 65, 59, 81, 28, 82, 80, 35, 3, 97, 42, 66, 28, 85, 98, 45, 15, 41, 61, 24, 53, 97, 86, 5, 65, 84, 18, 9, 32, 46, 52, 69, 44, 78, 98, 61, 64, 26, 11, 3, 19, 0, 90, 28, 72, 47, 8, 0, 74, 38, 63, 88, 43, 81, 61, 34, 24, 37, 53, 79, 72, 5, 77, 58, 3, 61, 56, 1, 3, 5, 61]) AS value
+SELECT
+ kurtSamp(value) AS ex_1,
+ (((pow(count(), 2) * (count() + 1)) / (((count() - 1) * (count() - 2)) * (count() - 3))) * ex_1) - ((3 * pow(count() - 1, 2)) / ((count() - 2) * (count() - 3))) AS G
+
+┌──────────────ex_1─┬───────────────────G─┐
+│ 1.864177212613638 │ -1.0924286152714027 │
+└───────────────────┴─────────────────────┘
+```
+
+
+[Google Collab](https://colab.research.google.com/drive/1xoWNi7QAJ9XZtCbmQqJFB8Z_mCreITPW?usp=sharing)
diff --git a/content/en/altinity-kb-integrations/ClickHouse_python_drivers.md b/content/en/altinity-kb-integrations/ClickHouse_python_drivers.md
new file mode 100644
index 0000000000..08c8bc89df
--- /dev/null
+++ b/content/en/altinity-kb-integrations/ClickHouse_python_drivers.md
@@ -0,0 +1,172 @@
+---
+title: "ClickHouse® python drivers"
+linkTitle: "ClickHouse® python drivers"
+weight: 100
+description: >-
+ Python main drivers/clients for ClickHouse®
+---
+
+There are two main python drivers that can be used with ClickHouse. They all have their different set of features and use cases:
+
+### ClickHouse driver AKA [clickhouse-driver](https://clickhouse-driver.readthedocs.io/en/latest/)
+
+The **`clickhouse-driver`** is a Python library used for interacting with ClickHouse. Here's a summary of its features:
+
+1. **Connectivity**: **`clickhouse-driver`** allows Python applications to connect to ClickHouse servers over TCP/IP Native Interface (9000/9440 ports) and also HTTP interface but it is experimental.
+2. **SQL Queries**: It enables executing SQL queries against ClickHouse databases from Python scripts, including data manipulation (insertion, deletion, updating) and data retrieval (select queries).
+3. **Query Parameters**: Supports parameterized queries, which helps in preventing SQL injection attacks and allows for more efficient execution of repeated queries with different parameter values.
+4. **Connection Pooling**: Provides support for connection pooling, which helps manage connections efficiently, especially in high-concurrency applications, by reusing existing connections instead of creating new ones for each query.
+5. **Data Types**: Handles conversion between Python data types and ClickHouse data types, ensuring compatibility and consistency when passing data between Python and ClickHouse.
+6. **Error Handling**: Offers comprehensive error handling mechanisms, including exceptions and error codes, to facilitate graceful error recovery and handling in Python applications.
+7. **Asynchronous Support**: Supports asynchronous execution of queries using `asyncio`, allowing for non-blocking query execution in asynchronous Python applications.
+8. **Customization**: Provides options for customizing connection settings, query execution behavior, and other parameters to suit specific application requirements and performance considerations.
+9. **Compatibility**: Works with various versions of ClickHouse, ensuring compatibility and support for different ClickHouse features and functionalities.
+10. **Documentation and Community**: Offers comprehensive documentation and active community support, including examples, tutorials, and forums, to assist developers in effectively using the library and addressing any issues or questions they may have.
+11. **Supports multiple host** **on connection string** https://clickhouse-driver.readthedocs.io/en/latest/features.html#multiple-hosts
+12. **Connection pooling** (aiohttp)
+
+**Python ecosystem libs/modules:**
+
+- Good Pandas/Numpy support: [https://clickhouse-driver.readthedocs.io/en/latest/features.html#numpy-pandas-support](https://clickhouse-driver.readthedocs.io/en/latest/features.html#numpy-pandas-support)
+- Good SQLALchemy support: [https://pypi.org/project/clickhouse-sqlalchemy/](https://pypi.org/project/clickhouse-sqlalchemy/)
+
+This was the first python driver for ClickHouse. It has a mature codebase. By default ClickHouse drivers uses [synchronous code](https://clickhouse-driver.readthedocs.io/en/latest/quickstart.html#async-and-multithreading). There is a wrapper to convert code to asynchronous, [https://github.com/long2ice/asynch](https://github.com/long2ice/asynch)
+
+Here you can get a basic working example from Altinity repo for ingestion/selection using clickhouse-driver:
+
+[https://github.com/lesandie/clickhouse-tests/blob/main/scripts/test_ch_driver.py](https://github.com/lesandie/clickhouse-tests/blob/main/scripts/test_ch_driver.py)
+
+### ClickHouse-connect AKA [clickhouse-connect](https://clickhouse.com/docs/en/integrations/python)
+
+The ClickHouse Connect Python driver is the ClickHouse, Inc supported-official Python library. Here's a summary of its key features:
+
+1. **Connectivity**: allows Python applications to connect to ClickHouse servers over HTTP Interface (8123/8443 ports).
+2. **Compatibility**: The driver is compatible with Python 3.x versions, ensuring that it can be used with modern Python applications without compatibility issues.
+3. **Performance**: The driver is optimized for performance, allowing for efficient communication with ClickHouse databases to execute queries and retrieve results quickly, which is crucial for applications requiring low latency and high throughput.
+4. **Query Execution**: Developers can use the driver to execute SQL queries against ClickHouse databases, including SELECT, INSERT, UPDATE, DELETE, and other SQL operations, enabling them to perform various data manipulation tasks from Python applications.
+5. **Parameterized Queries**: The driver supports parameterized queries, allowing developers to safely pass parameters to SQL queries to prevent SQL injection attacks and improve query performance by reusing query execution plans.
+6. **Data Type Conversion**: The driver automatically handles data type conversion between Python data types and ClickHouse data types, ensuring seamless integration between Python applications and ClickHouse databases without manual data type conversion.
+7. **Error Handling**: The driver provides robust error handling mechanisms, including exceptions and error codes, to help developers handle errors gracefully and take appropriate actions based on the type of error encountered during query execution.
+8. **Limited Asynchronous Support**: Some implementations of the driver offer asynchronous support, allowing developers to execute queries asynchronously to improve concurrency and scalability in asynchronous Python applications using asynchronous I/O frameworks like `asyncio`.
+9. **Configuration Options**: The driver offers various configuration options, such as connection parameters, authentication methods, and connection pooling settings, allowing developers to customize the driver's behavior to suit their specific requirements and environment.
+10. **Documentation and Community**: Offers comprehensive documentation and active community support, including examples, tutorials, and forums, to assist developers in effectively using the library and addressing any issues or questions they may have. [https://clickhouse.com/docs/en/integrations/language-clients/python/intro/](https://clickhouse.com/docs/en/integrations/language-clients/python/intro/)
+11. **Multiple host on connection string not supported** https://github.com/ClickHouse/clickhouse-connect/issues/74
+12. **Connection pooling** (urllib3)
+
+**Python ecosystem libs/modules:**
+
+- Good Pandas/Numpy support: [https://clickhouse.com/docs/en/integrations/python#consuming-query-results-with-numpy-pandas-or-arrow](https://clickhouse.com/docs/en/integrations/python#consuming-query-results-with-numpy-pandas-or-arrow)
+- Decent SQLAlchemy 1.3 and 1.4 support (limited feature set)
+
+It is the most recent driver with the latest feature set (query context and query streaming …. ), and in recent release [asyncio wrapper](https://github.com/ClickHouse/clickhouse-connect/releases/tag/v0.7.16)
+
+You can check multiple official examples here:
+
+[https://github.com/ClickHouse/clickhouse-connect/tree/457533df05fa685b2a1424359bea5654240ef971/examples](https://github.com/ClickHouse/clickhouse-connect/tree/457533df05fa685b2a1424359bea5654240ef971/examples)
+
+Also some Altinity examples from repo:
+
+[https://github.com/lesandie/clickhouse-tests/blob/main/scripts/test_ch_connect_asyncio_insert.py](https://github.com/lesandie/clickhouse-tests/blob/main/scripts/test_ch_connect_asyncio_insert.py)
+
+You can clone the repo and use the helper files like `DDL.sql` to setup some tests.
+
+
+### Most common use cases:
+
+#### Connection pooler:
+
+- Clickhouse-connect can use a connection pooler (based on urllib3) https://clickhouse.com/docs/en/integrations/python#customizing-the-http-connection-pool
+- Clickhouse-driver you can use **aiohttp** (https://docs.aiohttp.org/en/stable/client_advanced.html#limiting-connection-pool-size)
+
+#### Managing ClickHouse `session_id`:
+
+- clickhouse-driver
+ - Because it is using the Native Interface `session_id` is managed internally by clickhouse, so it is very rare (unless using asyncio) to get:
+
+ `Code: 373. DB::Exception: Session is locked by a concurrent client. (SESSION_IS_LOCKED)` .
+
+- clickhouse-connect: How to use clickhouse-connect in a pythonic way and avoid getting `SESSION_IS_LOCKED` exceptions:
+ - [https://clickhouse.com/docs/en/integrations/python#managing-clickhouse-session-ids](https://clickhouse.com/docs/en/integrations/python#managing-clickhouse-session-ids)
+ - If you want to specify a session_id per query you should be able to use the setting dictionary to pass a `session_id` for each query (note that ClickHouse will automatically generate a `session_id` if none is provided).
+
+ ```python
+ SETTINGS = {"session_id": "dagster-batch" + "-" + f"{time.time()}"}
+ client.query("INSERT INTO table ....", settings=SETTINGS)
+ ```
+
+
+Also in clickhouse documentation some explanation how to set `session_id` with another approach: [https://clickhouse.com/docs/en/integrations/python#managing-clickhouse-session-ids](https://clickhouse.com/docs/en/integrations/python#managing-clickhouse-session-ids)
+
+[ClickHouse Connect Driver API | ClickHouse Docs](https://clickhouse.com/docs/en/integrations/language-clients/python/driver-api#common-method-arguments)
+
+[Best practices with flask · Issue #73 · ClickHouse/clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect/issues/73#issuecomment-1325280242)
+
+#### Asyncio (asynchronous wrappers)
+
+##### clickhouse-connect
+
+New release with [asyncio wrapper for clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect/releases/tag/v0.7.16)
+
+How the wrapper works: https://clickhouse.com/docs/en/integrations/python#asyncclient-wrapper
+
+Wrapper and connection pooler example:
+
+```python
+import clickhouse_connect
+import asyncio
+from clickhouse_connect.driver.httputil import get_pool_manager
+
+async def main():
+ client = await clickhouse_connect.get_async_client(host='localhost', port=8123, pool_mgr=get_pool_manager())
+ for i in range(100):
+ result = await client.query("SELECT name FROM system.databases")
+ print(result.result_rows)
+
+asyncio.run(main())
+```
+
+`clickhouse-connect` code is synchronous by default and running synchronous functions in an async application is a workaround and might not be as efficient as using a library/wrapper designed for asynchronous operations from the ground up.. So you can use the current wrapper or you can use another approach with `asyncio` and `concurrent.futures` and `ThreadpoolExecutor` or `ProcessPoolExecutor`. Python GIL has a mutex over Threads but not to Processes so if you need performance at the cost of using processes instead of threads (not much different for medium workloads) you can use `ProcesspoolExecutor` instead.
+
+Some info about this from the tinybird guys https://www.tinybird.co/blog-posts/killing-the-processpoolexecutor
+
+For clickhouse-connect :
+
+```python
+import asyncio
+from concurrent.futures import ProcessPoolExecutor
+import clickhouse_connect
+
+# Function to execute a query using clickhouse-connect synchronously
+def execute_query_sync(query):
+ client = clickhouse_connect.get_client() # Adjust connection params as needed
+ result = client.query(query)
+ return result
+
+# Asynchronous wrapper function to run the synchronous function in a process pool
+async def execute_query_async(query):
+ loop = asyncio.get_running_loop()
+ # Use ProcessPoolExecutor to execute the synchronous function
+ with ProcessPoolExecutor() as pool:
+ result = await loop.run_in_executor(pool, execute_query_sync, query)
+ return result
+
+async def main():
+ query = "SELECT * FROM your_table LIMIT 10" # Example query
+ result = await execute_query_async(query)
+ print(result)
+
+# Run the async main function
+if __name__ == '__main__':
+ asyncio.run(main())
+```
+##### Clickhouse-driver
+
+`clickhouse-driver` code is also synchronous and suffers the same problem as `clickhouse-connect` https://clickhouse-driver.readthedocs.io/en/latest/quickstart.html#async-and-multithreading
+
+So to use asynchronous approach it is recommended to use a connection pool and some asyncio wrapper that can hide the complexity of using the `ThreadPoolExecutor/ProcessPoolExecutor`
+
+- To begin testing such environment [aiohttp](https://docs.aiohttp.org/) is a good approach. Here an example: https://github.com/lesandie/clickhouse-tests/blob/main/scripts/test_aiohttp_inserts.py
+ This will use simply requests module and aiohttp (you can tune the connection pooler https://docs.aiohttp.org/en/stable/client_advanced.html#limiting-connection-pool-size)
+
+- Also `aiochclient` is another good wrapper https://github.com/maximdanilchenko/aiochclient for the HTTP interface
+- For the native interface you can try https://github.com/long2ice/asynch, `asynch` is an asyncio ClickHouse Python Driver with native (TCP) interface support, which reuse most of [clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) and comply with [PEP249](https://www.python.org/dev/peps/pep-0249/).
diff --git a/content/en/altinity-kb-integrations/Spark.md b/content/en/altinity-kb-integrations/Spark.md
index e37f065f58..b1813c40cd 100644
--- a/content/en/altinity-kb-integrations/Spark.md
+++ b/content/en/altinity-kb-integrations/Spark.md
@@ -1,13 +1,9 @@
---
-title: "ClickHouse + Spark"
+title: "ClickHouse® + Spark"
linkTitle: "Spark"
weight: 100
-description: >-
- Spark
---
-## ClickHouse + Spark
-
### jdbc
The trivial & natural way to talk to ClickHouse from Spark is using jdbc. There are 2 jdbc drivers:
@@ -16,7 +12,7 @@ The trivial & natural way to talk to ClickHouse from Spark is using jdbc. There
ClickHouse-Native-JDBC has some hints about integration with Spark even in the main README file.
-'Official' driver does support some conversion of complex data types (Roarring bitmaps) for Spark-Clickhouse integration: https://github.com/ClickHouse/clickhouse-jdbc/pull/596
+'Official' driver does support some conversion of complex data types (Roaring bitmaps) for Spark-ClickHouse integration: https://github.com/ClickHouse/clickhouse-jdbc/pull/596
But proper partitioning of the data (to spark partitions) may be tricky with jdbc.
@@ -54,21 +50,18 @@ Arrays, Higher-order functions, machine learning, integration with lot of differ
## More info + some unordered links (mostly in Chinese / Russian)
-* Spark + ClickHouse: not a fight, but a symbiosis https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup28/spark_and_clickhouse.pdf (russian)
-* Using a bunch of ClickHouse and Spark in MFI Soft https://www.youtube.com/watch?v=ID8eTnmag0s (russian)
-* Spark read and write ClickHouse https://yerias.github.io/2020/12/08/clickhouse/9/#Jdbc%E6%93%8D%E4%BD%9Cclickhouse
-* Spark reads and writes ClickHouse through jdbc https://blog.katastros.com/a?ID=01800-e40e1b3c-5fa4-4ea0-a3a8-f5e89ef0ce14
-* Spark JDBC write clickhouse operation summary https://www.jianshu.com/p/43f78c8a025b?hmsr=toutiao.io&utm_campaign=toutiao.io&utm_medium=toutiao.io&utm_source=toutiao.io https://toutiao.io/posts/m63yw89/preview
-* Spark-sql is based on Clickhouse's DataSourceV2 data source extension (russian)
+* Spark + ClickHouse: not a fight, but a symbiosis (Russian) https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup28/spark_and_clickhouse.pdf (russian)
+* Using a bunch of ClickHouse and Spark in MFI Soft (Russian) https://www.youtube.com/watch?v=ID8eTnmag0s (russian)
+* Spark read and write ClickHouse (Chinese: Spark读写ClickHouse) https://yerias.github.io/2020/12/08/clickhouse/9/#Jdbc%E6%93%8D%E4%BD%9Cclickhouse
+* Spark JDBC write ClickHouse operation summary (Chinese: Spark JDBC 写 ClickHouse 操作总结) https://www.jianshu.com/p/43f78c8a025b?hmsr=toutiao.io&utm_campaign=toutiao.io&utm_medium=toutiao.io&utm_source=toutiao.io
+* Spark-sql is based on ClickHouse's DataSourceV2 data source extension (Chinese: spark-sql基于ClickHouse的DataSourceV2数据源扩展)
https://www.cnblogs.com/mengyao/p/4689866.html
-* Alibaba integration instructions https://www.alibabacloud.com/help/doc-detail/191192.htm
-* Tencent integration instructions https://intl.cloud.tencent.com/document/product/1026/35884
-* Yandex DataProc demo: loading files from S3 to ClickHouse with Spark https://www.youtube.com/watch?v=N3bZW0_rRzI
-* Clickhouse official documentation_Spark JDBC writes some pits of ClickHouse https://blog.csdn.net/weixin_39615984/article/details/111206050
-* ClickHouse data import (Flink, Spark, Kafka, MySQL, Hive) https://zhuanlan.zhihu.com/p/299094269
-* Baifendian Big Data Technical Team: Practice of ClickHouse data synchronization solutionbased on multiple Spark tasks. https://www.6aiq.com/article/1635461873075
-* SPARK-CLICKHOUSE-ES REAL-TIME PROJECT EIGHTH DAY-PRECISE ONE-TIME CONSUMPTION SAVE OFFSET. https://www.freesion.com/article/71421322524/
-* Still struggling with real-time data warehouse selection, Spark + ClickHouse makes yoamazing! https://dbaplus.cn/news-73-3806-1.html
-* HDFS+ClickHouse+Spark: A lightweight big data analysis system from 0 to 1. https://juejin.cn/post/6850418114962653198
-* ClickHouse Clustering for Spark Developer http://blog.madhukaraphatak.com/clickouse-clustering-spark-developer/
-* «Иногда приходится заглядывать в код Spark»: Александр Морозов (SEMrush) об использовании Scala, Spark и ClickHouse. https://habr.com/ru/company/jugru/blog/341288/
+* Alibaba integration instructions (English) https://www.alibabacloud.com/help/doc-detail/191192.htm
+* Tencent integration instructions (English) https://intl.cloud.tencent.com/document/product/1026/35884
+* Yandex DataProc demo: loading files from S3 to ClickHouse with Spark (Russian) https://www.youtube.com/watch?v=N3bZW0_rRzI
+* ClickHouse official documentation_Spark JDBC writes some pits of ClickHouse (Chinese: ClickHouse官方文档_Spark JDBC写ClickHouse的一些坑) https://blog.csdn.net/weixin_39615984/article/details/111206050
+* ClickHouse data import: Flink, Spark, Kafka, MySQL, Hive (Chinese: 篇五|ClickHouse数据导入 Flink、Spark、Kafka、MySQL、Hive) https://zhuanlan.zhihu.com/p/299094269
+* SPARK-CLICKHOUSE-ES REAL-TIME PROJECT EIGHTH DAY-PRECISE ONE-TIME CONSUMPTION SAVE OFFSET. (Chinese: SPARK-CLICKHOUSE-ES实时项目第八天-精确一次性消费保存偏移量) https://www.freesion.com/article/71421322524/
+* HDFS+ClickHouse+Spark: A lightweight big data analysis system from 0 to 1. (Chinese: HDFS+ClickHouse+Spark：从0到1实现一款轻量级大数据分析系统) https://juejin.cn/post/6850418114962653198
+* ClickHouse Clustering for Spark Developer (English) http://blog.madhukaraphatak.com/clickouse-clustering-spark-developer/
+* «Иногда приходится заглядывать в код Spark»: Александр Морозов (SEMrush) об использовании Scala, Spark и ClickHouse. (Russian) https://habr.com/ru/company/jugru/blog/341288/
diff --git a/content/en/altinity-kb-integrations/_index.md b/content/en/altinity-kb-integrations/_index.md
index dc1cd483d1..6951848f56 100644
--- a/content/en/altinity-kb-integrations/_index.md
+++ b/content/en/altinity-kb-integrations/_index.md
@@ -6,6 +6,6 @@ keywords:
- clickhouse bi
- clickhouse kafka
description: >
- Learn how you can integrate cloud services, BI tools, kafka, MySQL, Spark, MindsDB, and more with ClickHouse.
+ Learn how you can integrate cloud services, BI tools, kafka, MySQL, Spark, MindsDB, and more with ClickHouse®
weight: 4
---
diff --git a/content/en/altinity-kb-integrations/altinity-cloud/_index.md b/content/en/altinity-kb-integrations/altinity-cloud/_index.md
index fc03237ffc..94f729920e 100644
--- a/content/en/altinity-kb-integrations/altinity-cloud/_index.md
+++ b/content/en/altinity-kb-integrations/altinity-cloud/_index.md
@@ -1,7 +1,81 @@
---
-title: "Cloud Services"
-linkTitle: "Cloud Services"
+title: "Altinity Cloud Access Management"
+linkTitle: "Altinity Cloud Access Management"
description: >
- Tips and tricks for using ClickHouse with different cloud services.
-weight: 4
+ Enabling access_management for Altinity.Cloud databases.
+weight: 5
+alias: /altinity-kb-integrations/altinity-cloud
---
+Organizations that want to enable administrative users in their Altinity.Cloud ClickHouse® servers can do so by enabling `access_management` manually. This allows for administrative users to be created on the specific ClickHouse Cluster.
+
+{{% alert title="WARNING" color="warning" %}}
+Modifying the ClickHouse cluster settings manually can lead to the cluster not loading or other issues. Change settings only with full consultation with an Altinity.Cloud support team member, and be ready to remove settings if they cause any disruption of service.
+{{% /alert %}}
+
+To add the `access_management` setting to an Altinity.Cloud ClickHouse Cluster:
+
+1. Log into your Altinity.Cloud account.
+1. For the cluster to modify, select **Configure -> Settings**.
+
+ {{< figure src="/assets/altinity-cloud-cluster-settings-configure.png" width="400" title="Cluster setting configure" >}}
+
+1. From the Settings page, select **+ADD SETTING**.
+
+ {{< figure src="/assets/altinity-cloud-cluster-add-setting.png" title="Add cluster setting" >}}
+
+1. Set the following options:
+ 1. **Setting Type**: Select **users.d file**.
+ 1. **Filename**: `access_management.xml`
+ 1. **Contents**: Enter the following to allow the `clickhouse_operator` that controls the cluster through the `clickhouse-operator` the ability to set administrative options:
+
+ ```xml
+
+
+
+ 1
+
+
+ 1
+
+
+
+ ```
+
+ access_management=1 means that users `admin`, `clickhouse_operator` are able to create users and grant them privileges using SQL.
+
+1. Select **OK**. The cluster will restart, and users can now be created in the cluster that can be granted administrative access.
+
+1. If you are running ClickHouse 21.9 and above you can enable storing access management in ZooKeeper. in this case it will be automatically propagated to the cluster. This requires yet another configuration file:
+ 1. **Setting Type**: Select **config.d file**
+ 2. **Filename**: `user_directories.xml`
+ 3. **Contents**:
+
+ ```xml
+
+
+
+ /etc/clickhouse-server/users.xml
+
+
+ /clickhouse/access/
+
+
+ /var/lib/clickhouse/access/
+
+
+
+ ```
+
+[//]: # (---)
+
+[//]: # (title: "Cloud Services")
+
+[//]: # (linkTitle: "Cloud Services")
+
+[//]: # (description: >)
+
+[//]: # ( Tips and tricks for using ClickHouse® with different cloud services.)
+
+[//]: # (weight: 4)
+
+[//]: # (---)
diff --git a/content/en/altinity-kb-integrations/altinity-cloud/altinity-cloud-access-management.md b/content/en/altinity-kb-integrations/altinity-cloud/altinity-cloud-access-management.md
index 9803007c8f..b34ea1e85c 100644
--- a/content/en/altinity-kb-integrations/altinity-cloud/altinity-cloud-access-management.md
+++ b/content/en/altinity-kb-integrations/altinity-cloud/altinity-cloud-access-management.md
@@ -4,8 +4,10 @@ linkTitle: "Altinity Cloud Access Management"
description: >
Enabling access_management for Altinity.Cloud databases.
weight: 5
+alias: /altinity-kb-integrations/altinity-cloud
+draft: true
---
-Organizations that want to enable administrative users in their Altinity.Cloud ClickHouse servers can do so by enabling `access_management` manually. This allows for administrative users to be created on the specific ClickHouse Cluster.
+Organizations that want to enable administrative users in their Altinity.Cloud ClickHouse® servers can do so by enabling `access_management` manually. This allows for administrative users to be created on the specific ClickHouse Cluster.
{{% alert title="WARNING" color="warning" %}}
Modifying the ClickHouse cluster settings manually can lead to the cluster not loading or other issues. Change settings only with full consultation with an Altinity.Cloud support team member, and be ready to remove settings if they cause any disruption of service.
@@ -28,7 +30,7 @@ To add the `access_management` setting to an Altinity.Cloud ClickHouse Cluster:
1. **Contents**: Enter the following to allow the `clickhouse_operator` that controls the cluster through the `clickhouse-operator` the ability to set administrative options:
```xml
-
+ 1
@@ -37,7 +39,7 @@ To add the `access_management` setting to an Altinity.Cloud ClickHouse Cluster:
1
-
+
```
access_management=1 means that users `admin`, `clickhouse_operator` are able to create users and grant them privileges using SQL.
@@ -50,7 +52,7 @@ To add the `access_management` setting to an Altinity.Cloud ClickHouse Cluster:
3. **Contents**:
```xml
-
+ /etc/clickhouse-server/users.xml
@@ -58,6 +60,9 @@ To add the `access_management` setting to an Altinity.Cloud ClickHouse Cluster:
/clickhouse/access/
+
+ /var/lib/clickhouse/access/
+
-
+
```
diff --git a/content/en/altinity-kb-integrations/altinity-kb-google-s3-gcs.md b/content/en/altinity-kb-integrations/altinity-kb-google-s3-gcs.md
index b0fa975f78..eb1f5b99c1 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-google-s3-gcs.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-google-s3-gcs.md
@@ -1,9 +1,8 @@
---
title: "Google S3 (GCS)"
linkTitle: "Google S3 (GCS)"
-description: >
- "Google S3 GCS"
---
+
GCS with the table function - seems to work correctly for simple scenarios.
Essentially you can follow the steps from the [Migrating from Amazon S3 to Cloud Storage](https://cloud.google.com/storage/docs/aws-simple-migration).
@@ -11,8 +10,5 @@ Essentially you can follow the steps from the [Migrating from Amazon S3 to Cloud
1. Set up a GCS bucket.
2. This bucket must be set as part of the default project for the account. This configuration can be found in settings -> interoperability.
3. Generate a HMAC key for the account, can be done in settings -> interoperability, in the section for user account access keys.
-4. In ClickHouse, replace the S3 bucket endpoint with the GCS bucket endpoint This must be done with the path-style GCS endpoint: `https://storage.googleapis.com/BUCKET_NAME/OBJECT_NAME`.
+4. In ClickHouse®, replace the S3 bucket endpoint with the GCS bucket endpoint This must be done with the path-style GCS endpoint: `https://storage.googleapis.com/BUCKET_NAME/OBJECT_NAME`.
5. Replace the aws access key id and aws secret access key with the corresponding parts of the HMAC key.
-
-
-s3 Disk on the top of GCS and writing to GSC may be NOT working because GCS don't support some of bulk S3 API calls, see https://github.com/ClickHouse/ClickHouse/issues/24246
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/_index.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/_index.md
new file mode 100644
index 0000000000..da9b779694
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/_index.md
@@ -0,0 +1,7 @@
+---
+title: "Fundamentals"
+linkTitle: "Fundamentals"
+description: >
+ Core Kafka engine behavior and query semantics in ClickHouse.
+weight: 10
+---
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-adjusting-librdkafka-settings.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-adjusting-librdkafka-settings.md
new file mode 100644
index 0000000000..95b26245a3
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-adjusting-librdkafka-settings.md
@@ -0,0 +1,161 @@
+---
+title: "Adjusting librdkafka settings"
+linkTitle: "Adjusting librdkafka settings"
+description: >
+ Adjusting librdkafka settings
+---
+* To set rdkafka options - add to `` section in `config.xml` or preferably use a separate file in `config.d/`:
+ * [https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)
+
+Some random example using SSL certificates to authenticate:
+
+```xml
+
+
+ 60000
+ 60000
+ 10000
+ 5000
+ 60000
+ 20000
+ 500
+ 20971520
+ all
+ SSL
+ /etc/clickhouse-server/ssl/kafka-ca-qa.crt
+ /etc/clickhouse-server/ssl/client_clickhouse_client.pem
+ /etc/clickhouse-server/ssl/client_clickhouse_client.key
+ pass
+
+
+```
+
+## Authentication / connectivity
+
+Sometimes the consumer group needs to be explicitly allowed in the broker UI config.
+
+Use general Kafka/librdkafka settings from this page first, then apply provider-specific options from [Config by provider](./config-by-provider/).
+
+### Kerberos
+
+* [https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/\#kafka-kerberos-support](https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/#kafka-kerberos-support)
+* [https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_storage_kerberized_kafka/configs/kafka.xml](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_storage_kerberized_kafka/configs/kafka.xml)
+
+```xml
+
+
+ SASL_PLAINTEXT
+ /home/kafkauser/kafkauser.keytab
+ kafkauser/kafkahost@EXAMPLE.COM
+
+```
+
+## How to test connection settings
+
+Use kafkacat utility - it internally uses same library to access Kafla as ClickHouse itself and allows easily to test different settings.
+
+```bash
+kafkacat -b my_broker:9092 -C -o -10 -t my_topic \ (Google cloud and on-prem use 9092 port)
+ -X security.protocol=SASL_SSL \
+ -X sasl.mechanisms=PLAIN \
+ -X sasl.username=uerName \
+ -X sasl.password=Password
+
+```
+
+## Different configurations for different tables?
+
+> Is there some more documentation how to use this multiconfiguration for Kafka ?
+
+The whole logic is here:
+https://github.com/ClickHouse/ClickHouse/blob/da4856a2be035260708fe2ba3ffb9e437d9b7fef/src/Storages/Kafka/StorageKafka.cpp#L466-L475
+
+So it load the main config first, after that it load (with overwrites) the configs for all topics, **listed in `kafka_topic_list` of the table**.
+
+Also since v21.12 it's possible to use more straightforward way using named_collections:
+https://github.com/ClickHouse/ClickHouse/pull/31691
+
+So you can write a config file something like this:
+
+```xml
+
+
+
+ kafka1:19092
+ conf
+ conf
+
+
+
+
+
+
+
+
+ ...
+ foo.bar
+ foo.bar.group
+
+ ...
+ ...
+ ...
+ ...
+ smallest
+ https
+ probe
+
+
+
+
+
+```
+
+And after execute:
+
+```sql
+CREATE TABLE test.kafka (key UInt64, value UInt64) ENGINE = Kafka(kafka_preset1, kafka_format='CSV');
+```
+
+The same named collections can be created with SQL from v24.2+:
+
+```sql
+CREATE NAMED COLLECTION kafka_preset1 AS
+ kafka_broker_list = 'kafka1:19092',
+ kafka_topic_list = 'conf',
+ kafka_group_name = 'conf';
+```
+
+```sql
+CREATE NAMED COLLECTION kafka_preset2 AS
+ kafka_broker_list = '...',
+ kafka_topic_list = 'foo.bar',
+ kafka_group_name = 'foo.bar.group',
+ kafka.security_protocol = 'SASL_SSL',
+ kafka.sasl_mechanism = 'PLAIN',
+ kafka.sasl_username = '...',
+ kafka.sasl_password = '...',
+ kafka.auto_offset_reset = 'smallest',
+ kafka.ssl_endpoint_identification_algorithm = 'https',
+ kafka.ssl_ca_location = 'probe';
+```
+
+You can verify SQL-created named collections via:
+
+```sql
+SELECT
+ name,
+ source,
+ create_query
+FROM system.named_collections
+WHERE name IN ('kafka_preset1', 'kafka_preset2');
+```
+
+and remove them with:
+
+```sql
+DROP NAMED COLLECTION kafka_preset1;
+DROP NAMED COLLECTION kafka_preset2;
+```
+
+The same fragment of code in newer versions:
+- https://github.com/ClickHouse/ClickHouse/blob/d19e24f530c30f002488bc136da78f5fb55aedab/src/Storages/Kafka/StorageKafka.cpp#L474-L496
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-main-parsing-loop.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-main-parsing-loop.md
similarity index 67%
rename from content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-main-parsing-loop.md
rename to content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-main-parsing-loop.md
index 2b9bd975f9..cde9849208 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-main-parsing-loop.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-main-parsing-loop.md
@@ -4,7 +4,7 @@ linkTitle: "Kafka main parsing loop"
description: >
Kafka main parsing loop
---
-One of the threads from scheduled_pool (pre 20.9) / `background_message_broker_schedule_pool` (after 20.9) do that in infinite loop:
+One of the threads from scheduled_pool (pre ClickHouse® 20.9) / `background_message_broker_schedule_pool` (after 20.9) do that in infinite loop:
1. Batch poll (time limit: `kafka_poll_timeout_ms` 500ms, messages limit: `kafka_poll_max_batch_size` 65536)
2. Parse messages.
@@ -25,9 +25,14 @@ These usually should not be adjusted:
You may want to adjust those depending on your scenario:
-* `kafka_flush_interval_ms` = stream_poll_timeout_ms (7500ms)
+* `kafka_flush_interval_ms` = stream_flush_interval_ms (7500ms)
* `kafka_max_block_size` = max_insert_block_size / kafka_num_consumers (for the single consumer: 1048576)
## See also
[https://github.com/ClickHouse/ClickHouse/pull/11388](https://github.com/ClickHouse/ClickHouse/pull/11388)
+
+## Disable at-least-once delivery
+
+`kafka_commit_every_batch` = 1 will change the loop logic mentioned above. Consumed batch committed to the Kafka and the block of rows send to Materialized Views only after that. It could be resembled as at-most-once delivery mode as prevent duplicate creation but allow loss of data in case of failures.
+
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-virtual-columns.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-virtual-columns.md
new file mode 100644
index 0000000000..86e9519fd3
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-kafka-virtual-columns.md
@@ -0,0 +1,40 @@
+---
+title: "Kafka engine Virtual columns"
+linkTitle: "Kafka virtual columns"
+weight: 100
+description: >-
+ Kafka virtual columns
+---
+
+## Kafka engine virtual columns (built-in)
+
+[From the Kafka engine docs](https://clickhouse.com/docs/engines/table-engines/integrations/kafka?utm_source=chatgpt.com#virtual-columns), the supported virtual columns are:
+
+- `_topic` — Kafka topic (LowCardinality(String))
+
+- `_key` — message key (String)
+
+- `_offset` — message offset (UInt64)
+
+- `_timestamp` — message timestamp (Nullable(DateTime))
+
+- `_timestamp_ms` — timestamp with millisecond precision (Nullable(DateTime64(3)))
+
+- `_partition` — partition (UInt64)
+
+- `_headers.name` — header keys (Array(String))
+
+- `_headers.value` — header values (Array(String))
+
+Extra virtual columns when you enable parse-error streaming:
+
+If you set `kafka_handle_error_mode='stream'`, ClickHouse adds:
+
+- `_raw_message` — the raw message that failed to parse (String)
+
+- `_error` — the exception message from parsing failure (String)
+
+Note: `_raw_message` and `_error` are populated only when parsing fails; otherwise they’re empty.
+
+We can use these columns in a materialized view like this for example:
+
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-selects-from-engine-kafka.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-selects-from-engine-kafka.md
similarity index 100%
rename from content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-selects-from-engine-kafka.md
rename to content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/altinity-kb-selects-from-engine-kafka.md
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/config-by-provider.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/config-by-provider.md
new file mode 100644
index 0000000000..e9beb7758b
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/01-fundamentals/config-by-provider.md
@@ -0,0 +1,97 @@
+---
+title: "Config by provider"
+description: >
+ Kafka engine configuration examples grouped by managed Kafka provider.
+weight: 15
+---
+
+Sometimes the consumer group needs to be explicitly allowed in the broker UI config.
+
+Read [Adjusting librdkafka settings](./altinity-kb-adjusting-librdkafka-settings/) first, then apply the provider-specific settings below.
+
+### Amazon MSK | SASL/SCRAM
+
+```xml
+
+
+ sasl_ssl
+
+
+ root
+ toor
+
+
+```
+- [Broker ports detail](https://docs.aws.amazon.com/msk/latest/developerguide/port-info.html)
+- [Read here more](https://leftjoin.ru/blog/data-engineering/clickhouse-as-a-consumer-to-amazon-msk/) (Russian language)
+
+
+### on-prem / self-hosted Kafka broker
+
+```xml
+
+
+ sasl_ssl
+ SCRAM-SHA-512
+ root
+ toor
+
+ /path/to/cert/fullchain.pem
+
+
+```
+
+
+### Inline Kafka certs
+
+To connect to some Kafka cloud services you may need to use certificates.
+
+If needed they can be converted to pem format and inlined into ClickHouse® config.xml
+Example:
+
+```xml
+
+
+
+
+```
+
+See
+
+- [https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka](https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka)
+
+- [https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files](https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files)
+
+### Azure Event Hub
+
+See [https://github.com/ClickHouse/ClickHouse/issues/12609](https://github.com/ClickHouse/ClickHouse/issues/12609)
+
+### Confluent Cloud / Google Cloud
+
+```xml
+
+
+ smallest
+ SASL_SSL
+
+
+ PLAIN
+ username
+ password
+
+
+
+
+```
+- [https://docs.confluent.io/cloud/current/client-apps/config-client.html](https://docs.confluent.io/cloud/current/client-apps/config-client.html)
+- [https://cloud.google.com/managed-service-for-apache-kafka/docs/authentication-kafka](https://cloud.google.com/managed-service-for-apache-kafka/docs/authentication-kafka)
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/_index.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/_index.md
new file mode 100644
index 0000000000..e4d0a4861a
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/_index.md
@@ -0,0 +1,7 @@
+---
+title: "Consumption Patterns"
+linkTitle: "Consumption Patterns"
+description: >
+ Message consumption models, replay patterns, and delivery semantics.
+weight: 20
+---
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-exactly-once-semantics.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-exactly-once-semantics.md
similarity index 92%
rename from content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-exactly-once-semantics.md
rename to content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-exactly-once-semantics.md
index 3432db9fd5..fa54523134 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-exactly-once-semantics.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-exactly-once-semantics.md
@@ -4,7 +4,7 @@ linkTitle: "Exactly once semantics"
description: >
Exactly once semantics
---
-EOS consumer (isolation.level=read_committed) is enabled by default since librdkafka 1.2.0, so for ClickHouse - since 20.2
+EOS consumer (isolation.level=read_committed) is enabled by default since librdkafka 1.2.0, so for ClickHouse® - since 20.2
See:
@@ -18,6 +18,6 @@ We need to have something like transactions on ClickHouse side to be able to avo
## block-aggregator by eBay
-Block Aggregator is a data loader that subscribes to Kafka topics, aggregates the Kafka messages into blocks that follow the Clickhouse’s table schemas, and then inserts the blocks into ClickHouse. Block Aggregator provides exactly-once delivery guarantee to load data from Kafka to ClickHouse. Block Aggregator utilizes Kafka’s metadata to keep track of blocks that are intended to send to ClickHouse, and later uses this metadata information to deterministically re-produce ClickHouse blocks for re-tries in case of failures. The identical blocks are guaranteed to be deduplicated by ClickHouse.
+Block Aggregator is a data loader that subscribes to Kafka topics, aggregates the Kafka messages into blocks that follow the ClickHouse’s table schemas, and then inserts the blocks into ClickHouse. Block Aggregator provides exactly-once delivery guarantee to load data from Kafka to ClickHouse. Block Aggregator utilizes Kafka’s metadata to keep track of blocks that are intended to send to ClickHouse, and later uses this metadata information to deterministically re-produce ClickHouse blocks for re-tries in case of failures. The identical blocks are guaranteed to be deduplicated by ClickHouse.
[eBay/block-aggregator](https://github.com/eBay/block-aggregator)
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-mv-consuming.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-mv-consuming.md
new file mode 100644
index 0000000000..50c4b2ca6f
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-mv-consuming.md
@@ -0,0 +1,120 @@
+---
+title: "Multiple MVs attached to Kafka table"
+linkTitle: "Multiple MVs attached to Kafka table"
+description: >
+ How Multiple MVs attached to Kafka table consume and how they are affected by kafka_num_consumers/kafka_thread_per_consumer
+---
+
+Kafka Consumer is a thread inside the Kafka Engine table that is visible by Kafka monitoring tools like kafka-consumer-groups and in Clickhouse in system.kafka_consumers table.
+
+Having multiple consumers increases ingesting parallelism and can significantly speed up event processing. However, it comes with a trade-off: it's a CPU-intensive task, especially under high event load and/or complicated parsing of incoming data. Therefore, it's crucial to create as many consumers as you really need and ensure you have enough CPU cores to handle them. We don’t recommend creating too many Kafka Engines per server because it could lead to uncontrolled CPU usage in situations like bulk data upload or catching up a huge kafka lag due to excessive parallelism of the ingesting process.
+
+## kafka_thread_per_consumer meaning
+
+Consider a basic pipeline depicted as a Kafka table with 2 MVs attached. The Kafka broker has 2 topics and 4 partitions.
+
+### kafka_thread_per_consumer = 0
+
+Kafka engine table will act as 2 consumers, but only 1 insert thread for both of them. It is important to note that the topic needs to have as many partitions as consumers. For this scenario, we use these settings:
+
+```
+kafka_num_consumers = 2
+kafka_thread_per_consumer = 0
+```
+
+The same Kafka engine will create 2 streams, 1 for each consumer, and will join them in a union stream. And it will use 1 thread for inserting `[ 2385 ]`
+This is how we can see it in the logs:
+
+```log
+2022.11.09 17:49:34.282077 [ 2385 ] {} StorageKafka (kafka_table): Started streaming to 2 attached views
+```
+
+* How ClickHouse® calculates the number of threads depending on the `thread_per_consumer` setting:
+
+ ```c++
+ auto stream_count = thread_per_consumer ? 1 : num_created_consumers;
+ sources.reserve(stream_count);
+ pipes.reserve(stream_count);
+ for (size_t i = 0; i < stream_count; ++i)
+ {
+ ......
+ }
+ ```
+
+Details:
+
+https://github.com/ClickHouse/ClickHouse/blob/1b49463bd297ade7472abffbc931c4bb9bf213d0/src/Storages/Kafka/StorageKafka.cpp#L834
+
+
+Also, a detailed graph of the pipeline:
+
+
+
+With this approach, even if the number of consumers increased, the Kafka engine will still use only 1 thread to flush. The consuming/processing rate will probably increase a bit, but not linearly. For example, 5 consumers will not consume 5 times faster. Also, a good property of this approach is the `linearization` of INSERTS, which means that the order of the inserts is preserved and sequential. This option is good for small/medium Kafka topics.
+
+
+### kafka_thread_per_consumer = 1
+
+Kafka engine table will act as 2 consumers and 1 thread per consumer. For this scenario, we use these settings:
+
+```
+kafka_num_consumers = 2
+kafka_thread_per_consumer = 1
+```
+
+Here, the pipeline works like this:
+
+
+
+
+With this approach, the number of consumers remains the same, but each consumer will use their own insert/flush thread, and the consuming/processing rate should increase.
+
+## Background Pool
+
+In Clickhouse there is a special thread pool for background processes, such as streaming engines. Its size is controlled by the background_message_broker_schedule_pool_size setting and is 16 by default. If you exceed this limit across all tables on the server, you’ll likely encounter continuous Kafka rebalances, which will slow down processing considerably. For a server with a lot of CPU cores, you can increase that limit to a higher value, like 20 or even 40. `background_message_broker_schedule_pool_size` = 20 allows you to create 5 Kafka Engine tables with 4 consumers each of them has its own insert thread. This option is good for large Kafka topics with millions of messages per second.
+
+
+## Multiple Materialized Views
+
+Attaching multiple Materialized Views (MVs) to a Kafka Engine table can be used when you need to apply different transformations to the same topic and store the resulting data in different tables.
+
+(This approach also applies to the other streaming engines - RabbitMQ, s3queue, etc).
+
+All streaming engines begin processing data (reading from the source and producing insert blocks) only after at least one Materialized View is attached to the engine. Multiple Materialized Views can be connected to distribute data across various tables with different transformations. But how does it work when the server starts?
+
+Once the first Materialized View (MV) is loaded, started, and attached to the Kafka/s3queue table, data consumption begins immediately—data is read from the source, pushed to the destination, and the pointers advance to the next position. However, any other MVs that haven't started yet will miss the data consumed by the first MV, leading to some data loss.
+
+This issue worsens with asynchronous table loading. Tables are only loaded upon first access, and the loading process takes time. When multiple MVs direct the data stream to different tables, some tables might be ready sooner than others. As soon as the first table becomes ready, data consumption starts, and any tables still loading will miss the data consumed during that interval, resulting in further data loss for those tables.
+
+
+That means when you make a design with Multiple MVs `async_load_databases` should be switched off:
+
+```sql
+false
+```
+
+Also, you have to prevent starting to consume until all MVs are loaded and started. For that, you can add an additional Null table to the MV pipeline, so the Kafka table will pass the block to a single Null table first, and only then many MVs start their own transformations to many dest tables:
+
+ KafkaTable → dummy_MV -> NullTable -> [MV1, MV2, ….] → [Table1, Table2, …]
+
+```sql
+create table NullTable Engine=Null as KafkaTable;
+create materialized view dummy_MV to NullTable
+select * from KafkaTable
+--WHERE NOT ignore(throwIf(if((uptime() < 120), 1 , 0)))
+WHERE NOT ignore(throwIf(if((uptime() < 120), 1 + sleep(3), 0)))
+```
+
+120 seconds should be enough for loading all MVs.
+
+Using an intermediate Null table is also preferable because it's easier to make any changes with MVs:
+
+- drop the dummy_MV to stop consuming
+- make any changes to transforming MVs by drop/recreate
+- create dummy_MV again to resume consuming
+
+The fix for correctly starting multiple MVs will be available from 25.5 version - https://github.com/ClickHouse/ClickHouse/pull/72123
+
+
+
+
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-parallel-consuming.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-parallel-consuming.md
similarity index 64%
rename from content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-parallel-consuming.md
rename to content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-parallel-consuming.md
index 53c06a8972..4f7b62d5d3 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-kafka-parallel-consuming.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-kafka-parallel-consuming.md
@@ -4,7 +4,7 @@ linkTitle: "Kafka parallel consuming"
description: >
Kafka parallel consuming
---
-For very large topics when you need more parallelism (especially on the insert side) you may use several tables with the same pipeline (pre 20.9) or enable `kafka_thread_per_consumer` (after 20.9).
+For very large topics when you need more parallelism (especially on the insert side) you may use several tables with the same pipeline (pre ClickHouse® 20.9) or enable `kafka_thread_per_consumer` (after 20.9).
```ini
kafka_num_consumers = N,
@@ -15,5 +15,7 @@ Notes:
* the inserts will happen in parallel (without that setting inserts happen linearly)
* enough partitions are needed.
+* `kafka_num_consumers` is limited by number of physical cores (half of vCPUs). `kafka_disable_num_consumers_limit` can be used to override the limit.
+* `background_message_broker_schedule_pool_size` is 16 by default, you may need to increase if using more than 16 consumers
Before increasing `kafka_num_consumers` with keeping `kafka_thread_per_consumer=0` may improve consumption & parsing speed, but flushing & committing still happens by a single thread there (so inserts are linear).
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-rewind-fast-forward-replay.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-rewind-fast-forward-replay.md
new file mode 100644
index 0000000000..cc655531d7
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/02-consumption-patterns/altinity-kb-rewind-fast-forward-replay.md
@@ -0,0 +1,34 @@
+---
+title: "Rewind / fast-forward / replay"
+linkTitle: "Rewind / fast-forward / replay"
+description: >
+ Rewind / fast-forward / replay
+---
+* Step 1: Detach Kafka tables in ClickHouse®
+ ```
+ DETACH TABLE db.kafka_table_name ON CLUSTER '{cluster}';
+ ```
+* Step 2: `kafka-consumer-groups.sh --bootstrap-server kafka:9092 --topic topic:0,1,2 --group id1 --reset-offsets --to-latest --execute`
+ * More samples: [https://gist.github.com/filimonov/1646259d18b911d7a1e8745d6411c0cc](https://gist.github.com/filimonov/1646259d18b911d7a1e8745d6411c0cc)
+* Step 3: Attach Kafka tables back
+ ```
+ ATTACH TABLE db.kafka_table_name ON CLUSTER '{cluster}';
+ ```
+
+See also these configuration settings:
+
+```markup
+
+ smallest
+
+```
+### About Offset Consuming
+
+When a consumer joins the consumer group, the broker will check if it has a committed offset. If that is the case, then it will start from the latest offset. Both ClickHouse and librdKafka documentation state that the default value for `auto_offset_reset` is largest (or `latest` in new Kafka versions) but it is not, if the consumer is new:
+
+https://github.com/ClickHouse/ClickHouse/blob/f171ad93bcb903e636c9f38812b6aaf0ab045b04/src/Storages/Kafka/StorageKafka.cpp#L506
+
+ `conf.set("auto.offset.reset", "earliest"); // If no offset stored for this group, read all messages from the start`
+
+If there is no offset stored or it is out of range, for that particular consumer group, the consumer will start consuming from the beginning (`earliest`), and if there is some offset stored then it should use the `latest`.
+The log retention policy influences which offset values correspond to the `earliest` and `latest` configurations. Consider a scenario where a topic has a retention policy set to 1 hour. Initially, you produce 5 messages, and then, after an hour, you publish 5 more messages. In this case, the latest offset will remain unchanged from the previous example. However, due to Kafka removing the earlier messages, the earliest available offset will not be 0; instead, it will be 5.
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/_index.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/_index.md
new file mode 100644
index 0000000000..6223c3ecbc
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/_index.md
@@ -0,0 +1,7 @@
+---
+title: "Schema and Formats"
+linkTitle: "Schema and Formats"
+description: >
+ Schema inference and format-specific integration details.
+weight: 30
+---
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/kafka-schema-inference.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/kafka-schema-inference.md
new file mode 100644
index 0000000000..a47e65a6c9
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/03-schema-formats/kafka-schema-inference.md
@@ -0,0 +1,88 @@
+---
+title: "Inferring Schema from AvroConfluent Messages in Kafka for ClickHouse®"
+linkTitle: "Schema Inference for Kafka"
+weight: 100
+description: >-
+ Learn how to define Kafka table structures in ClickHouse® by using Avro's schema registry & sample message.
+---
+
+To consume messages from Kafka within ClickHouse®, you need to define the `ENGINE=Kafka` table structure with all the column names and types.
+This task can be particularly challenging when dealing with complex Avro messages, as manually determining the exact schema for
+ClickHouse is both tricky and time-consuming. This complexity is particularly frustrating in the case of Avro formats,
+where the column names and their types are already clearly defined in the schema registry.
+
+Although ClickHouse supports schema inference for files, it does not natively support this for Kafka streams.
+
+Here’s a workaround to infer the schema using AvroConfluent messages:
+
+## Step 1: Capture and Store a Raw Kafka Message
+
+First, create a table in ClickHouse to consume a raw message from Kafka and store it as a file:
+
+```sql
+CREATE TABLE test_kafka (raw String) ENGINE = Kafka
+SETTINGS kafka_broker_list = 'localhost:29092',
+ kafka_topic_list = 'movies-raw',
+ kafka_format = 'RawBLOB', -- Don't try to parse the message, return it 'as is'
+ kafka_group_name = 'tmp_test'; -- Using some dummy consumer group here.
+
+INSERT INTO FUNCTION file('./avro_raw_sample.avro', 'RawBLOB')
+SELECT * FROM test_kafka LIMIT 1
+SETTINGS max_block_size=1, stream_like_engine_allow_direct_select=1;
+
+DROP TABLE test_kafka;
+```
+
+## Step 2: Infer Schema Using the Stored File
+Using the stored raw message, let ClickHouse infer the schema based on the AvroConfluent format and a specified schema registry URL:
+
+```sql
+CREATE TEMPORARY TABLE test AS
+SELECT * FROM file('./avro_raw_sample.avro', 'AvroConfluent')
+SETTINGS format_avro_schema_registry_url='http://localhost:8085';
+
+SHOW CREATE TEMPORARY TABLE test\G;
+```
+The output from the `SHOW CREATE` command will display the inferred schema, for example:
+
+```plaintext
+Row 1:
+──────
+statement: CREATE TEMPORARY TABLE test
+(
+ `movie_id` Int64,
+ `title` String,
+ `release_year` Int64
+)
+ENGINE = Memory
+```
+
+## Step 3: Create the Kafka Table with the Inferred Schema
+Now, use the inferred schema to create the Kafka table:
+
+```sql
+CREATE TABLE movies_kafka
+(
+ `movie_id` Int64,
+ `title` String,
+ `release_year` Int64
+)
+ENGINE = Kafka
+SETTINGS kafka_broker_list = 'localhost:29092',
+ kafka_topic_list = 'movies-raw',
+ kafka_format = 'AvroConfluent',
+ kafka_group_name = 'movies',
+ kafka_schema_registry_url = 'http://localhost:8085';
+```
+
+This approach reduces manual schema definition efforts and enhances data integration workflows by utilizing the schema inference capabilities of ClickHouse for AvroConfluent messages.
+
+## Appendix
+
+**Avro** is a binary serialization format used within Apache Kafka for efficiently serializing data with a compact binary format. It relies on schemas, which define the structure of the serialized data, to ensure robust data compatibility and type safety.
+
+**Schema Registry** is a service that provides a centralized repository for Avro schemas. It helps manage and enforce schemas across applications, ensuring that the data exchanged between producers and consumers adheres to a predefined format, and facilitates schema evolution in a safe manner.
+
+In ClickHouse, the **Avro** format is used for data that contains the schema embedded directly within the file or message. This means the structure of the data is defined and included with the data itself, allowing for self-describing messages. However, embedding the schema within every message is not optimal for streaming large volumes of data, as it increases the workload and network overhead. Repeatedly passing the same schema with each message can be inefficient, particularly in high-throughput environments.
+
+On the other hand, the **AvroConfluent** format in ClickHouse is specifically designed to work with the Confluent Schema Registry. This format expects the schema to be managed externally in a schema registry rather than being embedded within each message. It retrieves schema information from the Schema Registry, which allows for centralized schema management and versioning, facilitating easier schema evolution and enforcement across different applications using Kafka.
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/_index.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/_index.md
new file mode 100644
index 0000000000..c2ecfd2aca
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/_index.md
@@ -0,0 +1,7 @@
+---
+title: "Operations and Troubleshooting"
+linkTitle: "Operations & Troubleshooting"
+description: >
+ Runtime tuning, resource settings, and error diagnostics.
+weight: 40
+---
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/background_message_broker_schedule_pool_size.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/background_message_broker_schedule_pool_size.md
new file mode 100644
index 0000000000..108fc13992
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/background_message_broker_schedule_pool_size.md
@@ -0,0 +1,131 @@
+---
+title: "Setting the background message broker schedule pool size"
+linkTitle: "Setting the background message broker schedule pool size"
+weight: 100
+description: >-
+ Guide to managing the `background_message_broker_schedule_pool_size` setting for Kafka, RabbitMQ, and NATS table engines in your database.
+---
+
+## Overview
+
+When using Kafka, RabbitMQ, or NATS table engines in ClickHouse®, you may encounter issues related to a saturated background thread pool. One common symptom is a warning similar to the following:
+
+```
+2025.03.14 08:44:26.725868 [ 344 ] {} StorageKafka (events_kafka): [rdk:MAXPOLL] [thrd:main]: Application maximum poll interval (60000ms) exceeded by 159ms (adjust max.poll.interval.ms for long-running message processing): leaving group
+```
+
+This warning typically appears **not because ClickHouse fails to poll**, but because **there are no available threads** in the background pool to handle the polling in time. In rare cases, the same error might also be caused by long flushing operations to Materialized Views (MVs), especially if their logic is complex or chained.
+
+To resolve this, you should monitor and, if needed, increase the value of the `background_message_broker_schedule_pool_size` setting.
+
+---
+
+## Step 1: Check Thread Pool Utilization
+
+Run the following SQL query to inspect the current status of your background message broker thread pool:
+
+```sql
+SELECT
+ (
+ SELECT value
+ FROM system.metrics
+ WHERE metric = 'BackgroundMessageBrokerSchedulePoolTask'
+ ) AS tasks,
+ (
+ SELECT value
+ FROM system.metrics
+ WHERE metric = 'BackgroundMessageBrokerSchedulePoolSize'
+ ) AS pool_size,
+ pool_size - tasks AS free_threads
+```
+
+If you have `metric_log` enabled, you can also monitor the **minimum number of free threads over the day**:
+
+```sql
+SELECT min(CurrentMetric_BackgroundMessageBrokerSchedulePoolSize - CurrentMetric_BackgroundMessageBrokerSchedulePoolTask) AS min_free_threads
+FROM system.metric_log
+WHERE event_date = today()
+```
+
+**If `free_threads` is close to zero or negative**, it means your thread pool is saturated and should be increased.
+
+---
+
+## Step 2: Estimate Required Pool Size
+
+To estimate a reasonable value for `background_message_broker_schedule_pool_size`, run the following query:
+
+```sql
+WITH
+ toUInt32OrDefault(extract(engine_full, 'kafka_num_consumers\s*=\s*(\d+)')) as kafka_num_consumers,
+ extract(engine_full, 'kafka_thread_per_consumer\s*=\s*(\d+|\'true\')') not in ('', '0') as kafka_thread_per_consumer,
+ multiIf(
+ engine = 'Kafka',
+ if(kafka_thread_per_consumer AND kafka_num_consumers > 0, kafka_num_consumers, 1),
+ engine = 'RabbitMQ',
+ 3,
+ engine = 'NATS',
+ 3,
+ 0 /* should not happen */
+ ) as threads_needed
+SELECT
+ ceil(sum(threads_needed) * 1.25)
+FROM
+ system.tables
+WHERE
+ engine in ('Kafka', 'RabbitMQ', 'NATS')
+```
+
+This will return an estimate that includes a 25% buffer to accommodate spikes in load.
+
+---
+
+## Step 3: Apply the New Setting
+
+1. **Create or update** the following configuration file:
+
+ **Path:** `/etc/clickhouse-server/config.d/background_message_broker_schedule_pool_size.xml`
+
+ **Content:**
+ ```xml
+
+ 120
+
+ ```
+
+ Replace `120` with the value recommended from Step 2 (rounded up if needed).
+
+2. **(Only for ClickHouse versions 23.8 and older)**
+
+ Add the same setting to the default user profile:
+
+ **Path:** `/etc/clickhouse-server/users.d/background_message_broker_schedule_pool_size.xml`
+
+ **Content:**
+ ```xml
+
+
+
+ 120
+
+
+
+ ```
+
+---
+
+## Step 4: Restart ClickHouse
+
+After applying the configuration, restart ClickHouse to apply the changes:
+
+```bash
+sudo systemctl restart clickhouse-server
+```
+
+---
+
+## Summary
+
+A saturated background message broker thread pool can lead to missed Kafka polls and consumer group dropouts. Monitoring your metrics and adjusting `background_message_broker_schedule_pool_size` accordingly ensures stable operation of Kafka, RabbitMQ, and NATS integrations.
+
+If the problem persists even after increasing the pool size, consider investigating slow MV chains or flushing logic as a potential bottleneck.
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/error-handling.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/error-handling.md
similarity index 71%
rename from content/en/altinity-kb-integrations/altinity-kb-kafka/error-handling.md
rename to content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/error-handling.md
index e9e2f533de..a0b19e2e10 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/error-handling.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/04-operations-troubleshooting/error-handling.md
@@ -14,7 +14,7 @@ It's also possible to skip up to N malformed messages for each block, with used
## After 21.6
-It's possible to stream messages which could not be parsed, this behavior could be enabled via setting: `kafka_handle_error_mode='stream'` and clickhouse wil write error and message from Kafka itself to two new virtual columns: `_error, _raw_message`.
+It's possible to stream messages which could not be parsed, this behavior could be enabled via setting: `kafka_handle_error_mode='stream'` and ClickHouse® wil write error and message from Kafka itself to two new virtual columns: `_error, _raw_message`.
So you can create another Materialized View which would collect to a separate table all errors happening while parsing with all important information like offset and content of message.
@@ -31,7 +31,7 @@ kafka_group_name = 'clickhouse',
kafka_format = 'JSONEachRow',
kafka_handle_error_mode='stream';
-CREATE MATERIALIZED VIEW default.kafka_errors
+CREATE TABLE default.kafka_errors
(
`topic` String,
`partition` Int64,
@@ -41,7 +41,11 @@ CREATE MATERIALIZED VIEW default.kafka_errors
)
ENGINE = MergeTree
ORDER BY (topic, partition, offset)
-SETTINGS index_granularity = 8192 AS
+SETTINGS index_granularity = 8192
+
+
+CREATE MATERIALIZED VIEW default.kafka_errors_mv TO default.kafka_errors
+AS
SELECT
_topic AS topic,
_partition AS partition,
@@ -52,10 +56,19 @@ FROM default.kafka_engine
WHERE length(_error) > 0
```
-
-
-[https://github.com/ClickHouse/ClickHouse/pull/20249\#issuecomment-779054737](https://github.com/ClickHouse/ClickHouse/pull/20249\#issuecomment-779054737)
+[https://github.com/ClickHouse/ClickHouse/pull/20249](https://github.com/ClickHouse/ClickHouse/pull/20249)
[https://github.com/ClickHouse/ClickHouse/pull/21850](https://github.com/ClickHouse/ClickHouse/pull/21850)
[https://altinity.com/blog/clickhouse-kafka-engine-faq](https://altinity.com/blog/clickhouse-kafka-engine-faq)
+
+
+## Since 25.8
+
+dead letter queue can be used via setting: `kafka_handle_error_mode='dead_letter_queue'` [https://github.com/ClickHouse/ClickHouse/pull/68873](https://github.com/ClickHouse/ClickHouse/pull/68873)
+
+and error related data will be saved in `system.dead_letter_queue` table.
+
+
+
+
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/_index.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/_index.md
index 4a6de0c312..3faf6e9c4f 100644
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/_index.md
+++ b/content/en/altinity-kb-integrations/altinity-kb-kafka/_index.md
@@ -1,15 +1,20 @@
---
-title: "Kafka"
-linkTitle: "Kafka"
+title: "Kafka engine"
+linkTitle: "Kafka engine"
description: >
- Kafka
+ Kafka engine
---
+## librdkafka changelog
+
+This changelog tracks the librdkafka version bundled with ClickHouse and notable related fixes.
+
```bash
git log -- contrib/librdkafka | git name-rev --stdin
```
-| **ClickHouse version** | **librdkafka version** |
+| **ClickHouse® version** | **librdkafka version** |
| :--- | :--- |
+| 25.3+ ([\#63697](https://github.com/ClickHouse/ClickHouse/issues/63697)) | [2.8.0](https://github.com/confluentinc/librdkafka/blob/v2.8.0/CHANGELOG.md) + few [fixes](https://gist.github.com/filimonov/ad252aa601d4d99fb57d4d76f14aa2bf) |
| 21.10+ ([\#27883](https://github.com/ClickHouse/ClickHouse/pull/27883)) | [1.6.1](https://github.com/edenhill/librdkafka/blob/v1.6.1/CHANGELOG.md) + snappy fixes + boring ssl + illumos_build fixes + edenhill#3279 fix|
| 21.6+ ([\#23874](https://github.com/ClickHouse/ClickHouse/pull/23874)) | [1.6.1](https://github.com/edenhill/librdkafka/blob/v1.6.1/CHANGELOG.md) + snappy fixes + boring ssl + illumos_build fixes|
| 21.1+ ([\#18671](https://github.com/ClickHouse/ClickHouse/pull/18671)) | [1.6.0-RC3](https://github.com/edenhill/librdkafka/blob/v1.6.0-RC3/CHANGELOG.md) + snappy fixes + boring ssl |
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-adjusting-librdkafka-settings.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-adjusting-librdkafka-settings.md
deleted file mode 100644
index 0a693d0648..0000000000
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-adjusting-librdkafka-settings.md
+++ /dev/null
@@ -1,175 +0,0 @@
----
-title: "Adjusting librdkafka settings"
-linkTitle: "Adjusting librdkafka settings"
-description: >
- Adjusting librdkafka settings
----
-* To set rdkafka options - add to `` section in `config.xml` or preferably use a separate file in `config.d/`:
- * [https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)
-
-Some random example:
-
-```xml
-
- 60000
- 60000
- 10000
- 5000
- 60000
- 20000
- 500
- 20971520
- all
- SSL
- /etc/clickhouse-server/ssl/kafka-ca-qa.crt
- /etc/clickhouse-server/ssl/client_clickhouse_client.pem
- /etc/clickhouse-server/ssl/client_clickhouse_client.key
- pass
-
-```
-
-## Authentication / connectivity
-
-### Amazon MSK
-
-```xml
-
-
- sasl_ssl
- root
- toor
-
-
-```
-
-### SASL/SCRAM
-
-```xml
-
-
- sasl_ssl
- SCRAM-SHA-512
- root
- toor
-
-
-```
-
-[https://leftjoin.ru/all/clickhouse-as-a-consumer-to-amazon-msk/](https://leftjoin.ru/all/clickhouse-as-a-consumer-to-amazon-msk/)
-
-### Inline Kafka certs
-
-To connect to some Kafka cloud services you may need to use certificates.
-
-If needed they can be converted to pem format and inlined into ClickHouse config.xml
-Example:
-
-```xml
-
-
-
-
-```
-
-See xml
-
-[https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka](https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka)
-
-[https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files](https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files)
-
-### Azure Event Hub
-
-See [https://github.com/ClickHouse/ClickHouse/issues/12609](https://github.com/ClickHouse/ClickHouse/issues/12609)
-
-### Kerberos
-
-* [https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/\#kafka-kerberos-support](https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/#kafka-kerberos-support)
-* [https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_storage_kerberized_kafka/configs/kafka.xml](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_storage_kerberized_kafka/configs/kafka.xml)
-
-```xml
-
-
- SASL_PLAINTEXT
- /home/kafkauser/kafkauser.keytab
- kafkauser/kafkahost@EXAMPLE.COM
-
-```
-
-### confluent cloud
-
-```xml
-
-
- smallest
- SASL_SSL
- https
- PLAIN
-xmlusername
- password
- probe
-
-
-
-```
-
-[https://docs.confluent.io/cloud/current/client-apps/config-client.html](https://docs.confluent.io/cloud/current/client-apps/config-client.html)
-
-## How to test connection settings
-
-Use kafkacat utility - it internally uses same library to access Kafla as clickhouse itself and allows easily to test different settings.
-
-```bash
-kafkacat -b my_broker:9092 -C -o -10 -t my_topic \
- -X security.protocol=SASL_SSL \
- -X sasl.mechanisms=PLAIN \
- -X sasl.username=uerName \
- -X sasl.password=Password
-
-```
-
-# Different configurations for different tables?
-
-> Is there some more documentation how to use this multiconfiguration for Kafka ?
-
-The whole logic is here:
-https://github.com/ClickHouse/ClickHouse/blob/da4856a2be035260708fe2ba3ffb9e437d9b7fef/src/Storages/Kafka/StorageKafka.cpp#L466-L475
-
-So it load the main config first, after that it load (with overwrites) the configs for all topics, **listed in `kafka_topic_list` of the table**.
-
-Also since v21.12 it's possible to use more straght-forward way using named_collections:
-https://github.com/ClickHouse/ClickHouse/pull/31691
-
-So you can say something like
-
-```sql
-CREATE TABLE test.kafka (key UInt64, value UInt64) ENGINE = Kafka(kafka1, kafka_format='CSV');
-```
-
-And after that in configuration:
-
-```xml
-
-
-
- kafka1:19092
- conf
- conf
-
-
-
-```
-
-The same fragment of code in newer versions:
-https://github.com/ClickHouse/ClickHouse/blob/d19e24f530c30f002488bc136da78f5fb55aedab/src/Storages/Kafka/StorageKafka.cpp#L474-L496
diff --git a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-rewind-fast-forward-replay.md b/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-rewind-fast-forward-replay.md
deleted file mode 100644
index a2dae43a0d..0000000000
--- a/content/en/altinity-kb-integrations/altinity-kb-kafka/altinity-kb-rewind-fast-forward-replay.md
+++ /dev/null
@@ -1,18 +0,0 @@
----
-title: "Rewind / fast-forward / replay"
-linkTitle: "Rewind / fast-forward / replay"
-description: >
- Rewind / fast-forward / replay
----
-* Step 1: Detach Kafka tables in ClickHouse
-* Step 2: `kafka-consumer-groups.sh --bootstrap-server kafka:9092 --topic topic:0,1,2 --group id1 --reset-offsets --to-latest --execute`
- * More samples: [https://gist.github.com/filimonov/1646259d18b911d7a1e8745d6411c0cc](https://gist.github.com/filimonov/1646259d18b911d7a1e8745d6411c0cc)
-* Step: Attach Kafka tables back
-
-See also these configuration settings:
-
-```markup
-
- smallest
-
-```
diff --git a/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/_index.md b/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/_index.md
new file mode 100644
index 0000000000..93ca8288d0
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/_index.md
@@ -0,0 +1,28 @@
+---
+title: "RabbitMQ"
+linkTitle: "RabbitMQ"
+description: >
+ RabbitMQ engine in ClickHouse® 24.3+
+---
+
+### Settings
+
+Basic RabbitMQ settings and use cases: https://clickhouse.com/docs/en/engines/table-engines/integrations/rabbitmq
+
+### Latest improvements/fixes
+
+##### (v23.10+)
+
+- **Allow to save unparsed records and errors in RabbitMQ**:
+NATS and FileLog engines. Add virtual columns `_error` and `_raw_message` (for NATS and RabbitMQ), `_raw_record` (for FileLog) that are filled when ClickHouse fails to parse new record.
+The behaviour is controlled under storage settings `nats_handle_error_mode` for NATS, `rabbitmq_handle_error_mode` for RabbitMQ, `handle_error_mode` for FileLog similar to `kafka_handle_error_mode`.
+If it's set to `default`, en exception will be thrown when ClickHouse fails to parse a record, if it's set to `stream`, error and raw record will be saved into virtual columns.
+Closes [#36035](https://github.com/ClickHouse/ClickHouse/issues/36035) and [#55477](https://github.com/ClickHouse/ClickHouse/pull/55477)
+
+
+##### (v24+)
+
+- [#45350 RabbitMq Storage Engine should NACK messages if exception is thrown during processing](https://github.com/ClickHouse/ClickHouse/issues/45350)
+- [#59775 rabbitmq: fix having neither acked nor nacked messages](https://github.com/ClickHouse/ClickHouse/pull/59775)
+- [#60312 Make rabbitmq nack broken messages](https://github.com/ClickHouse/ClickHouse/pull/60312)
+- [#61320 Fix logical error in RabbitMQ storage with MATERIALIZED columns](https://github.com/ClickHouse/ClickHouse/pull/61320)
diff --git a/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/error-handling.md b/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/error-handling.md
new file mode 100644
index 0000000000..4acbb34434
--- /dev/null
+++ b/content/en/altinity-kb-integrations/altinity-kb-rabbitmq/error-handling.md
@@ -0,0 +1,56 @@
+---
+title: "RabbitMQ Error handling"
+linkTitle: "RabbitMQ Error handling"
+description: >
+ Error handling for RabbitMQ table engine
+---
+
+Same approach as in Kafka but virtual columns are different. Check https://clickhouse.com/docs/en/engines/table-engines/integrations/rabbitmq#virtual-columns
+
+```sql
+CREATE TABLE IF NOT EXISTS rabbitmq.broker_errors_queue
+(
+ exchange_name String,
+ channel_id String,
+ delivery_tag UInt64,
+ redelivered UInt8,
+ message_id String,
+ timestamp UInt64
+)
+engine = RabbitMQ
+SETTINGS
+ rabbitmq_host_port = 'localhost:5672',
+ rabbitmq_exchange_name = 'exchange-test', -- required parameter even though this is done via the rabbitmq config
+ rabbitmq_queue_consume = true,
+ rabbitmq_queue_base = 'test-errors',
+ rabbitmq_format = 'JSONEachRow',
+ rabbitmq_username = 'guest',
+ rabbitmq_password = 'guest',
+ rabbitmq_handle_error_mode = 'stream';
+
+CREATE MATERIALIZED VIEW IF NOT EXISTS rabbitmq.broker_errors_mv
+(
+ exchange_name String,
+ channel_id String,
+ delivery_tag UInt64,
+ redelivered UInt8,
+ message_id String,
+ timestamp UInt64
+ raw_message String,
+ error String
+)
+ENGINE = MergeTree
+ORDER BY (error)
+SETTINGS index_granularity = 8192 AS
+SELECT
+ _exchange_name AS exchange_name,
+ _channel_id AS channel_id,
+ _delivery_tag AS delivery_tag,
+ _redelivered AS redelivered,
+ _message_id AS message_id,
+ _timestamp AS timestamp,
+ _raw_message AS raw_message,
+ _error AS error
+FROM rabbitmq.broker_errors_queue
+WHERE length(_error) > 0
+```
diff --git a/content/en/altinity-kb-integrations/bi-tools.md b/content/en/altinity-kb-integrations/bi-tools.md
index 0928c8b263..ee1d2f09ea 100644
--- a/content/en/altinity-kb-integrations/bi-tools.md
+++ b/content/en/altinity-kb-integrations/bi-tools.md
@@ -7,7 +7,7 @@ description: >
* Superset: [https://superset.apache.org/docs/databases/clickhouse](https://superset.apache.org/docs/databases/clickhouse)
* Metabase: [https://github.com/enqueue/metabase-clickhouse-driver](https://github.com/enqueue/metabase-clickhouse-driver)
* Querybook: [https://www.querybook.org/docs/setup_guide/connect_to_query_engines/\#all-query-engines](https://www.querybook.org/docs/setup_guide/connect_to_query_engines/#all-query-engines)
-* Tableau: [Clickhouse Tableau connector odbc](https://github.com/Altinity/clickhouse-tableau-connector-odbc)
+* Tableau: [Altinity Tableau Connector for ClickHouse®](https://github.com/Altinity/tableau-connector-for-clickhouse) support both JDBC & ODBC drivers
* Looker: [https://docs.looker.com/setup-and-management/database-config/clickhouse](https://docs.looker.com/setup-and-management/database-config/clickhouse)
* Apache Zeppelin
* SeekTable
diff --git a/content/en/altinity-kb-integrations/catboost-mindsdb-fast.ai.md b/content/en/altinity-kb-integrations/catboost-mindsdb-fast.ai.md
index acecf80f5a..ee41487d1c 100644
--- a/content/en/altinity-kb-integrations/catboost-mindsdb-fast.ai.md
+++ b/content/en/altinity-kb-integrations/catboost-mindsdb-fast.ai.md
@@ -11,7 +11,7 @@ Article is based on feedback provided by one of Altinity clients.
CatBoost:
* It uses gradient boosting - a hard to use technique which can outperform neural networks. Gradient boosting is powerful but it's easy to shoot yourself in the foot using it.
-* The documentation on how to use it is quite lacking. The only good source of information on how to properly configure a model to yield good results is this video: [https://www.youtube.com/watch?v=usdEWSDisS0](https://www.youtube.com/watch?v=usdEWSDisS0) . We had to dig around GitHub issues to find out how to make it work with ClickHouse.
+* The documentation on how to use it is quite lacking. The only good source of information on how to properly configure a model to yield good results is this video: [https://www.youtube.com/watch?v=usdEWSDisS0](https://www.youtube.com/watch?v=usdEWSDisS0) . We had to dig around GitHub issues to find out how to make it work with ClickHouse®.
* CatBoost is fast. Other libraries will take ~5X to ~10X as long to do what CatBoost does.
* CatBoost will do preprocessing out of the box (fills nulls, apply standard scaling, encodes strings as numbers).
* CatBoost has all functions you'd need (metrics, plotters, feature importance)
diff --git a/content/en/altinity-kb-integrations/clickhouse-odbc.md b/content/en/altinity-kb-integrations/clickhouse-odbc.md
index eec3a2e5c6..d8c66cf95f 100644
--- a/content/en/altinity-kb-integrations/clickhouse-odbc.md
+++ b/content/en/altinity-kb-integrations/clickhouse-odbc.md
@@ -1,16 +1,14 @@
---
-title: "ODBC Driver for ClickHouse"
-linkTitle: "ODBC Driver for ClickHouse"
+title: "ODBC Driver for ClickHouse®"
+linkTitle: "ODBC Driver for ClickHouse®"
weight: 100
description: >-
- ODBC Driver for ClickHouse
+ ODBC Driver for ClickHouse®
---
-# ODBC Driver for ClickHouse.
+[ODBC](https://docs.microsoft.com/en-us/sql/odbc/reference/odbc-overview) interface for ClickHouse® RDBMS.
-[ODBC](https://docs.microsoft.com/en-us/sql/odbc/reference/odbc-overview) interface for [ClickHouse](https://clickhouse.yandex) RDBMS.
-
-Licensed under the [Apache 2.0](LICENSE).
+Licensed under the [Apache 2.0](https://github.com/ClickHouse/clickhouse-odbc?tab=Apache-2.0-1-ov-file#readme).
## Installation and usage
@@ -21,7 +19,7 @@ Licensed under the [Apache 2.0](LICENSE).
3. Configure ClickHouse DSN.
Note: that install driver linked against MDAC (which is default for Windows), some non-windows native
-applications (cygwin / msys64 based) may require driver linked agains unixodbc. Build section below.
+applications (cygwin / msys64 based) may require driver linked against unixodbc. Build section below.
### MacOS
@@ -30,7 +28,7 @@ applications (cygwin / msys64 based) may require driver linked agains unixodbc.
```bash
brew install https://raw.githubusercontent.com/proller/homebrew-core/chodbc/Formula/clickhouse-odbc.rb
```
-3. Add clickhouse DSN configuration into ~/.odbc.ini file. ([sample]())
+3. Add ClickHouse DSN configuration into ~/.odbc.ini file. ([sample]())
Note: that install driver linked against iodbc (which is default for Mac), some homebrew applications
(like python) may require unixodbc driver to work properly. In that case see Build section below.
@@ -38,7 +36,7 @@ Note: that install driver linked against iodbc (which is default for Mac), some
### Linux
1. DEB/RPM packaging is not provided yet, please build & install the driver from sources.
-2. Add clickhouse DSN configuration into ~/.odbc.ini file. ([sample]())
+2. Add ClickHouse DSN configuration into ~/.odbc.ini file. ([sample]())
## Configuration
@@ -49,29 +47,29 @@ On Windows you can create/edit DSN using GUI tool through Control Panel.
The list of DSN parameters recognized by the driver is as follows:
-| Parameter | Default value | Description |
-| :-----------------: | :----------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `Url` | empty | URL that points to a running ClickHouse instance, may include username, password, port, database, etc. |
-| `Proto` | deduced from `Url`, or from `Port` and `SSLMode`: `https` if `443` or `8443` or `SSLMode` is not empty, `http` otherwise | Protocol, one of: `http`, `https` |
-| `Server` or `Host` | deduced from `Url` | IP or hostname of a server with a running ClickHouse instance on it |
-| `Port` | deduced from `Url`, or from `Proto`: `8443` if `https`, `8123` otherwise | Port on which the ClickHouse instance is listening |
-| `Path` | `/query` | Path portion of the URL |
-| `UID` or `Username` | `default` | User name |
-| `PWD` or `Password` | empty | Password |
-| `Database` | `default` | Database name to connect to |
-| `Timeout` | `30` | Connection timeout |
-| `SSLMode` | empty | Certificate verification method (used by TLS/SSL connections, ignored in Windows), one of: `allow`, `prefer`, `require`, use `allow` to enable [`SSL_VERIFY_PEER`](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_verify.html) TLS/SSL certificate verification mode, [`SSL_VERIFY_PEER \| SSL_VERIFY_FAIL_IF_NO_PEER_CERT`](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_verify.html) is used otherwise |
-| `PrivateKeyFile` | empty | Path to private key file (used by TLS/SSL connections), can be empty if no private key file is used |
-| `CertificateFile` | empty | Path to certificate file (used by TLS/SSL connections, ignored in Windows), if the private key and the certificate are stored in the same file, this can be empty if `PrivateKeyFile` is specified |
-| `CALocation` | empty | Path to the file or directory containing the CA/root certificates (used by TLS/SSL connections, ignored in Windows) |
-| `DriverLog` | `on` if `CMAKE_BUILD_TYPE` is `Debug`, `off` otherwise | Enable or disable the extended driver logging |
-| `DriverLogFile` | `\temp\clickhouse-odbc-driver.log` on Windows, `/tmp/clickhouse-odbc-driver.log` otherwise | Path to the extended driver log file (used when `DriverLog` is `on`) |
+| Parameter | Default value | Description |
+| :-----------------: | :----------------------------------------------------------------------------------------------------------------------: |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `Url` | empty | URL that points to a running ClickHouse instance, may include username, password, port, database, etc. |
+| `Proto` | deduced from `Url`, or from `Port` and `SSLMode`: `https` if `443` or `8443` or `SSLMode` is not empty, `http` otherwise | Protocol, one of: `http`, `https` |
+| `Server` or `Host` | deduced from `Url` | IP or hostname of a server with a running ClickHouse instance on it |
+| `Port` | deduced from `Url`, or from `Proto`: `8443` if `https`, `8123` otherwise | Port on which the ClickHouse instance is listening |
+| `Path` | `/query` | Path portion of the URL |
+| `UID` or `Username` | `default` | User name |
+| `PWD` or `Password` | empty | Password |
+| `Database` | `default` | Database name to connect to |
+| `Timeout` | `30` | Connection timeout |
+| `SSLMode` | empty | Certificate verification method (used by TLS/SSL connections, ignored in Windows), one of: `allow`, `prefer`, `require`, use `allow` to enable [SSL_VERIFY_PEER](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_verify.html) TLS/SSL certificate verification mode, [SSL_VERIFY_PEER \| SSL_VERIFY_FAIL_IF_NO_PEER_CERT](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_verify.html) is used otherwise |
+| `PrivateKeyFile` | empty | Path to private key file (used by TLS/SSL connections), can be empty if no private key file is used |
+| `CertificateFile` | empty | Path to certificate file (used by TLS/SSL connections, ignored in Windows), if the private key and the certificate are stored in the same file, this can be empty if `PrivateKeyFile` is specified |
+| `CALocation` | empty | Path to the file or directory containing the CA/root certificates (used by TLS/SSL connections, ignored in Windows) |
+| `DriverLog` | `on` if `CMAKE_BUILD_TYPE` is `Debug`, `off` otherwise | Enable or disable the extended driver logging |
+| `DriverLogFile` | `\temp\clickhouse-odbc-driver.log` on Windows, `/tmp/clickhouse-odbc-driver.log` otherwise | Path to the extended driver log file (used when `DriverLog` is `on`) |
## Troubleshooting & bug reporting
-If some software doesn't work properly with that driver, but works good with other drivers - we will be appritiate if you will be able to collect debug info.
+If some software doesn't work properly with that driver, but works good with other drivers - we will be appropriate if you will be able to collect debug info.
To debug issues with the driver, first things that need to be done are:
- enabling driver manager tracing. Links may contain some irrelevant vendor-specific details.
@@ -142,7 +140,7 @@ brew install git cmake make poco openssl libiodbc # You may use unixodbc INSTEAD
**Note:** usually on Linux you use unixODBC driver manager, and on Mac - iODBC.
In some (rare) cases you may need use other driver manager, please do it only
-if you clearly understand the differencies. Driver should be used with the driver
+if you clearly understand the differences. Driver should be used with the driver
manager it was linked to.
Clone the repo with submodules:
diff --git a/content/en/altinity-kb-integrations/mysql-clickhouse.md b/content/en/altinity-kb-integrations/mysql-clickhouse.md
index 73166c6fb4..34a7d4b9c5 100644
--- a/content/en/altinity-kb-integrations/mysql-clickhouse.md
+++ b/content/en/altinity-kb-integrations/mysql-clickhouse.md
@@ -1,28 +1,36 @@
---
title: "MySQL"
-linkTitle: "Integration Clickhouse with MySQL"
+linkTitle: "Integrating ClickHouse® with MySQL"
weight: 100
-description: >-
- Integration Clickhouse with MySQL
---
### Replication using MaterializeMySQL.
-- https://clickhouse.tech/docs/en/engines/database-engines/materialized-mysql/
+- https://clickhouse.com/docs/en/engines/database-engines/materialized-mysql
- https://translate.google.com/translate?sl=auto&tl=en&u=https://www.jianshu.com/p/d0d4306411b3
- https://raw.githubusercontent.com/ClickHouse/clickhouse-presentations/master/meetup47/materialize_mysql.pdf
-It reads mysql binlog directly and transform queries into something which clickhouse can support. Supports updates and deletes (under the hood implemented via something like ReplacingMergeTree with enforced FINAL and 'deleted' flag). Status is 'experimental', there are quite a lot of known limitations and issues, but some people use it. The original author of that went to another project, and the main team don't have a lot of resource to improve that for now (more important thing in the backlog)
+It reads mysql binlog directly and transform queries into something which ClickHouse® can support. Supports updates and deletes (under the hood implemented via something like ReplacingMergeTree with enforced FINAL and 'deleted' flag). Status is 'experimental', there are quite a lot of known limitations and issues, but some people use it. The original author of that went to another project, and the main team don't have a lot of resource to improve that for now (more important thing in the backlog)
The replication happens on the mysql database level.
-### Replication using debezium + Kafka
+### Replication using debezium + Kafka (+ Altinity Sink Connector for ClickHouse)
-Debezium can read the binlog and transform it to Kafka messages. You can later capture the stream of message on ClickHouse side and process it as you like.
-Please remeber that currently Kafka engine supports only at-least-once delivery guarantees.
+Debezium can read the binlog and transform it to Kafka messages.
+You can later capture the stream of message on ClickHouse side and process it as you like.
+Please remember that currently Kafka engine supports only at-least-once delivery guarantees.
It's used by several companies, quite nice & flexible. But initial setup may require some efforts.
+#### Altinity Sink Connector for ClickHouse
+
+Can handle transformation of debezium messages (with support for DELETEs and UPDATEs) and exactly-once delivery for you.
+
+Links:
+* https://altinity.com/blog/fast-mysql-to-clickhouse-replication-announcing-the-altinity-sink-connector-for-clickhouse
+* https://altinity.com/mysql-to-clickhouse/
+* https://github.com/Altinity/clickhouse-sink-connector
+
#### Same as above but using https://maxwells-daemon.io/ instead of debezium.
Have no experience / feedback there, but should be very similar to debezium.
@@ -32,16 +40,16 @@ Have no experience / feedback there, but should be very similar to debezium.
See https://altinity.com/blog/2018/6/30/realtime-mysql-clickhouse-replication-in-practice
That was done long time ago in altinity for one use-case, and it seem like it was never used outside of that.
-It's a python application with lot of switches which can copy a schema or read binlog from mysql and put it to clickhouse.
+It's a python application with lot of switches which can copy a schema or read binlog from mysql and put it to ClickHouse.
Not supported currently. But it's just a python, so maybe can be adjusted to different needs.
-### Accessing MySQL data via integration engines from inside clickhouse.
+### Accessing MySQL data via integration engines from inside ClickHouse.
-MySQL [table engine](https://clickhouse.com/docs/en/engines/table-engines/integrations/mysql/) / [table function](https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/), or [MySQL database engine](https://clickhouse.com/docs/en/engines/database-engines/mysql/) - clickhouse just connects to mysql server as a client, and can do normal selects.
+MySQL [table engine](https://clickhouse.com/docs/en/engines/table-engines/integrations/mysql/) / [table function](https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/), or [MySQL database engine](https://clickhouse.com/docs/en/engines/database-engines/mysql/) - ClickHouse just connects to mysql server as a client, and can do normal selects.
We had webinar about that a year ago: https://www.youtube.com/watch?v=44kO3UzIDLI
-Using that you can easily create some ETL script which will copy the data from mysql to clickhouse regularly, i.e. something like
+Using that you can easily create some ETL script which will copy the data from mysql to ClickHouse regularly, i.e. something like
```sql
INSERT INTO clickhouse_table SELECT * FROM mysql_table WHERE id > ...
@@ -49,7 +57,7 @@ INSERT INTO clickhouse_table SELECT * FROM mysql_table WHERE id > ...
Works great if you have append only table in MySQL.
-In newer clickhouse versions you can query this was also sharded / replicated MySQL cluster - see [ExternalDistributed](https://clickhouse.com/docs/en/engines/table-engines/integrations/ExternalDistributed/)
+In newer ClickHouse versions you can query this was also sharded / replicated MySQL cluster - see [ExternalDistributed](https://clickhouse.com/docs/en/engines/table-engines/integrations/ExternalDistributed/)
### MySQL dictionaries
diff --git a/content/en/altinity-kb-interfaces/_index.md b/content/en/altinity-kb-interfaces/_index.md
index 5fb1e32fa8..e8b5b644f7 100644
--- a/content/en/altinity-kb-interfaces/_index.md
+++ b/content/en/altinity-kb-interfaces/_index.md
@@ -4,6 +4,6 @@ linkTitle: "Interfaces"
keywords:
- clickhouse interface
description: >
- See the frequent questions users have about clickhouse-client.
+ Frequent questions users have about `clickhouse-client`
weight: 9
---
diff --git a/content/en/altinity-kb-interfaces/altinity-kb-clickhouse-client.md b/content/en/altinity-kb-interfaces/altinity-kb-clickhouse-client.md
index 4ffaba09a4..4346abd242 100644
--- a/content/en/altinity-kb-interfaces/altinity-kb-clickhouse-client.md
+++ b/content/en/altinity-kb-interfaces/altinity-kb-clickhouse-client.md
@@ -4,7 +4,7 @@ linkTitle: "clickhouse-client"
keywords:
- clickhouse client
description: >
- ClickHouse client
+ ClickHouse® client
---
Q. How can I input multi-line SQL code? can you guys give me an example?
@@ -50,4 +50,4 @@ Also, it’s possible to have several client config files and pass one of them t
References:
-* [https://clickhouse.tech/docs/en/interfaces/cli/](https://clickhouse.tech/docs/en/interfaces/cli/)
+* [https://clickhouse.com/docs/en/interfaces/cli](https://clickhouse.com/docs/en/interfaces/cli)
diff --git a/content/en/altinity-kb-kubernetes/_index.md b/content/en/altinity-kb-kubernetes/_index.md
index d569a64ee2..09ca08dc0d 100644
--- a/content/en/altinity-kb-kubernetes/_index.md
+++ b/content/en/altinity-kb-kubernetes/_index.md
@@ -1,13 +1,565 @@
---
-title: "Kubernetes"
-linkTitle: "Kubernetes"
+title: "Using the Altinity Kubernetes Operator for ClickHouse®"
+linkTitle: "Using the Altinity Kubernetes Operator for ClickHouse®"
keywords:
- clickhouse in kubernetes
- kubernetes issues
+- ALtinity Kubernetes operator for ClickHouse
description: >
- Run ClickHouse in Kubernetes without any issues.
+ Run ClickHouse® in Kubernetes without any issues.
weight: 8
+aliases:
+ /altinity-kb-kubernetes/altinity-kb-possible-issues-with-running-clickhouse-in-k8s/
---
-## clickhouse-backup
+## Useful links
+
+The Altinity Kubernetes Operator for ClickHouse® repo has very useful documentation:
+
+- [Quick Start Guide](https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md)
+- [Operator Custom Resource Definition explained](https://github.com/Altinity/clickhouse-operator/blob/master/docs/custom_resource_explained.md)
+- [Examples - YAML files to deploy the operator in many common configurations](https://github.com/Altinity/clickhouse-operator/tree/master/docs/chi-examples)
+- [Main documentation](https://github.com/Altinity/clickhouse-operator/tree/master/docs#table-of-contents)
+
+## ClickHouse Operator ip filter
+
+- In the current version of operator default user is limited to IP addresses of the cluster pods. We plan to have a password option for 0.20.0 and use a 'secret' authentication for distributed queries
+
+## Start/Stop cluster
+
+- Don't delete the operator using:
+
+```bash
+kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
+```
+
+- kubectl delete chi cluster-name # chi is the name of the CRD clickhouseInstallation
+
+## DELETE PVCs
+
+https://altinity.com/blog/preventing-clickhouse-storage-deletion-with-the-altinity-kubernetes-operator-reclaimpolicy
+
+## Scaling
+
+Best way is to scale down the deployments to 0 replicas, after that reboot the node and scale up again:
+
+1. first check that all your PVCs have the retain policy:
+
+```bash
+kubectl get pv -o=custom-columns=PV:.metadata.name,NAME:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy
+# Patch it if you need
+kubectl patch pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
+```
+
+```yaml
+spec:
+ templates:
+ volumeClaimTemplates:
+ - name: XXX
+ reclaimPolicy: Retain
+```
+
+2. After that just create a stop.yaml and `kubectl apply -f stop.yaml`
+
+```yaml
+kind: ClickHouseInstallation
+spec:
+ stop: yes
+```
+
+3. Reboot kubernetes node
+4. Scale up deployment changing the stop property to no and do an `kubectl apply -f stop.yml`
+
+```yaml
+kind: ClickHouseInstallation
+spec:
+ stop: no
+```
+
+## Check where pods are executing
+
+```bash
+kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName -n zk
+# Check which hosts in which AZs
+kubectl get node -o=custom-columns=NODE:.metadata.name,ZONE:.metadata.labels.'failure-domain\.beta\.kubernetes\.io/zone'
+```
+
+## Check node instance types:
+
+```sql
+kubectl get nodes -o json|jq -Cjr '.items[] | .metadata.name," ",.metadata.labels."beta.kubernetes.io/instance-type"," ",.metadata.labels."beta.kubernetes.io/arch", "\n"'|sort -k3 -r
+
+ip-10-3-9-2.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-9-236.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-9-190.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-9-138.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-9-110.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-8-39.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-8-219.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-8-189.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-13-40.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-12-248.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-12-216.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-12-170.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-11-229.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-11-188.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-11-175.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-10-218.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-10-160.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-10-145.eu-central-1.compute.internal t4g.large arm64
+ip-10-3-9-57.eu-central-1.compute.internal m5.large amd64
+ip-10-3-8-146.eu-central-1.compute.internal m5.large amd64
+ip-10-3-13-1.eu-central-1.compute.internal m5.xlarge amd64
+ip-10-3-11-52.eu-central-1.compute.internal m5.xlarge amd64
+ip-10-3-11-187.eu-central-1.compute.internal m5.xlarge amd64
+ip-10-3-10-217.eu-central-1.compute.internal m5.xlarge amd64
+```
+
+## Search for missing affinity rules:
+
+```bash
+kubectl get pods -o json -n zk |\
+jq -r "[.items[] | {name: .metadata.name,\
+ affinity: .spec.affinity}]"
+[
+ {
+ "name": "zookeeper-0",
+ "affinity": null
+ },
+ . . .
+]
+```
+
+## Storage classes
+
+```bash
+kubectl get pvc -o=custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,CLASS:.spec.storageClassName,VOLUME:.spec.volumeName
+...
+NAME SIZE CLASS VOLUME
+datadir-volume-zookeeper-0 25Gi gp2 pvc-9a3...9ee
+
+kubectl get storageclass/gp2
+...
+NAME PROVISIONER RECLAIMPOLICY...
+gp2 (default) ebs.csi.aws.com Delete
+```
+
+## Using CSI driver to protect storage:
+
+```yaml
+allowVolumeExpansion: true
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+ name: gp2-protected
+parameters:
+ encrypted: "true"
+ type: gp2
+provisioner: ebs.csi.aws.com
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+```
+
+## Enable Resize of Volumes
+
+Operator does not delete volumes, so those were probably deleted by Kubernetes. In some new versions there is a feature flag that deletes PVCs attached to STS when STS is deleted.
+
+Please try do the following: Use operator 0.20.3. Add the following to the defaults:
+``
+
+```yaml
+ defaults:
+ storageManagement:
+ provisioner: Operator
+```
+
+That enables storage management by operator, instead of STS. It allows to extend volumes without re-creating STS, and us increase Volume size without restart of clickhouse statefulset pods for CSI drivers which support `allowVolumeExpansion` in storage classes because statefulset template don't change and we don't need delete/create statefulset
+
+## Change server settings:
+
+https://github.com/Altinity/clickhouse-operator/issues/828
+
+```yaml
+kind: ClickHouseInstallation
+spec:
+ configuration:
+ settings:
+ max_concurrent_queries: 150
+```
+
+Or **edit ClickHouseInstallation:**
+
+```bash
+kubectl -n get chi
+
+NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE
+dnieto-test 1 4 Completed 211d
+mbak-test 1 1 Completed 44d
+rory-backupmar8 1 4 Completed 42h
+
+kubectl -n edit ClickHouseInstallation dnieto-test
+```
+
+## Clickhouse-backup for CHOP
+
+Examples for use clickhouse-backup + clickhouse-operator for EKS cluster which not managed by `altinity.cloud`
+
+Main idea: second container in clickhouse pod + CronJob which will insert and poll `system.backup_actions` commands to execute clickhouse-backup commands
+
+https://github.com/AlexAkulov/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes
+
+## Configurations:
+
+How to modify yaml configs:
+
+https://github.com/Altinity/clickhouse-operator/blob/dc6cdc6f2f61fc333248bb78a8f8efe792d14ca2/tests/e2e/manifests/chi/test-016-settings-04.yaml#L26
+
+## clickhouse-operator install Example:
+
+use latest release if possible
+https://github.com/Altinity/clickhouse-operator/releases
+
+- No. Nodes/replicas: 2 to 3 nodes with 500GB per node minimum
+- Zookeeper: 3 node ensemble
+- Type of instances: m6i.x4large to start with and you can go up to m6i.16xlarge
+- Persistent Storage/volumes: EBS gp2 for data and logs and gp3 for zookeeper
+
+### Install operator in namespace
+
+```bash
+#!/bin/bash
+
+# Namespace to install operator into
+OPERATOR_NAMESPACE="${OPERATOR_NAMESPACE:-dnieto-test-chop}"
+# Namespace to install metrics-exporter into
+METRICS_EXPORTER_NAMESPACE="${OPERATOR_NAMESPACE}"
+# Operator's docker image
+OPERATOR_IMAGE="${OPERATOR_IMAGE:-altinity/clickhouse-operator:latest}"
+# Metrics exporter's docker image
+METRICS_EXPORTER_IMAGE="${METRICS_EXPORTER_IMAGE:-altinity/metrics-exporter:latest}"
+
+# Setup clickhouse-operator into specified namespace
+kubectl apply --namespace="${OPERATOR_NAMESPACE}" -f <( \
+ curl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-template.yaml | \
+ OPERATOR_IMAGE="${OPERATOR_IMAGE}" \
+ OPERATOR_NAMESPACE="${OPERATOR_NAMESPACE}" \
+ METRICS_EXPORTER_IMAGE="${METRICS_EXPORTER_IMAGE}" \
+ METRICS_EXPORTER_NAMESPACE="${METRICS_EXPORTER_NAMESPACE}" \
+ envsubst \
+)
+```
+
+### Install zookeeper ensemble
+
+zookeepers will be named like zookeeper-0.zoons
+
+```bash
+> kubectl create ns zoo3ns
+> kubectl -n zoo3ns apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/zookeeper/quick-start-persistent-volume/zookeeper-3-nodes-1GB-for-tests-only.yaml
+
+# check names they should be like:
+# zookeeper.zoo3ns if using a new namespace
+# If using the same namespace zookeeper.
+# zookeeper must be accessed using the service like service_name.namespace
+```
+
+### Deploy test cluster
+
+```bash
+> kubectl -n dnieto-test-chop apply -f dnieto-test-chop.yaml
+```
+
+```yaml
+# dnieto-test-chop.yaml
+apiVersion: "clickhouse.altinity.com/v1"
+kind: "ClickHouseInstallation"
+metadata:
+ name: "dnieto-dev"
+spec:
+ configuration:
+ settings:
+ max_concurrent_queries: "200"
+ merge_tree/ttl_only_drop_parts: "1"
+ profiles:
+ default/queue_max_wait_ms: "10000"
+ readonly/readonly: "1"
+ users:
+ admin/networks/ip:
+ - 0.0.0.0/0
+ - '::/0'
+ admin/password_sha256_hex: ""
+ admin/profile: default
+ admin/access_management: 1
+ zookeeper:
+ nodes:
+ - host: zookeeper.dnieto-test-chop
+ port: 2181
+ clusters:
+ - name: dnieto-dev
+ templates:
+ podTemplate: pod-template-with-volumes
+ serviceTemplate: chi-service-template
+ layout:
+ shardsCount: 1
+ # put the number of desired nodes 3 by default
+ replicasCount: 2
+ templates:
+ podTemplates:
+ - name: pod-template-with-volumes
+ spec:
+ containers:
+ - name: clickhouse
+ image: clickhouse/clickhouse-server:22.3
+ # separate data from logs
+ volumeMounts:
+ - name: data-storage-vc-template
+ mountPath: /var/lib/clickhouse
+ - name: log-storage-vc-template
+ mountPath: /var/log/clickhouse-server
+ serviceTemplates:
+ - name: chi-service-template
+ generateName: "service-{chi}"
+ # type ObjectMeta struct from k8s.io/meta/v1
+ metadata:
+ annotations:
+ # https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
+ # this tags for elb load balancer
+ #service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
+ #service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
+ #https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support
+ service.beta.kubernetes.io/aws-load-balancer-internal: "true"
+ service.beta.kubernetes.io/aws-load-balancer-type: nlb
+ spec:
+ ports:
+ - name: http
+ port: 8123
+ - name: client
+ port: 9000
+ type: LoadBalancer
+ volumeClaimTemplates:
+ - name: data-storage-vc-template
+ spec:
+ # no storageClassName - means use default storageClassName
+ # storageClassName: default
+ # here if you have a storageClassName defined for gp2 you can use it.
+ # kubectl get storageclass
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 50Gi
+ reclaimPolicy: Retain
+ - name: log-storage-vc-template
+ spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 2Gi
+```
+
+### Install monitoring:
+
+In order to setup prometheus as a backend for all the asynchronous_metric_log / metric_log tables and also set up grafana dashboards:
+
+- https://github.com/Altinity/clickhouse-operator/blob/master/docs/prometheus_setup.md
+- https://github.com/Altinity/clickhouse-operator/blob/master/docs/grafana_setup.md
+- [clickhouse-operator/monitoring_setup.md at master · Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator/blob/master/docs/monitoring_setup.md)
+
+## Extra configs
+
+There is an admin user by default in the deployment that is used to admin stuff
+
+## KUBECTL chi basic comands:
+
+```bash
+*> kubectl get crd*
+
+NAME CREATED AT
+clickhouseinstallations.clickhouse.altinity.com 2021-10-11T13:46:43Z
+clickhouseinstallationtemplates.clickhouse.altinity.com 2021-10-11T13:46:44Z
+clickhouseoperatorconfigurations.clickhouse.altinity.com 2021-10-11T13:46:44Z
+eniconfigs.crd.k8s.amazonaws.com 2021-10-11T13:41:23Z
+grafanadashboards.integreatly.org 2021-10-11T13:54:37Z
+grafanadatasources.integreatly.org 2021-10-11T13:54:38Z
+grafananotificationchannels.integreatly.org 2022-05-17T14:27:48Z
+grafanas.integreatly.org 2021-10-11T13:54:37Z
+provisioners.karpenter.sh 2022-05-17T14:27:49Z
+securitygrouppolicies.vpcresources.k8s.aws 2021-10-11T13:41:27Z
+volumesnapshotclasses.snapshot.storage.k8s.io 2022-04-22T13:34:20Z
+volumesnapshotcontents.snapshot.storage.k8s.io 2022-04-22T13:34:20Z
+volumesnapshots.snapshot.storage.k8s.io 2022-04-22T13:34:20Z
+
+> *kubectl -n test-clickhouse-operator-dnieto2 get chi*
+NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE
+simple-01 70m
+
+> *kubectl -n test-clickhouse-operator-dnieto2 describe chi simple-01*
+Name: simple-01
+Namespace: test-clickhouse-operator-dnieto2
+Labels:
+Annotations:
+API Version: clickhouse.altinity.com/v1
+Kind: ClickHouseInstallation
+Metadata:
+ Creation Timestamp: 2023-01-09T20:38:06Z
+ Generation: 1
+ Managed Fields:
+ API Version: clickhouse.altinity.com/v1
+ Fields Type: FieldsV1
+ fieldsV1:
+ f:metadata:
+ f:annotations:
+ .:
+ f:kubectl.kubernetes.io/last-applied-configuration:
+ f:spec:
+ .:
+ f:configuration:
+ .:
+ f:clusters:
+ Manager: kubectl-client-side-apply
+ Operation: Update
+ Time: 2023-01-09T20:38:06Z
+ Resource Version: 267483138
+ UID: d7018efa-2b13-42fd-b1c5-b798fc6d0098
+Spec:
+ Configuration:
+ Clusters:
+ Name: simple
+Events:
+
+> *kubectl get chi --all-namespaces*
+
+NAMESPACE NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE
+andrey-dev source 1 1 Completed 38d
+eu chi-dnieto-test-common-configd 1 1 Completed 161d
+eu dnieto-test 1 4 Completed 151d
+laszlo-dev node-rescale-2 1 4 Completed 5d13h
+laszlo-dev single 1 1 Completed 5d13h
+laszlo-dev2 zk2 1 1 Completed 52d
+test-clickhouse-operator-dnieto2 simple-01
+
+> *kubectl -n test-clickhouse-operator-dnieto2 edit clickhouseinstallations.clickhouse.altinity.com simple-01
+
+# Troubleshoot operator stuff
+> kubectl -n test-clickhouse-operator-ns edit chi
+> kubectl -n test-clickhouse-operator describe chi
+> kubectl -n test-clickhouse-operator get chi -o yaml
+
+# Check operator logs usually located in kube-system or specific namespace
+> kubectl -n test-ns logs chi-operator-pod -f
+
+# Check output to yaml
+> kubectl -n test-ns get services -o yaml*
+```
+
+## Problem with DELETE finalizers:
+
+https://github.com/Altinity/clickhouse-operator/issues/830
+
+There's a problem with stuck finalizers that can cause old CHI installations to hang. The sequence of operations looks like this.
+
+1. You delete the existing ClickHouse operator using `kubectl delete -f operator-installation.yaml` with running CHI clusters.
+2. You then drop the namespace where the CHI clusters are running, e.g., `kubectl delete ns my-namespace`
+3. This hangs. You run `kubectl get ns my-namespace -o yaml` and you'll see a message like the following: "message: 'Some content in the namespace has finalizers remaining: [finalizer.clickhouseinstallation.altinity.com](http://finalizer.clickhouseinstallation.altinity.com/)"
+
+That means the CHI can't delete because its finalizer was deleted out from under it.
+
+The fix is to figure out the chi name which should still be visible and edit it to remove the finalizer reference.
+
+1. `kubectl -n my-namespace get chi`
+2. `kubectl -n my-namespace edit [clickhouseinstallations.clickhouse.altinity.com](http://clickhouseinstallations.clickhouse.altinity.com/) my-clickhouse-cluster`
+
+Remove the finalizer from the spec, save it, and everything will delete properly.
+
+**`TIP: if you delete the ns too and there is no ns just create it and apply the above method`**
+
+## Karpenter scaler
+
+```sql
+> kubectl -n karpenter get all
+NAME READY STATUS RESTARTS AGE
+pod/karpenter-75c8b7667b-vbmj4 1/1 Running 0 16d
+pod/karpenter-75c8b7667b-wszxt 1/1 Running 0 16d
+
+NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+service/karpenter ClusterIP 172.20.129.188 8080/TCP,443/TCP 16d
+
+NAME READY UP-TO-DATE AVAILABLE AGE
+deployment.apps/karpenter 2/2 2 2 16d
+
+NAME DESIRED CURRENT READY AGE
+replicaset.apps/karpenter-75c8b7667b 2 2 2 16d
+
+> kubectl -n karpenter logs pod/karpenter-75c8b7667b-vbmj4
+
+2023-02-06T06:33:44.269Z DEBUG Successfully created the logger.
+2023-02-06T06:33:44.269Z DEBUG Logging level set to: debug
+{"level":"info","ts":1675665224.2755454,"logger":"fallback","caller":"injection/injection.go:63","msg":"Starting informers..."}
+2023-02-06T06:33:44.376Z DEBUG controller waiting for configmaps {"commit": "f60dacd", "configmaps": ["karpenter-global-settings"]}
+2023-02-06T06:33:44.881Z DEBUG controller karpenter-global-settings config "karpenter-global-settings" config was added or updated: settings.Settings{BatchMaxDuration:v1.Duration{Duration:10000000000}, BatchIdleDuration:v1.Duration{Duration:1000000000}} {"commit": "f60dacd"}
+2023-02-06T06:33:44.881Z DEBUG controller karpenter-global-settings config "karpenter-global-settings" config was added or updated: settings.Settings{ClusterName:"eu", ClusterEndpoint:"https://79974769E264251E43B18AF4CA31CE8C.gr7.eu-central-1.eks.amazonaws.com", DefaultInstanceProfile:"KarpenterNodeInstanceProfile-eu", EnablePodENI:false, EnableENILimitedPodDensity:true, IsolatedVPC:false, NodeNameConvention:"ip-name", VMMemoryOverheadPercent:0.075, InterruptionQueueName:"Karpenter-eu", Tags:map[string]string{}} {"commit": "f60dacd"}
+2023-02-06T06:33:45.001Z DEBUG controller.aws discovered region {"commit": "f60dacd", "region": "eu-central-1"}
+2023-02-06T06:33:45.003Z DEBUG controller.aws unable to detect the IP of the kube-dns service, services "kube-dns" is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot get resource "services" in API group "" in the namespace "kube-system" {"commit": "f60dacd"}
+2023/02/06 06:33:45 Registering 2 clients
+2023/02/06 06:33:45 Registering 2 informer factories
+2023/02/06 06:33:45 Registering 3 informers
+2023/02/06 06:33:45 Registering 6 controllers
+2023-02-06T06:33:45.080Z DEBUG controller.aws discovered version {"commit": "f60dacd", "version": "v0.20.0"}
+2023-02-06T06:33:45.082Z INFO controller Starting server {"commit": "f60dacd", "path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
+2023-02-06T06:33:45.082Z INFO controller Starting server {"commit": "f60dacd", "kind": "health probe", "addr": "[::]:8081"}
+I0206 06:33:45.182600 1 leaderelection.go:248] attempting to acquire leader lease karpenter/karpenter-leader-election...
+2023-02-06T06:33:45.226Z INFO controller Starting informers... {"commit": "f60dacd"}
+2023-02-06T06:33:45.417Z INFO controller.aws.pricing updated spot pricing with instance types and offerings {"commit": "f60dacd", "instance-type-count": 607, "offering-count": 1400}
+2023-02-06T06:33:47.670Z INFO controller.aws.pricing updated on-demand pricing {"commit": "f60dacd", "instance-type-count": 505}
+```
+
+## Operator Affinities:
+
+
+
+## Deploy operator with clickhouse-keeper
+
+https://github.com/Altinity/clickhouse-operator/issues/959
[setup-example.yaml](https://github.com/Altinity/clickhouse-operator/blob/eb3fc4e28514d0d6ea25a40698205b02949bcf9d/docs/chi-examples/03-persistent-volume-07-do-not-chown.yaml)
+
+## Possible issues with running ClickHouse in K8s
+
+The biggest problem with running ClickHouse® in K8s, happens when clickhouse-server can't start for some reason and pod is falling in CrashloopBackOff, so you can't easily get in the pod and check/fix/restart ClickHouse.
+
+There is multiple possible reasons for this, some of them can be fixed without manual intervention in pod:
+
+1. Wrong configuration files Fix: Check templates which are being used for config file generation and fix them.
+2. While upgrade some backward incompatible changes prevents ClickHouse from start. Fix: Downgrade and check backward incompatible changes for all versions in between.
+
+Next reasons would require to have manual intervention in pod/volume.
+There is two ways, how you can get access to data:
+
+1. Change entry point of ClickHouse pod to something else, so pod wouldn’t be terminated due ClickHouse error.
+2. Attach ClickHouse data volume to some generic pod (like Ubuntu).
+3. Unclear restart which produced broken files and/or state on disk is differs too much from state in zookeeper for replicated tables. Fix: Create `force_restore_data` flag.
+4. Wrong file permission for ClickHouse files in pod. Fix: Use chown to set right ownership for files and directories.
+5. Errors in ClickHouse table schema prevents ClickHouse from start. Fix: Rename problematic `table.sql` scripts to `table.sql.bak`
+6. Occasional failure of distributed queries because of wrong user/password. Due nature of k8s with dynamic ip allocations, it's possible that ClickHouse would cache wrong ip-> hostname combination and disallow connections because of mismatched hostname. Fix: run `SYSTEM DROP DNS CACHE;` `1` in config.xml.
+
+Caveats:
+
+1. Not all configuration/state folders are being covered by persistent volumes. ([geobases](https://clickhouse.tech/docs/en/sql-reference/functions/ym-dict-functions/#multiple-geobases))
+2. Page cache belongs to k8s node and pv are being mounted to pod, in case of fast shutdown there is possibility to loss some data(needs to be clarified)
+3. Some cloud providers (GKE) can have slow unlink command, which is important for ClickHouse because it's needed for parts management. (`max_part_removal_threads` setting)
+
+Useful commands:
+
+```bash
+kubectl logs chi-chcluster-2-1-0 -c clickhouse-pod -n chcluster --previous
+kubectl describe pod chi-chcluster-2-1-0 -n chcluster
+```
+
+Q. ClickHouse is caching the Kafka pod's IP and trying to connect to the same ip even when there is a new Kafka pod running and the old one is deprecated. Is there some setting where we could refresh the connection
+
+`1` in config.xml
+
+### ClickHouse init process failed
+
+It's due to low value for env `CLICKHOUSE_INIT_TIMEOUT` value. Consider increasing it up to 1 min.
+[https://github.com/ClickHouse/ClickHouse/blob/9f5cd35a6963cc556a51218b46b0754dcac7306a/docker/server/entrypoint.sh\#L120](https://github.com/ClickHouse/ClickHouse/blob/9f5cd35a6963cc556a51218b46b0754dcac7306a/docker/server/entrypoint.sh#L120)
diff --git a/content/en/altinity-kb-kubernetes/altinity-kb-istio-user-issue-k8s.md b/content/en/altinity-kb-kubernetes/altinity-kb-istio-user-issue-k8s.md
new file mode 100644
index 0000000000..5e162d8377
--- /dev/null
+++ b/content/en/altinity-kb-kubernetes/altinity-kb-istio-user-issue-k8s.md
@@ -0,0 +1,76 @@
+---
+title: "Istio Issues"
+linkTitle: "Istio Issues"
+weight: 100
+description:
+ Working with the popular service mesh
+keywords:
+ - istio
+---
+
+## What is Istio?
+
+Per documentation on [Istio Project\'s website](https://istio.io/latest/docs/overview/what-is-istio/), Istio is "an open source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring – with few or no service code changes."
+
+Istio works quite well at providing this functionality, and does so through controlling service-to-service communication in a Cluster, find-grained control of traffic behavior, routing rules, load-balancing, a policy layer and configuration API supporting access controls, rate limiting, etc.
+
+It also provides metrics about all traffic in a cluster. One can get an amazing amount of metrics from it. Datadog even has a provider that when turned on is a bit like a firehose of information.
+
+Istio essentially uses a proxy to intercapt all network traffic and provides the ability to configured for providing a appliction-aware features.
+
+## ClickHouse and Istio
+
+The implications for ClickHouse need to be taken into consideration however, and this page will attempt to address this from real-life scenarios that Altinity devops, infrastructural, and support engineers have had to solve.
+
+### Operator High Level Description
+
+The Altinity ClickHouse Operator, when installed using a deployment, also creates four custom resources:
+
+- clickhouseinstallations.clickhouse.altinity.com (chi)
+- clickhousekeeperinstallations.clickhouse-keeper.altinity.com (chk)
+- clickhouseinstallationtemplates.clickhouse.altinity.com (chit)
+- clickhouseoperatorconfigurations.clickhouse.altinity.com (chopconf)
+
+For the first two, it uses StatefullSets to run both Keeper and and ClickHouse clusters. For Keeper, it manages how many replicas specified, and for ClickHouse, it manages both how many replicas and shards are specified.
+
+In managing `ClickHouseInstallations`, it requires that the operator can interact with the database running on clusters it creates using a specific `clickhouse_operator` user and needs network access rules that allow connection to the ClickHouse pods.
+
+Many of the issues with Istio can pertain to issues where this can be a problem, particularly in the case where the IP address of the Operator pod changes and no longer is allowed to connect to it's ClickHouse clusters that it manages.
+
+### Issue: Authentication error of clickhouse-operator
+
+This was a ClickHouse cluster running in a Kubernetes setup with Istio.
+
+- The clickhouse operator was unable to query the clickhouse pods because of authentication errors. After a period of time, the operator gave up yet the ClickHouse cluster (ClickHouseInstallation) worked normally.
+- Errors showed `AUTHENTICATION_FAILED` and `connections from :ffff:127.0.0.6 are not allowed` as well as `IP_ADDRESS_NOT_ALLOWED`
+- Also, the `clickhouse_operator` user correctly configured
+- There was a recent issue that on the surface looked similar to a recent issue with https://altinity.com/blog/deepseek-clickhouse-and-the-altinity-kubernetes-operator (disabled network access for default user due to issue with DeepSeek) and one idea seemed as if upgrading the operator (which would fix the issue if it were default user).
+- However, the key to this issue is that the problem was with the `clickhouse_operator` user, not `default` user, hence not due to the aforementioned issue.
+- More consiration was given in light of how Istio effects what services can connect which made it more obvious that it was an issue with using Istio in the operator vs. operator version
+- The suggestion was given to remove istio from the clickhouse operator `ClickHouseInstallation` and references this issue https://github.com/Altinity/clickhouse-operator/issues/1261#issuecomment-1797895080
+- The change required would be something of the sort:
+
+```yaml
+---
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: clickhouse-operator
+spec:
+ template:
+ metadata:
+ annotations:
+ sidecar.istio.io/inject: "false"
+
+---
+
+apiVersion: [clickouse.altinity.com/v1](http://clickouse.altinity.com/v1)
+kind: ClickHouseInstallation
+metadata:
+ name: your-chi
+ annotations:
+ sidecar.istio.io/inject: "false"
+
+```
+
diff --git a/content/en/altinity-kb-kubernetes/altinity-kb-possible-issues-with-running-clickhouse-in-k8s.md b/content/en/altinity-kb-kubernetes/altinity-kb-possible-issues-with-running-clickhouse-in-k8s.md
index de1c5240ed..c908418304 100644
--- a/content/en/altinity-kb-kubernetes/altinity-kb-possible-issues-with-running-clickhouse-in-k8s.md
+++ b/content/en/altinity-kb-kubernetes/altinity-kb-possible-issues-with-running-clickhouse-in-k8s.md
@@ -1,10 +1,11 @@
---
-title: "Possible issues with running ClickHouse in k8s"
-linkTitle: "Possible issues with running ClickHouse in k8s"
+title: "Possible issues with running ClickHouse® in K8s"
+linkTitle: "Possible issues with running ClickHouse® in K8s"
description: >
- Possible issues with running ClickHouse in k8s
+ Possible issues with running ClickHouse® in K8s
+draft: true
---
-The biggest problem with running ClickHouse in k8s, happens when clickhouse-server can't start for some reason and pod is falling in CrashloopBackOff, so you can't easily get in the pod and check/fix/restart ClickHouse.
+The biggest problem with running ClickHouse® in K8s, happens when clickhouse-server can't start for some reason and pod is falling in CrashloopBackOff, so you can't easily get in the pod and check/fix/restart ClickHouse.
There is multiple possible reasons for this, some of them can be fixed without manual intervention in pod:
@@ -25,7 +26,7 @@ Caveats:
1. Not all configuration/state folders are being covered by persistent volumes. ([geobases](https://clickhouse.tech/docs/en/sql-reference/functions/ym-dict-functions/#multiple-geobases))
2. Page cache belongs to k8s node and pv are being mounted to pod, in case of fast shutdown there is possibility to loss some data(needs to be clarified)
-3. Some cloud providers (GKE) can have slow unlink command, which is important for clickhouse because it's needed for parts management. (`max_part_removal_threads` setting)
+3. Some cloud providers (GKE) can have slow unlink command, which is important for ClickHouse because it's needed for parts management. (`max_part_removal_threads` setting)
Useful commands:
@@ -34,7 +35,7 @@ kubectl logs chi-chcluster-2-1-0 -c clickhouse-pod -n chcluster --previous
kubectl describe pod chi-chcluster-2-1-0 -n chcluster
```
-Q. Clickhouse is caching the Kafka pod's IP and trying to connect to the same ip even when there is a new Kafka pod running and the old one is deprecated. Is there some setting where we could refresh the connection
+Q. ClickHouse is caching the Kafka pod's IP and trying to connect to the same ip even when there is a new Kafka pod running and the old one is deprecated. Is there some setting where we could refresh the connection
`1` in config.xml
diff --git a/content/en/altinity-kb-queries-and-syntax/_index.md b/content/en/altinity-kb-queries-and-syntax/_index.md
index e771c1d459..46d710897d 100644
--- a/content/en/altinity-kb-queries-and-syntax/_index.md
+++ b/content/en/altinity-kb-queries-and-syntax/_index.md
@@ -5,6 +5,6 @@ keywords:
- clickhouse queries
- clickhouse joins
description: >
- Learn about ClickHouse queries & syntax, including Joins & Window Functions.
+ Learn about ClickHouse® queries & syntax, including Joins & Window Functions.
weight: 1
---
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-alter-modify-column-is-stuck-the-column-is-inaccessible.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-alter-modify-column-is-stuck-the-column-is-inaccessible.md
index 45be4c3aaf..ce63579370 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-alter-modify-column-is-stuck-the-column-is-inaccessible.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-alter-modify-column-is-stuck-the-column-is-inaccessible.md
@@ -6,13 +6,13 @@ description: >
---
## Problem
-You have table:
+You’ve created a table in ClickHouse with the following structure:
```sql
CREATE TABLE modify_column(column_n String) ENGINE=MergeTree() ORDER BY tuple();
```
-Populate it with data:
+You populated the table with some data:
```sql
INSERT INTO modify_column VALUES ('key_a');
@@ -20,13 +20,13 @@ INSERT INTO modify_column VALUES ('key_b');
INSERT INTO modify_column VALUES ('key_c');
```
-Tried to apply alter table query with changing column type:
+Next, you attempted to change the column type using this query:
```sql
ALTER TABLE modify_column MODIFY COLUMN column_n Enum8('key_a'=1, 'key_b'=2);
```
-But it didn’t succeed and you see an error in system.mutations table:
+However, the operation failed, and you encountered an error when inspecting the system.mutations table:
```sql
SELECT *
@@ -51,7 +51,12 @@ latest_fail_time: 2021-03-03 18:38:59
latest_fail_reason: Code: 36, e.displayText() = DB::Exception: Unknown element 'key_c' for type Enum8('key_a' = 1, 'key_b' = 2): while executing 'FUNCTION CAST(column_n :: 0, 'Enum8(\'key_a\' = 1, \'key_b\' = 2)' :: 1) -> cast(column_n, 'Enum8(\'key_a\' = 1, \'key_b\' = 2)') Enum8('key_a' = 1, 'key_b' = 2) : 2': (while reading from part /var/lib/clickhouse/data/default/modify_column/all_3_3_0/): While executing MergeTree (version 21.3.1.6041)
```
-And you can’t query that column anymore:
+The mutation result showed an error indicating that the value 'key_c' was not recognized in the Enum8 definition:
+```sql
+Unknown element 'key_c' for type Enum8('key_a' = 1, 'key_b' = 2)
+```
+
+Now, when trying to query the column, ClickHouse returns an exception and the column becomes inaccessible:
```sql
SELECT column_n
@@ -70,36 +75,54 @@ Received exception from server (version 21.3.1):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Unknown element 'key_c' for type Enum8('key_a' = 1, 'key_b' = 2): while executing 'FUNCTION CAST(column_n :: 0, 'Enum8(\'key_a\' = 1, \'key_b\' = 2)' :: 1) -> cast(column_n, 'Enum8(\'key_a\' = 1, \'key_b\' = 2)') Enum8('key_a' = 1, 'key_b' = 2) : 2': (while reading from part /var/lib/clickhouse/data/default/modify_column/all_3_3_0/): While executing MergeTreeThread.
```
-### Solution
+This query results in:
+```sql
+Code: 36. DB::Exception: Unknown element 'key_c' for type Enum8('key_a' = 1, 'key_b' = 2)
+```
-You should do the following:
+### Root Cause
+The failure occurred because the Enum8 type only allows for predefined values. Since 'key_c' wasn't included in the definition, the mutation failed and left the table in an inconsistent state.
+
+### Solution
-Check which mutation is stuck and kill it:
+1. Identify and Terminate the Stuck Mutation
+First, you need to locate the mutation that’s stuck in an incomplete state.
```sql
SELECT * FROM system.mutations WHERE table = 'modify_column' AND is_done=0 FORMAT Vertical;
+```
+
+Once you’ve identified the mutation, terminate it using:
+```sql
KILL MUTATION WHERE table = 'modify_column' AND mutation_id = 'id_of_stuck_mutation';
```
+This will stop the operation and allow you to revert the changes.
-Apply reverting modify column query to convert table to previous column type:
+2. Revert the Column Type
+Next, revert the column back to its original type, which was String, to restore the table’s accessibility:
```sql
ALTER TABLE modify_column MODIFY COLUMN column_n String;
```
-Check if column is accessible now:
+3. Verify the Column is Accessible Again
+To ensure the column is functioning normally, run a simple query to verify its data:
```sql
SELECT column_n, count() FROM modify_column GROUP BY column_n;
```
-Run fixed ALTER MODIFY COLUMN query.
+4. Apply the Correct Column Modification
+Now that the column is accessible, you can safely reapply the ALTER query, but this time include all the required enum values:
```sql
ALTER TABLE modify_column MODIFY COLUMN column_n Enum8('key_a'=1, 'key_b'=2, 'key_c'=3);
```
-You can monitor progress of column type change with system.mutations or system.parts_columns tables:
+5. Monitor Progress
+You can monitor the progress of the column modification using the system.mutations or system.parts_columns tables to ensure everything proceeds as expected:
+
+To track mutation progress:
```sql
SELECT
@@ -107,8 +130,12 @@ SELECT
parts_to_do,
is_done
FROM system.mutations
-WHERE table = 'modify_column'
+WHERE table = 'modify_column';
+```
+
+To review the column's active parts:
+```sql
SELECT
column,
type,
@@ -119,5 +146,5 @@ FROM system.parts_columns
WHERE (table = 'modify_column') AND (column = 'column_n') AND active
GROUP BY
column,
- type
+ type;
```
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md
index ade3331949..fc156f2811 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md
@@ -6,18 +6,34 @@ description: >
---
`SELECT * FROM table FINAL`
-* Before 20.5 - always executed in a single thread and slow.
+### History
+
+* Before ClickHouse® 20.5 - always executed in a single thread and slow.
* Since 20.5 - final can be parallel, see [https://github.com/ClickHouse/ClickHouse/pull/10463](https://github.com/ClickHouse/ClickHouse/pull/10463)
-* Since 20.10 - you can use `do_not_merge_across_partitions_select_final` setting.
-* Sinse 22.6 - final even more parallel, see [https://github.com/ClickHouse/ClickHouse/pull/36396](https://github.com/ClickHouse/ClickHouse/pull/36396)
+* Since 20.10 - you can use `do_not_merge_across_partitions_select_final` setting. See [https://github.com/ClickHouse/ClickHouse/pull/15938](https://github.com/ClickHouse/ClickHouse/pull/15938) and [https://github.com/ClickHouse/ClickHouse/issues/11722](https://github.com/ClickHouse/ClickHouse/issues/11722)
+* Since 22.6 - final even more parallel, see [https://github.com/ClickHouse/ClickHouse/pull/36396](https://github.com/ClickHouse/ClickHouse/pull/36396)
+* Since 22.8 - final doesn't read excessive data, see [https://github.com/ClickHouse/ClickHouse/pull/47801](https://github.com/ClickHouse/ClickHouse/pull/47801)
+* Since 23.5 - final use less memory, see [https://github.com/ClickHouse/ClickHouse/pull/50429](https://github.com/ClickHouse/ClickHouse/pull/50429)
+* Since 23.9 - final doesn't read PK columns if unneeded ie only one part in partition, see [https://github.com/ClickHouse/ClickHouse/pull/53919](https://github.com/ClickHouse/ClickHouse/pull/53919)
+* Since 23.12 - final applied only for intersecting ranges of parts, see [https://github.com/ClickHouse/ClickHouse/pull/58120](https://github.com/ClickHouse/ClickHouse/pull/58120)
+* Since 24.1 - final doesn't compare rows from the same part with level > 0, see [https://github.com/ClickHouse/ClickHouse/pull/58142](https://github.com/ClickHouse/ClickHouse/pull/58142)
+* Since 24.1 - final use vertical algorithm (more cache friendly), see [https://github.com/ClickHouse/ClickHouse/pull/54366](https://github.com/ClickHouse/ClickHouse/pull/54366)
+* Since 25.6 - final supports skip indexes (`use_skip_indexes_if_final=1` by default)
+* Since 25.12 - `apply_prewhere_after_final` and `apply_row_policy_after_final` settings for correct PREWHERE/row policy handling with FINAL
+* Since 26.2 - `enable_automatic_decision_for_merging_across_partitions_for_final=1` by default (auto-enables cross-partition optimization when safe)
+
-See [https://github.com/ClickHouse/ClickHouse/pull/15938](https://github.com/ClickHouse/ClickHouse/pull/15938) and [https://github.com/ClickHouse/ClickHouse/issues/11722](https://github.com/ClickHouse/ClickHouse/issues/11722)
+### Partitioning
-So it can work in the following way:
+Proper partition design could speed up FINAL processing.
-1. Daily partitioning
-2. After day end + some time interval during which you can get some updates - for example at 3am / 6am you do `OPTIMIZE TABLE xxx PARTITION 'prev_day' FINAL`
-3. In that case using that FINAL with `do_not_merge_across_partitions_select_final` will be cheap.
+For example, if you have a table with Daily partitioning, you can:
+- After day end + some time interval during which you can get some updates run `OPTIMIZE TABLE xxx PARTITION 'prev_day' FINAL`
+- or add table SETTINGS min_age_to_force_merge_seconds=86400,min_age_to_force_merge_on_partition_only=1
+
+In that case, using FINAL with `do_not_merge_across_partitions_select_final` will be cheap or even zero.
+
+Example:
```sql
DROP TABLE IF EXISTS repl_tbl;
@@ -81,3 +97,69 @@ SELECT count() FROM repl_tbl FINAL WHERE NOT ignore(*)
/* only 0.35 sec slower, and while partitions have about the same size that extra cost will be about constant */
```
+
+Since 26.2, `enable_automatic_decision_for_merging_across_partitions_for_final=1` (default) auto-enables this when partition key columns are included in PRIMARY KEY
+
+### Light ORDER BY
+
+All columns specified in ORDER BY will be read during FINAL processing, creating additional disk load. Use fewer columns and lighter column types to create faster queries.
+
+Example: UUID vs UInt64
+```
+CREATE TABLE uuid_table (id UUID, value UInt64) ENGINE = ReplacingMergeTree() ORDER BY id;
+CREATE TABLE uint64_table (id UInt64,value UInt64) ENGINE = ReplacingMergeTree() ORDER BY id;
+
+INSERT INTO uuid_table SELECT generateUUIDv4(), number FROM numbers(5E7);
+INSERT INTO uint64_table SELECT number, number FROM numbers(5E7);
+
+SELECT sum(value) FROM uuid_table FINAL format JSON;
+SELECT sum(value) FROM uint64_table FINAL format JSON;
+```
+[Results](https://fiddle.clickhouse.com/e2441e5d-ccb6-4f67-bee0-7cc2c4e3f43e):
+```
+ "elapsed": 0.58738197,
+ "rows_read": 50172032,
+ "bytes_read": 1204128768
+
+ "elapsed": 0.189792142,
+ "rows_read": 50057344,
+ "bytes_read": 480675040
+```
+
+### Vertical FINAL Algorithm (24.1+)
+
+When `enable_vertical_final=1` (default since 24.1), ClickHouse uses a different deduplication strategy:
+- Marks duplicate rows as deleted instead of merging them immediately
+- Filters deleted rows in a later processing step
+- Reads different columns from different parts in parallel
+
+This improves performance for queries that read only a subset of columns, as non-ORDER BY columns can be read independently from different parts.
+
+### PREWHERE and Row Policies with FINAL (25.12+)
+
+By default, PREWHERE and row policies are applied **before** FINAL deduplication. This can cause incorrect results when:
+- PREWHERE references columns that differ across duplicate rows
+- Row policies should filter based on the "winning" row values after deduplication
+
+Use these settings when needed:
+- `apply_prewhere_after_final=1` - Apply PREWHERE after deduplication
+- `apply_row_policy_after_final=1` - Apply row policies after deduplication
+
+Example problem: if you have `ReplacingMergeTree` with a `deleted` column and PREWHERE filters on it, without `apply_prewhere_after_final=1` you may get wrong results because PREWHERE sees rows before FINAL picks the winner.
+
+### FINAL with skip indexes:
+- Both `use_skip_indexes_if_final` and `use_skip_indexes_if_final_exact_mode` are enabled by default since 25.6
+- Skip indexes on PRIMARY KEY columns have lower overhead (no extra rescan needed since 26.1), see [https://github.com/ClickHouse/ClickHouse/pull/78350](https://github.com/ClickHouse/ClickHouse/pull/78350)
+
+
+### Settings reference
+
+| Setting | Default | Since | Description |
+|---------|---------|-------|-------------|
+| `do_not_merge_across_partitions_select_final` | 0 | 20.10 | Skip cross-partition merging when partitions are pre-optimized |
+| `max_final_threads` | 0 (auto) | 20.5 | Thread limit for FINAL processing |
+| `enable_vertical_final` | 1 | 24.1 | Read columns in parallel from different parts |
+| `use_skip_indexes_if_final` | 1 | 25.6 | Allow skip indexes with FINAL |
+| `use_skip_indexes_if_final_exact_mode` | 1 | 25.6 | Rescan newer parts to ensure correctness with skip indexes |
+| `apply_prewhere_after_final` | 0 | 25.12 | Apply PREWHERE after deduplication (needed when PREWHERE references non-PK columns) |
+| `enable_automatic_decision_for_merging_across_partitions_for_final` | 1 | 26.2 | Auto-enable `do_not_merge_across_partitions_select_final` when partition key is in PK |
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-kill-query.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-kill-query.md
index 8cb6d48148..255bc07de0 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-kill-query.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-kill-query.md
@@ -7,12 +7,12 @@ description: >
Unfortunately not all queries can be killed.
`KILL QUERY` only sets a flag that must be checked by the query.
A query pipeline is checking this flag before a switching to next block. If the pipeline has stuck somewhere in the middle it cannot be killed.
-If a query does not stop, the only way to get rid of it is to restart ClickHouse.
+If a query does not stop, the only way to get rid of it is to restart ClickHouse®.
-See also
+See also:
-[https://github.com/ClickHouse/ClickHouse/issues/3964](https://github.com/ClickHouse/ClickHouse/issues/3964)
-[https://github.com/ClickHouse/ClickHouse/issues/1576](https://github.com/ClickHouse/ClickHouse/issues/1576)
+* [https://github.com/ClickHouse/ClickHouse/issues/3964](https://github.com/ClickHouse/ClickHouse/issues/3964)
+* [https://github.com/ClickHouse/ClickHouse/issues/1576](https://github.com/ClickHouse/ClickHouse/issues/1576)
## How to replace a running query
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-optimize-vs-optimize-final.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-optimize-vs-optimize-final.md
index 8d525b7bf0..a2f0a245a8 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-optimize-vs-optimize-final.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-optimize-vs-optimize-final.md
@@ -12,7 +12,7 @@ You have 40 parts in 3 partitions. This unscheduled merge selects some partition
`OPTIMIZE TABLE xyz FINAL` -- initiates a cycle of unscheduled merges.
-ClickHouse merges parts in this table until will remains 1 part in each partition (if a system has enough free disk space). As a result, you get 3 parts, 1 part per partition. In this case, CH rewrites parts even if they are already merged into a single part. It creates a huge CPU / Disk load if the table ( XYZ) is huge. ClickHouse reads / uncompress / merge / compress / writes all data in the table.
+ClickHouse® merges parts in this table until will remains 1 part in each partition (if a system has enough free disk space). As a result, you get 3 parts, 1 part per partition. In this case, ClickHouse rewrites parts even if they are already merged into a single part. It creates a huge CPU / Disk load if the table (XYZ) is huge. ClickHouse reads / uncompress / merge / compress / writes all data in the table.
If this table has size 1TB it could take around 3 hours to complete.
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-parameterized-views.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-parameterized-views.md
index ab0499f8e3..27b1a53b7b 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-parameterized-views.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-parameterized-views.md
@@ -4,6 +4,36 @@ linkTitle: "Parameterized views"
description: >
Parameterized views
---
+
+ClickHouse® versions 23.1+ (23.1.6.42, 23.2.5.46, 23.3.1.2823)
+have inbuilt support for [parametrized views](https://clickhouse.com/docs/en/sql-reference/statements/create/view#parameterized-view):
+
+```sql
+CREATE VIEW my_new_view AS
+SELECT *
+FROM deals
+WHERE category_id IN (
+ SELECT category_id
+ FROM deal_categories
+ WHERE category = {category:String}
+)
+
+SELECT * FROM my_new_view(category = 'hot deals');
+```
+### One more example
+
+```sql
+CREATE OR REPLACE VIEW v AS SELECT 1::UInt32 x WHERE x IN ({xx:Array(UInt32)});
+
+select * from v(xx=[1,2,3]);
+┌─x─┐
+│ 1 │
+└───┘
+```
+
+
+## ClickHouse versions pre 23.1
+
Custom settings allows to emulate parameterized views.
You need to enable custom settings and define any prefixes for settings.
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-possible-deadlock-avoided.-client-should-retry.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-possible-deadlock-avoided.-client-should-retry.md
index cd8727780c..e5c1be17b0 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-possible-deadlock-avoided.-client-should-retry.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-possible-deadlock-avoided.-client-should-retry.md
@@ -4,7 +4,7 @@ linkTitle: "Possible deadlock avoided. Client should retry"
description: >
Possible deadlock avoided. Client should retry
---
-In version 19.14 a serious issue was found: a race condition that can lead to server deadlock. The reason for that was quite fundamental, and a temporary workaround for that was added ("possible deadlock avoided").
+In ClickHouse® version 19.14 a serious issue was found: a race condition that can lead to server deadlock. The reason for that was quite fundamental, and a temporary workaround for that was added ("possible deadlock avoided").
Those locks are one of the fundamental things that the core team was actively working on in 2020.
@@ -20,4 +20,8 @@ In 20.6 all table-level locks which were possible to remove were removed, so alt
Typically issue was happening when doing some concurrent select on `system.parts` / `system.columns` / `system.table` with simultaneous table manipulations (doing some kind of ALTERS / TRUNCATES / DROP)I
-If that exception happens often in your use-case: An update is recommended. In the meantime, check which queries are running (especially to system.tables / system.parts and other system tables) and check if killing them / avoiding them helps to solve the issue.
+If that exception happens often in your use-case:
+- use recent clickhouse versions
+- ensure you use Atomic engine for the database (not Ordinary) (can be checked in system.databases)
+
+Sometime you can try to workaround issue by finding the queries which uses that table concurently (especially to system.tables / system.parts and other system tables) and try killing them (or avoiding them).
diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-sample-by.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-sample-by.md
index 6dfa4cecbe..7f7e010091 100644
--- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-sample-by.md
+++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-sample-by.md
@@ -8,17 +8,17 @@ The execution pipeline is embedded in the partition reading code.
So that works this way:
-1. ClickHouse does partition pruning based on `WHERE` conditions.
+1. ClickHouse® does partition pruning based on `WHERE` conditions.
2. For every partition, it picks a columns ranges (aka 'marks' / 'granulas') based on primary key conditions.
3. Here the sampling logic is applied: a) in case of `SAMPLE k` (`k` in `0..1` range) it adds conditions `WHERE sample_key < k * max_int_of_sample_key_type` b) in case of `SAMPLE k OFFSET m` it adds conditions `WHERE sample_key BETWEEN m * max_int_of_sample_key_type AND (m + k) * max_int_of_sample_key_type`c) in case of `SAMPLE N` (N>1) if first estimates how many rows are inside the range we need to read and based on that convert it to 3a case (calculate k based on number of rows in ranges and desired number of rows)
4. on the data returned by those other conditions are applied (so here the number of rows can be decreased here)
-[Source Code](https://github.com/ClickHouse/ClickHouse/blob/92c937db8b50844c7216d93c5c398d376e82f6c3/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp#L355)
+* [Source Code](https://github.com/ClickHouse/ClickHouse/blob/92c937db8b50844c7216d93c5c398d376e82f6c3/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp#L355)
## SAMPLE by
-[Docs](https://clickhouse.yandex/docs/en/query_language/select/#select-sample-clause)
-[Source Code](https://github.com/ClickHouse/ClickHouse/blob/92c937db8b50844c7216d93c5c398d376e82f6c3/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp#L355)
+* [Docs](https://clickhouse.yandex/docs/en/query_language/select/#select-sample-clause)
+* [Source Code](https://github.com/ClickHouse/ClickHouse/blob/92c937db8b50844c7216d93c5c398d376e82f6c3/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp#L355)
SAMPLE key
Must be:
@@ -56,4 +56,4 @@ SELECT count() FROM table WHERE ... AND cityHash64(some_high_card_key) % 10 = 0;
SELECT count() FROM table WHERE ... AND rand() % 10 = 0; -- Non-deterministic
```
-ClickHouse will read more data from disk compared to an example with a good SAMPLE key, but it's more universal and can be used if you can't change table ORDER BY key.
\ No newline at end of file
+ClickHouse will read more data from disk compared to an example with a good SAMPLE key, but it's more universal and can be used if you can't change table ORDER BY key. (To learn more about ClickHouse internals, [Administrator Training for ClickHouse](https://altinity.com/clickhouse-training/) is available.)
\ No newline at end of file
diff --git a/content/en/altinity-kb-queries-and-syntax/ansi-sql-mode.md b/content/en/altinity-kb-queries-and-syntax/ansi-sql-mode.md
index 61d3973b5d..ab4306ecda 100644
--- a/content/en/altinity-kb-queries-and-syntax/ansi-sql-mode.md
+++ b/content/en/altinity-kb-queries-and-syntax/ansi-sql-mode.md
@@ -4,13 +4,49 @@ linkTitle: "ANSI SQL mode"
description: >
ANSI SQL mode
---
-It's possible to tune some settings which would make ClickHouse more ANSI SQL compatible(and slower):
+To make ClickHouse® more compatible with ANSI SQL standards (at the expense of some performance), you can adjust several settings. These configurations will bring ClickHouse closer to ANSI SQL behavior but may introduce a slowdown in query performance:
```sql
-SET join_use_nulls=1; -- introduced long ago
-SET cast_keep_nullable=1; -- introduced in 20.5
-SET union_default_mode='DISTINCT'; -- introduced in 21.1
-SET allow_experimental_window_functions=1; -- introduced in 21.3
-SET prefer_column_name_to_alias=1; -- introduced in 21.4;
-SET group_by_use_nulls=1; -- introduced in 22.7;
+join_use_nulls=1
```
+Introduced in: early versions
+Ensures that JOIN operations return NULL for non-matching rows, aligning with standard SQL behavior.
+
+
+```sql
+cast_keep_nullable=1
+```
+Introduced in: v20.5
+Preserves the NULL flag when casting between data types, which is typical in ANSI SQL.
+
+
+```sql
+union_default_mode='DISTINCT'
+```
+Introduced in: v21.1
+Makes the UNION operation default to UNION DISTINCT, which removes duplicate rows, following ANSI SQL behavior.
+
+
+```sql
+allow_experimental_window_functions=1
+```
+Introduced in: v21.3
+Enables support for window functions, which are a standard feature in ANSI SQL.
+
+
+```sql
+prefer_column_name_to_alias=1
+```
+Introduced in: v21.4
+This setting resolves ambiguities by preferring column names over aliases, following ANSI SQL conventions.
+
+
+```sql
+group_by_use_nulls=1
+```
+Introduced in: v22.7
+Allows NULL values to appear in the GROUP BY clause, consistent with ANSI SQL behavior.
+
+By enabling these settings, ClickHouse becomes more ANSI SQL-compliant, although this may come with a trade-off in terms of performance. Each of these options can be enabled as needed, based on the specific SQL compatibility requirements of your application.
+
+
diff --git a/content/en/altinity-kb-queries-and-syntax/array-functions-as-window.md b/content/en/altinity-kb-queries-and-syntax/array-functions-as-window.md
index 8a66dde750..73a08bdca8 100644
--- a/content/en/altinity-kb-queries-and-syntax/array-functions-as-window.md
+++ b/content/en/altinity-kb-queries-and-syntax/array-functions-as-window.md
@@ -2,15 +2,14 @@
title: "Using array functions to mimic window-functions alike behavior"
linkTitle: "Using array functions to mimic window-functions alike behavior"
weight: 100
-description: >-
- Using array functions to mimic window-functions alike behavior.
---
-# Using array functions to mimic window functions alike behavior
+There are cases where you may need to mimic window functions using arrays in ClickHouse. This could be for optimization purposes, to better manage memory, or to enable on-disk spilling, especially if you’re working with an older version of ClickHouse that doesn't natively support window functions.
-There are some usecases when you may want to mimic window functions using Arrays - as an optimization step, or to contol the memory better / use on-disk spiling, or just if you have old ClickHouse version.
+Here’s an example demonstrating how to mimic a window function like runningDifference() using arrays:
-## Running difference sample
+#### Step 1: Create Sample Data
+We’ll start by creating a test table with some sample data:
```sql
DROP TABLE IS EXISTS test_running_difference
@@ -24,10 +23,8 @@ SELECT
FROM numbers(100)
-SELECT * FROM test_running_difference
-```
+SELECT * FROM test_running_difference;
-```text
┌─id─┬──────────────────ts─┬────val─┐
│ 0 │ 2010-01-01 00:00:00 │ -1209 │
│ 1 │ 2010-01-01 00:00:00 │ 43 │
@@ -134,13 +131,15 @@ SELECT * FROM test_running_difference
100 rows in set. Elapsed: 0.003 sec.
```
+This table contains IDs, timestamps (ts), and values (val), where each id appears multiple times with different timestamps.
+
+#### Step 2: Running Difference Example
+If you try using runningDifference directly, it works block by block, which can be problematic when the data needs to be ordered or when group changes occur.
+
-runningDifference works only in blocks & require ordered data & problematic when group changes
```sql
select id, val, runningDifference(val) from (select * from test_running_difference order by id, ts);
-```
-```
┌─id─┬────val─┬─runningDifference(val)─┐
│ 0 │ -1209 │ 0 │
│ 0 │ 66839 │ 68048 │
@@ -248,13 +247,15 @@ select id, val, runningDifference(val) from (select * from test_running_differen
```
-## Arrays !
+The output may look inconsistent because runningDifference requires ordered data within blocks.
-### 1. Group & Collect the data into array
+#### Step 3: Using Arrays for Grouping and Calculation
+Instead of using runningDifference, we can utilize arrays to group data, sort it, and apply similar logic more efficiently.
+**Grouping Data into Arrays** -
+You can group multiple columns into arrays by using the groupArray function. For example, to collect several columns as arrays of tuples, you can use the following query:
-you can collect several column by builing array of tuples:
-```
+```sql
SELECT
id,
groupArray(tuple(ts, val))
@@ -285,10 +286,9 @@ GROUP BY id
└────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
-### Do needed ordering in each array
-
-For example - by second element of tuple:
-```
+**Sorting Arrays** -
+To sort the arrays by a specific element, for example, by the second element of the tuple, you can use the arraySort function:
+```sql
SELECT
id,
arraySort(x -> (x.2), groupArray((ts, val)))
@@ -321,9 +321,11 @@ GROUP BY id
20 rows in set. Elapsed: 0.004 sec.
```
-That can be rewritten like this:
+This sorts each array by the val (second element of the tuple) for each id.
-```
+Simplified Sorting Example - We can rewrite the query in a more concise way using WITH clauses for better readability:
+
+```sql
WITH
groupArray(tuple(ts, val)) as window_rows,
arraySort(x -> x.1, window_rows) as sorted_window_rows
@@ -334,9 +336,10 @@ FROM test_running_difference
GROUP BY id
```
-### Apply needed logic arrayMap / arrayDifference etc
+**Applying Calculations with Arrays** -
+Once the data is sorted, you can apply array functions like arrayMap and arrayDifference to calculate differences between values in the arrays:
-```
+```sql
WITH
groupArray(tuple(ts, val)) as window_rows,
arraySort(x -> x.1, window_rows) as sorted_window_rows,
@@ -347,10 +350,7 @@ SELECT
sorted_window_rows_val_column_diff
FROM test_running_difference
GROUP BY id
-```
-
-```
┌─id─┬─sorted_window_rows_val_column_diff─┐
│ 0 │ [0,68048,68243,72389,67860] │
│ 1 │ [0,19397,17905,16978,18345] │
@@ -380,10 +380,8 @@ GROUP BY id
You can do also a lot of magic with arrayEnumerate and accessing different values by their ids.
-### Now you can return you arrays back to rows
-
-
-use arrayJoin
+**Reverting Arrays Back to Rows** -
+You can convert the arrays back into rows using arrayJoin:
```sql
WITH
@@ -398,9 +396,7 @@ SELECT
FROM test_running_difference
GROUP BY id
```
-
-
- or ARRAY JOIN
+Or use ARRAY JOIN to join the arrays back to the original structure:
```sql
SELECT
@@ -421,8 +417,6 @@ FROM test_running_difference
GROUP BY id
) as t1
ARRAY JOIN sorted_window_rows_val_column_diff as diff, sorted_window_rows_ts_column as ts
-
```
-
-etc.
+This allows you to manipulate and analyze data within arrays effectively, using powerful functions such as arrayMap, arrayDifference, and arrayEnumerate.
diff --git a/content/en/altinity-kb-queries-and-syntax/async-inserts.md b/content/en/altinity-kb-queries-and-syntax/async-inserts.md
new file mode 100644
index 0000000000..2bb0486557
--- /dev/null
+++ b/content/en/altinity-kb-queries-and-syntax/async-inserts.md
@@ -0,0 +1,157 @@
+---
+title: "Async INSERTs"
+linkTitle: "Async INSERTs"
+description: >
+ Comprehensive guide to ClickHouse Async INSERTs - configuration, best practices, and monitoring
+---
+
+## Overview
+
+Async INSERTs is a ClickHouse® feature that enables automatic server-side batching of data. While we generally recommend batching at the application/ingestor level for better control and decoupling, async inserts are valuable when you have hundreds or thousands of clients performing small inserts and client-side batching is not feasible.
+
+**Key Documentation:** [Official Async Inserts Documentation](https://clickhouse.com/docs/en/optimize/asynchronous-inserts)
+
+## How Async Inserts Work
+
+When `async_insert=1` is enabled, ClickHouse buffers incoming inserts and flushes them to disk when one of these conditions is met:
+1. Buffer reaches specified size (`async_insert_max_data_size`)
+2. Time threshold elapses (`async_insert_busy_timeout_ms`)
+3. Maximum number of queries accumulate (`async_insert_max_query_number`)
+
+## Critical Configuration Settings
+
+### Core Settings
+
+```sql
+-- Enable async inserts (0=disabled, 1=enabled)
+SET async_insert = 1;
+
+-- Wait behavior (STRONGLY RECOMMENDED: use 1)
+-- 0 = fire-and-forget mode (risky - no error feedback)
+-- 1 = wait for data to be written to storage
+SET wait_for_async_insert = 1;
+
+-- Buffer flush conditions
+SET async_insert_max_data_size = 1000000; -- 1MB default
+SET async_insert_busy_timeout_ms = 1000; -- 1 second
+SET async_insert_max_query_number = 100; -- max queries before flush
+```
+
+### Adaptive Timeout (Since 24.3)
+
+```sql
+-- Adaptive timeout automatically adjusts flush timing based on server load
+-- Default: 1 (enabled) - OVERRIDES manual timeout settings
+-- Set to 0 for deterministic behavior with manual settings
+SET async_insert_use_adaptive_busy_timeout = 0;
+```
+
+## Important Behavioral Notes
+
+### What Works and What Doesn't
+
+✅ **Works with Async Inserts:**
+- Direct INSERT with VALUES
+- INSERT with FORMAT (JSONEachRow, CSV, etc.)
+- Native protocol inserts (since 22.x)
+
+❌ **Does NOT Work:**
+- `INSERT .. SELECT` statements - Other strategies are needed for managing performance and load. Do not use `async_insert`.
+
+### Data Safety Considerations
+
+**ALWAYS use `wait_for_async_insert = 1` in production!**
+
+Risks with `wait_for_async_insert = 0`:
+- **Silent data loss** on errors (read-only table, disk full, too many parts)
+- Data loss on sudden restart (no fsync by default)
+- Data not immediately queryable after acknowledgment
+- No error feedback to client
+
+### Deduplication Behavior
+
+- **Sync inserts:** Automatic deduplication enabled by default
+- **Async inserts:** Deduplication disabled by default
+- Enable with `async_insert_deduplicate = 1` (since 22.x)
+- **Warning:** Don't use with `deduplicate_blocks_in_dependent_materialized_views = 1`
+
+# features / improvements
+
+* Async insert dedup: Support block deduplication for asynchronous inserts. Before this change, async inserts did not support deduplication, because multiple small inserts coexisted in one inserted batch:
+ - [#38075](https://github.com/ClickHouse/ClickHouse/issues/38075)
+ - [#43304](https://github.com/ClickHouse/ClickHouse/pull/43304)
+* Added system table `asynchronous_insert_log`. It contains information about asynchronous inserts (including results of queries in fire-and-forget mode. (with wait_for_async_insert=0)) for better introspection [#42040](https://github.com/ClickHouse/ClickHouse/pull/42040)
+* Support async inserts in **clickhouse-client** for queries with inlined data **(Native protocol)**:
+ - [#34267](https://github.com/ClickHouse/ClickHouse/pull/34267)
+ - [#54098](https://github.com/ClickHouse/ClickHouse/issues/54098)
+ - [#54381](https://github.com/ClickHouse/ClickHouse/issues/54381)
+* Async insert backpressure [#4762](https://github.com/ClickHouse/ClickHouse/issues/47623)
+* Limit the deduplication overhead when using `async_insert_deduplicate` [#46549](https://github.com/ClickHouse/ClickHouse/pull/46549)
+* `SYSTEM FLUSH ASYNC INSERTS` [#49160](https://github.com/ClickHouse/ClickHouse/pull/49160)
+* Adjustable asynchronous insert timeouts [#58486](https://github.com/ClickHouse/ClickHouse/pull/58486)
+
+
+## bugfixes
+
+- Fixed bug which could lead to deadlock while using asynchronous inserts [#43233](https://github.com/ClickHouse/ClickHouse/pull/43233).
+- Fix crash when async inserts with deduplication are used for ReplicatedMergeTree tables using a nondefault merging algorithm [#51676](https://github.com/ClickHouse/ClickHouse/pull/51676)
+- Async inserts not working with log_comment setting [48430](https://github.com/ClickHouse/ClickHouse/issues/48430)
+- Fix misbehaviour with async inserts with deduplication [#50663](https://github.com/ClickHouse/ClickHouse/pull/50663)
+- Reject Insert if `async_insert=1` and `deduplicate_blocks_in_dependent_materialized_views=1`[#60888](https://github.com/ClickHouse/ClickHouse/pull/60888)
+- Disable `async_insert_use_adaptive_busy_timeout` correctly with compatibility settings [#61486](https://github.com/ClickHouse/ClickHouse/pull/61468)
+
+
+## observability / introspection
+
+In 22.x versions, it is not possible to relate `part_log/query_id` column with `asynchronous_insert_log/query_id` column. We need to use `query_log/query_id`:
+
+`asynchronous_insert_log` shows up the `query_id` and `flush_query_id` of each async insert. The `query_id` from `asynchronous_insert_log` shows up in the `system.query_log` as `type = 'QueryStart'` but the same `query_id` does not show up in the `query_id` column of the `system.part_log`. Because the `query_id` column in the `part_log` is the identifier of the INSERT query that created a data part, and it seems it is for sync INSERTS but not for async inserts.
+
+So in `asynchronous_inserts` table you can check the current batch that still has not been flushed. In the `asynchronous_insert_log` you can find a log of all the flushed async inserts.
+
+This has been improved in **ClickHouse 23.7** Flush queries for async inserts (the queries that do the final push of data) are now logged in the `system.query_log` where they appear as `query_kind = 'AsyncInsertFlush'` [#51160](https://github.com/ClickHouse/ClickHouse/pull/51160)
+
+
+## Versions
+
+- **23.8** is a good version to start using async inserts because of the improvements and bugfixes.
+- **24.3** the new adaptive timeout mechanism has been added so ClickHouse will throttle the inserts based on the server load.[#58486](https://github.com/ClickHouse/ClickHouse/pull/58486) This new feature is enabled by default and will OVERRRIDE current async insert settings, so better to disable it if your async insert settings are working. Here's how to do it in a clickhouse-client session: `SET async_insert_use_adaptive_busy_timeout = 0;` You can also add it as a setting on the INSERT or as a profile setting.
+
+
+## Metrics
+
+```sql
+SELECT name
+FROM system.columns
+WHERE (`table` = 'metric_log') AND ((name ILIKE '%asyncinsert%') OR (name ILIKE '%asynchronousinsert%'))
+
+┌─name─────────────────────────────────────────────┐
+│ ProfileEvent_AsyncInsertQuery │
+│ ProfileEvent_AsyncInsertBytes │
+│ ProfileEvent_AsyncInsertRows │
+│ ProfileEvent_AsyncInsertCacheHits │
+│ ProfileEvent_FailedAsyncInsertQuery │
+│ ProfileEvent_DistributedAsyncInsertionFailures │
+│ CurrentMetric_AsynchronousInsertThreads │
+│ CurrentMetric_AsynchronousInsertThreadsActive │
+│ CurrentMetric_AsynchronousInsertThreadsScheduled │
+│ CurrentMetric_AsynchronousInsertQueueSize │
+│ CurrentMetric_AsynchronousInsertQueueBytes │
+│ CurrentMetric_PendingAsyncInsert │
+│ CurrentMetric_AsyncInsertCacheSize │
+└──────────────────────────────────────────────────┘
+
+SELECT *
+FROM system.metrics
+WHERE (metric ILIKE '%asyncinsert%') OR (metric ILIKE '%asynchronousinsert%')
+
+┌─metric─────────────────────────────┬─value─┬─description─────────────────────────────────────────────────────────────┐
+│ AsynchronousInsertThreads │ 1 │ Number of threads in the AsynchronousInsert thread pool. │
+│ AsynchronousInsertThreadsActive │ 0 │ Number of threads in the AsynchronousInsert thread pool running a task. │
+│ AsynchronousInsertThreadsScheduled │ 0 │ Number of queued or active jobs in the AsynchronousInsert thread pool. │
+│ AsynchronousInsertQueueSize │ 1 │ Number of pending tasks in the AsynchronousInsert queue. │
+│ AsynchronousInsertQueueBytes │ 680 │ Number of pending bytes in the AsynchronousInsert queue. │
+│ PendingAsyncInsert │ 7 │ Number of asynchronous inserts that are waiting for flush. │
+│ AsyncInsertCacheSize │ 0 │ Number of async insert hash id in cache │
+└────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────┘
+```
diff --git a/content/en/altinity-kb-queries-and-syntax/atomic-insert.md b/content/en/altinity-kb-queries-and-syntax/atomic-insert.md
index 91351e2dfa..61a96e7d42 100644
--- a/content/en/altinity-kb-queries-and-syntax/atomic-insert.md
+++ b/content/en/altinity-kb-queries-and-syntax/atomic-insert.md
@@ -10,12 +10,18 @@ An insert will create one part if:
* Data is inserted directly into a MergeTree table
* Data is inserted into a single partition.
+* Smaller blocks are properly squashed up to the configured block size (`min_insert_block_size_rows` and `min_insert_block_size_bytes`)
* For INSERT FORMAT:
* Number of rows is less than `max_insert_block_size` (default is `1048545`)
- * Parallel formatting is disabled (For TSV, TKSV, CSV, and JSONEachRow formats setting `input_format_parallel_parsing=0` is set).
-* For INSERT SELECT:
- * Number of rows is less than `max_block_size`
-* Smaller blocks are properly squashed up to the configured block size (`min_insert_block_size_rows` and `min_insert_block_size_bytes`)
+ * Parallel formatting is disabled (For TSV, TSKV, CSV, and JSONEachRow formats setting `input_format_parallel_parsing=0` is set).
+* For INSERT SELECT (including all variants with table functions), data for insert should be created fully deterministically.
+ * non-deterministic functions there like rand() not used in SELECT
+ * Number of rows/bytes is less than `min_insert_block_size_rows` and `min_insert_block_size_bytes`
+ * And one of:
+ * setting max_threads to 1
+ * adding ORDER BY to the table's DDL (not ordering by tuple)
+ * There is some ORDER BY inside SELECT
+ * See [example](https://fiddle.clickhouse.com/48d38d3d-668d-4513-ba21-e595276b3136)
* The MergeTree table doesn't have Materialized Views (there is no atomicity Table <> MV)
https://github.com/ClickHouse/ClickHouse/issues/9195#issuecomment-587500824
@@ -25,23 +31,23 @@ https://github.com/ClickHouse/ClickHouse/issues/5148#issuecomment-487757235
### Generate test data in Native and TSV format ( 100 millions rows )
-Text formats and Native format require different set of settings, here I want to find / demonstrate mandatory minumum of settings for any case.
+Text formats and Native format require different set of settings, here I want to find / demonstrate mandatory minimum of settings for any case.
```bash
clickhouse-client -q \
- 'select toInt64(number) A, toString(number) S from numbers(100000000) format Native' > t.native
+ 'SELECT toInt64(number) A, toString(number) S FROM numbers(100000000) FORMAT Native' > t.native
clickhouse-client -q \
- 'select toInt64(number) A, toString(number) S from numbers(100000000) format TSV' > t.tsv
+ 'SELECT toInt64(number) A, toString(number) S FROM numbers(100000000) FORMAT TSV' > t.tsv
```
### Insert with default settings (not atomic)
```bash
-drop table if exists trg;
-create table trg(A Int64, S String) Engine=MergeTree order by A;
+DROP TABLE IF EXISTS trg;
+CREATE TABLE trg(A Int64, S String) Engine=MergeTree ORDER BY A;
-- Load data in Native format
-clickhouse-client -q 'insert into trg format Native' (y <= ts), state_arr, ts_arr)) AS uniq
+ toStartOfDay(ts) AS ts,
+ uniqExactMerge(uniqExactState(user_id)) OVER (ORDER BY ts ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS uniq
+FROM events
+GROUP BY ts
+ORDER BY ts ASC
+
+┌──────────────────ts─┬─uniq─┐
+│ 2021-04-29 00:00:00 │ 2 │
+│ 2021-04-30 00:00:00 │ 3 │
+│ 2021-05-01 00:00:00 │ 4 │
+│ 2021-05-02 00:00:00 │ 5 │
+│ 2021-05-03 00:00:00 │ 7 │
+└─────────────────────┴──────┘
+
+SELECT
+ ts,
+ uniqExactMerge(state) OVER (ORDER BY ts ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS uniq
FROM
(
SELECT
- toStartOfDay(ts) AS _ts,
+ toStartOfDay(ts) AS ts,
uniqExactState(user_id) AS state
FROM events
- GROUP BY _ts
+ GROUP BY ts
)
ORDER BY ts ASC
@@ -46,11 +58,17 @@ ORDER BY ts ASC
│ 2021-05-02 00:00:00 │ 5 │
│ 2021-05-03 00:00:00 │ 7 │
└─────────────────────┴──────┘
+```
-WITH arrayJoin(range(toUInt32(_ts) AS int, least(int + toUInt32((3600 * 24) * 5), toUInt32(toDateTime('2021-05-04 00:00:00'))), 3600 * 24)) AS ts_expanded
+## Using arrays
+
+```sql
+WITH
+ groupArray(_ts) AS ts_arr,
+ groupArray(state) AS state_arr
SELECT
- toDateTime(ts_expanded) AS ts,
- uniqExactMerge(state) AS uniq
+ arrayJoin(ts_arr) AS ts,
+ arrayReduce('uniqExactMerge', arrayFilter((x, y) -> (y <= ts), state_arr, ts_arr)) AS uniq
FROM
(
SELECT
@@ -59,7 +77,6 @@ FROM
FROM events
GROUP BY _ts
)
-GROUP BY ts
ORDER BY ts ASC
┌──────────────────ts─┬─uniq─┐
@@ -69,22 +86,20 @@ ORDER BY ts ASC
│ 2021-05-02 00:00:00 │ 5 │
│ 2021-05-03 00:00:00 │ 7 │
└─────────────────────┴──────┘
-```
-## Using window functions (starting from Clickhouse 21.3)
-
-```sql
+WITH arrayJoin(range(toUInt32(_ts) AS int, least(int + toUInt32((3600 * 24) * 5), toUInt32(toDateTime('2021-05-04 00:00:00'))), 3600 * 24)) AS ts_expanded
SELECT
- ts,
- uniqExactMerge(state) OVER (ORDER BY ts ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS uniq
+ toDateTime(ts_expanded) AS ts,
+ uniqExactMerge(state) AS uniq
FROM
(
SELECT
- toStartOfDay(ts) AS ts,
+ toStartOfDay(ts) AS _ts,
uniqExactState(user_id) AS state
FROM events
- GROUP BY ts
+ GROUP BY _ts
)
+GROUP BY ts
ORDER BY ts ASC
┌──────────────────ts─┬─uniq─┐
diff --git a/content/en/altinity-kb-queries-and-syntax/data-types-on-disk-and-in-ram.md b/content/en/altinity-kb-queries-and-syntax/data-types-on-disk-and-in-ram.md
index 2150d339f9..daa99a3301 100644
--- a/content/en/altinity-kb-queries-and-syntax/data-types-on-disk-and-in-ram.md
+++ b/content/en/altinity-kb-queries-and-syntax/data-types-on-disk-and-in-ram.md
@@ -39,4 +39,4 @@ description: >
-See also [https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup41/data_processing.pdf](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup41/data_processing.pdf) (slide 17-22)
+See also the presentation [Data processing into ClickHouse®](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup41/data_processing.pdf), especially slides 17-22.
diff --git a/content/en/altinity-kb-queries-and-syntax/datetime64.md b/content/en/altinity-kb-queries-and-syntax/datetime64.md
index fae72c9f16..601cf478d6 100644
--- a/content/en/altinity-kb-queries-and-syntax/datetime64.md
+++ b/content/en/altinity-kb-queries-and-syntax/datetime64.md
@@ -2,11 +2,9 @@
title: "DateTime64"
linkTitle: "DateTime64"
weight: 100
-description: >-
- DateTime64 data type
---
-## Substract fractional seconds
+## Subtract fractional seconds
```sql
WITH toDateTime64('2021-09-07 13:41:50.926', 3) AS time
diff --git a/content/en/altinity-kb-queries-and-syntax/delete-via-tombstone-column.md b/content/en/altinity-kb-queries-and-syntax/delete-via-tombstone-column.md
index 32b7292d9f..61ccb70496 100644
--- a/content/en/altinity-kb-queries-and-syntax/delete-via-tombstone-column.md
+++ b/content/en/altinity-kb-queries-and-syntax/delete-via-tombstone-column.md
@@ -4,6 +4,12 @@ linkTitle: "DELETE via tombstone column"
description: >
DELETE via tombstone column
---
+
+This article provides an overview of the different methods to handle row deletion in ClickHouse, using tombstone columns and ALTER UPDATE or DELETE. The goal is to highlight the performance impacts of different techniques and storage settings, including a scenario using S3 for remote storage.
+
+1. Creating a Test Table
+We will start by creating a simple MergeTree table with a tombstone column (is_active) to track active rows:
+
```sql
CREATE TABLE test_delete
(
@@ -16,7 +22,10 @@ CREATE TABLE test_delete
)
ENGINE = MergeTree
ORDER BY key;
-
+```
+2. Inserting Data
+Insert sample data into the table:
+```sql
INSERT INTO test_delete (key, ts, value_a, value_b, value_c) SELECT
number,
1,
@@ -25,8 +34,12 @@ INSERT INTO test_delete (key, ts, value_a, value_b, value_c) SELECT
concat('string', toString(number))
FROM numbers(10000000);
-INSERT INTO test_delete (key, ts, value_a, value_b, value_c) VALUES (400000, 2, 'totally different string', 'another totally different string', 'last string');
+INSERT INTO test_delete (key, ts, value_a, value_b, value_c) VALUES (400000, 2, 'totally different string', 'another totally different string', 'last string');
+```
+3. Querying the Data
+To verify the inserted data:
+```sql
SELECT *
FROM test_delete
WHERE key = 400000;
@@ -37,31 +50,49 @@ WHERE key = 400000;
┌────key─┬─ts─┬─value_a──────────────────┬─value_b────────────────┬─value_c──────┬─is_active─┐
│ 400000 │ 1 │ some_looong_string400000 │ another_long_str400000 │ string400000 │ 1 │
└────────┴────┴──────────────────────────┴────────────────────────┴──────────────┴───────────┘
+```
+This should return two rows with different ts values.
+
+4. Soft Deletion Using ALTER UPDATE
+Instead of deleting a row, you can mark it as inactive by setting is_active to 0:
+```sql
SET mutations_sync = 2;
ALTER TABLE test_delete
UPDATE is_active = 0 WHERE (key = 400000) AND (ts = 1);
-
Ok.
0 rows in set. Elapsed: 0.058 sec.
-
+```
+After updating, you can filter out inactive rows:
+```sql
SELECT *
FROM test_delete
-WHERE (key = 400000) AND is_active;
-
-┌────key─┬─ts─┬─value_a──────────────────┬─value_b──────────────────────────┬─value_c─────┬─is_active─┐
-│ 400000 │ 2 │ totally different string │ another totally different string │ last string │ 1 │
-└────────┴────┴──────────────────────────┴──────────────────────────────────┴─────────────┴───────────┘
+WHERE (key = 400000) AND is_active=0;
+┌────key─┬─ts─┬─value_a──────────────────┬─value_b────────────────┬─value_c──────┬─is_active─┐
+│ 400000 │ 1 │ some_looong_string400000 │ another_long_str400000 │ string400000 │ 0 │
+└────────┴────┴──────────────────────────┴────────────────────────┴──────────────┴───────────┘
+```
+5. Hard Deletion Using ALTER DELETE
+If you need to completely remove a row from the table, you can use ALTER DELETE:
+```sql
ALTER TABLE test_delete
DELETE WHERE (key = 400000) AND (ts = 1);
Ok.
0 rows in set. Elapsed: 1.101 sec. -- 20 times slower!!!
+```
+However, this operation is significantly slower compared to the ALTER UPDATE approach. For example:
+
+ALTER DELETE: Takes around 1.1 seconds
+ALTER UPDATE: Only 0.05 seconds
+The reason for this difference is that DELETE modifies the physical data structure, while UPDATE merely changes a column value.
+
+```sql
SELECT *
FROM test_delete
WHERE key = 400000;
@@ -70,7 +101,7 @@ WHERE key = 400000;
│ 400000 │ 2 │ totally different string │ another totally different string │ last string │ 1 │
└────────┴────┴──────────────────────────┴──────────────────────────────────┴─────────────┴───────────┘
--- For ReplacingMergeTree
+-- For ReplacingMergeTree -> https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replacingmergetree
OPTIMIZE TABLE test_delete FINAL;
@@ -86,3 +117,94 @@ WHERE key = 400000
│ 400000 │ 2 │ totally different string │ another totally different string │ last string │ 1 │
└────────┴────┴──────────────────────────┴──────────────────────────────────┴─────────────┴───────────┘
```
+
+Soft Deletion (via ALTER UPDATE): A quicker approach that does not involve physical data deletion but rather updates the tombstone column.
+Hard Deletion (via ALTER DELETE): Can take significantly longer, especially with large datasets stored in remote storage like S3.
+
+6. Optimizing for Faster Deletion with S3 Storage
+If using S3 for storage, the DELETE operation becomes even slower due to the overhead of handling remote data. Here’s an example with a table using S3-backed storage:
+
+```sql
+CREATE TABLE test_delete
+(
+ `key` UInt32,
+ `value_a` String,
+ `value_b` String,
+ `value_c` String,
+ `is_deleted` UInt8 DEFAULT 0
+)
+ENGINE = MergeTree
+ORDER BY key
+SETTINGS storage_policy = 's3tiered';
+
+INSERT INTO test_delete (key, value_a, value_b, value_c) SELECT
+ number,
+ concat('some_looong_string', toString(number)),
+ concat('another_long_str', toString(number)),
+ concat('really long string', toString(arrayMap(i -> cityHash64(i*number), range(50))))
+FROM numbers(10000000);
+
+OPTIMIZE TABLE test_delete FINAL;
+
+ALTER TABLE test_delete MOVE PARTITION tuple() TO DISK 's3disk';
+
+SELECT count() FROM test_delete;
+┌──count()─┐
+│ 10000000 │
+└──────────┘
+1 row in set. Elapsed: 0.002 sec.
+```
+
+7. DELETE Using ALTER UPDATE and Row Policy
+You can also control visibility at the query level using row policies. For example, to only show rows where is_active = 1:
+
+To delete a row using ALTER UPDATE:
+
+```sql
+CREATE ROW POLICY pol1 ON test_delete USING is_active=1 TO all;
+
+SELECT count() FROM test_delete; -- select count() became much slower, it reads data now, not metadata
+┌──count()─┐
+│ 10000000 │
+└──────────┘
+1 row in set. Elapsed: 0.314 sec. Processed 10.00 million rows, 10.00 MB (31.84 million rows/s., 31.84 MB/s.)
+
+ALTER TABLE test_delete UPDATE is_active = 0 WHERE (key = 400000) settings mutations_sync = 2;
+0 rows in set. Elapsed: 1.256 sec.
+
+SELECT count() FROM test_delete;
+┌─count()─┐
+│ 9999999 │
+└─────────┘
+```
+This impacts the performance of queries like SELECT count(), as ClickHouse now needs to scan data instead of reading metadata.
+
+8. DELETE Using ALTER DELETE - https://clickhouse.com/docs/en/sql-reference/statements/alter/delete
+To delete a row using ALTER DELETE:
+
+```sql
+ALTER TABLE test_delete DELETE WHERE (key = 400001) settings mutations_sync = 2;
+0 rows in set. Elapsed: 955.672 sec.
+
+SELECT count() FROM test_delete;
+┌─count()─┐
+│ 9999998 │
+└─────────┘
+```
+This operation may take significantly longer compared to soft deletions (around 955 seconds in this example for large datasets):
+
+9. DELETE Using DELETE Statement - https://clickhouse.com/docs/en/sql-reference/statements/delete
+The DELETE statement can also be used to remove data from a table:
+
+```sql
+DELETE FROM test_delete WHERE (key = 400002);
+0 rows in set. Elapsed: 1.281 sec.
+
+SELECT count() FROM test_delete;
+┌─count()─┐
+│ 9999997 │
+└─────────┘
+```
+This operation is faster, with an elapsed time of around 1.28 seconds in this case:
+
+The choice between ALTER UPDATE and ALTER DELETE depends on your use case. For soft deletes, updating a tombstone column is significantly faster and easier to manage. However, if you need to physically remove rows, be mindful of the performance costs, especially with remote storage like S3.
diff --git a/content/en/altinity-kb-queries-and-syntax/distinct-vs-group-by-vs-limit-by.md b/content/en/altinity-kb-queries-and-syntax/distinct-vs-group-by-vs-limit-by.md
index 2ec384e2a9..20779629b2 100644
--- a/content/en/altinity-kb-queries-and-syntax/distinct-vs-group-by-vs-limit-by.md
+++ b/content/en/altinity-kb-queries-and-syntax/distinct-vs-group-by-vs-limit-by.md
@@ -2,8 +2,6 @@
title: "DISTINCT & GROUP BY & LIMIT 1 BY what the difference"
linkTitle: "DISTINCT & GROUP BY & LIMIT 1 BY what the difference"
weight: 100
-description: >-
- Page description for heading and indexes.
---
## DISTINCT
@@ -94,7 +92,7 @@ MemoryTracker: Peak memory usage (for query): 4.05 GiB.
0 rows in set. Elapsed: 4.852 sec. Processed 100.00 million rows, 800.00 MB (20.61 million rows/s., 164.88 MB/s.)
-This query faster than first, because ClickHouse doesn't need to merge states for all keys, only for first 1000 (based on LIMIT)
+This query faster than first, because ClickHouse® doesn't need to merge states for all keys, only for first 1000 (based on LIMIT)
SELECT number % 1000 AS key
@@ -119,7 +117,7 @@ MemoryTracker: Peak memory usage (for query): 3.77 MiB.
```
* Multi threaded
-* Will return result only after competion of aggregation
+* Will return result only after completion of aggregation
## LIMIT BY
diff --git a/content/en/altinity-kb-queries-and-syntax/explain-query.md b/content/en/altinity-kb-queries-and-syntax/explain-query.md
index 9517af142b..685453e570 100644
--- a/content/en/altinity-kb-queries-and-syntax/explain-query.md
+++ b/content/en/altinity-kb-queries-and-syntax/explain-query.md
@@ -10,10 +10,12 @@ description: >
```sql
EXPLAIN AST
SYNTAX
- PLAN header = 0,
+ PLAN indexes = 0,
+ header = 0,
description = 1,
actions = 0,
optimize = 1
+ json = 0
PIPELINE header = 0,
graph = 0,
compact = 1
@@ -25,7 +27,9 @@ SELECT ...
* `SYNTAX` - query text after AST-level optimizations
* `PLAN` - query execution plan
* `PIPELINE` - query execution pipeline
-* `ESTIMATE` - https://github.com/ClickHouse/ClickHouse/pull/26131 (since 21.9)
+* `ESTIMATE` - See [Estimates for select query](https://github.com/ClickHouse/ClickHouse/pull/26131), available since ClickHouse® 21.9
+* `indexes=1` supported starting from 21.6 (https://github.com/ClickHouse/ClickHouse/pull/22352 )
+* `json=1` supported starting from 21.6 (https://github.com/ClickHouse/ClickHouse/pull/23082)
References
diff --git a/content/en/altinity-kb-queries-and-syntax/group-by/_index.md b/content/en/altinity-kb-queries-and-syntax/group-by/_index.md
index e7866bc3a9..5a79605fad 100644
--- a/content/en/altinity-kb-queries-and-syntax/group-by/_index.md
+++ b/content/en/altinity-kb-queries-and-syntax/group-by/_index.md
@@ -6,7 +6,7 @@ keywords:
- clickhouse group by
- clickhouse memory
description: >
- Learn about GROUP BY clause in ClickHouse.
+ Learn about the GROUP BY clause in ClickHouse®
weight: 1
---
@@ -14,7 +14,7 @@ weight: 1
[Code](https://github.com/ClickHouse/ClickHouse/blob/8ab5270ded39c8b044f60f73c1de00c8117ab8f2/src/Interpreters/Aggregator.cpp#L382)
-ClickHouse uses non-blocking? hash tables, so each thread has at least one hash table.
+ClickHouse® uses non-blocking? hash tables, so each thread has at least one hash table.
It makes easier to not care about sync between multiple threads, but has such disadvantages as:
1. Bigger memory usage.
@@ -52,7 +52,7 @@ https://clickhouse.com/docs/en/sql-reference/statements/select/group-by/#select-
## optimize_aggregation_in_order GROUP BY
-Usually it works slower than regular GROUP BY, because ClickHouse need's to read and process data in specific ORDER, which makes it much more complicated to parallelize reading and aggregating.
+Usually it works slower than regular GROUP BY, because ClickHouse needs to read and process data in specific ORDER, which makes it much more complicated to parallelize reading and aggregating.
But it use much less memory, because ClickHouse can stream resultset and there is no need to keep it in memory.
@@ -143,7 +143,7 @@ Size of keys participated in GROUP BY
2. States of aggregation functions:
-Be careful with function, which state can use unrestricted amount of memory and grow indefenetely:
+Be careful with function, which state can use unrestricted amount of memory and grow indefinitely:
- groupArray (groupArray(1000)())
- uniqExact (uniq,uniqCombined)
diff --git a/content/en/altinity-kb-queries-and-syntax/group-by/tricks.md b/content/en/altinity-kb-queries-and-syntax/group-by/tricks.md
index 1483073b7d..4595d6a279 100644
--- a/content/en/altinity-kb-queries-and-syntax/group-by/tricks.md
+++ b/content/en/altinity-kb-queries-and-syntax/group-by/tricks.md
@@ -54,7 +54,7 @@ FROM numbers_mt(1000000000)
-All queries and datasets are unique, so in different situations different hacks could work better or worsen.
+All queries and datasets are unique, so in different situations different hacks could work better or worse.
### PreFilter values before GROUP BY
@@ -90,11 +90,11 @@ FORMAT `Null`
### Use Fixed-width data types instead of String
-EG you have 2 strings which has values in special form like this
+For example, you have 2 strings which has values in special form like this
'ABX 1412312312313'
-You can just remove 4 first characters and convert rest of them to UInt64
+You can just remove the first 4 characters and convert the rest to UInt64
toUInt64(substr('ABX 1412312312313',5))
@@ -193,7 +193,7 @@ Elapsed: 6.247 sec. Processed 1.00 billion rows, 27.00 GB (160.09 million rows/s
```
-It can be especially useful when you tries to do GROUP BY lc_column_1, lc_column_2 and ClickHouse falls back to serialized algorytm.
+It can be especially useful when you tries to do GROUP BY lc_column_1, lc_column_2 and ClickHouse® falls back to serialized algorithm.
### Two LowCardinality Columns in GROUP BY
@@ -281,9 +281,9 @@ Elapsed: 2.910 sec. Processed 1.00 billion rows, 27.00 GB (343.64 million rows/s
```
### Shard your data by one of common high cardinal GROUP BY key
-So on each shard you will have 1/N of all unique combination and this will result in smaller hash table.
+So on each shard you will have 1/N of all unique combination and this will result in smaller hash tables.
-Lets create 2 distributed tables with different distribution: rand() and by user_id
+Let's create 2 distributed tables with different distribution: rand() and by user_id
```sql
CREATE TABLE sessions_distributed AS sessions
@@ -728,7 +728,7 @@ MemoryTracker: Peak memory usage (for query): 14.55 GiB.
### Reduce number of threads
-Because each thread use independent hash table, if you lower thread amount it will reduce number of hash tables as well and lower memory usage at the cost of slower query execution.
+Because each thread uses an independent hash table, if you lower thread amount it will reduce number of hash tables as well and lower memory usage at the cost of slower query execution.
```sql
@@ -1093,7 +1093,7 @@ https://github.com/ClickHouse/ClickHouse/pull/33439
### GROUP BY in external memory
-Slow
+Slow!
### Use hash function for GROUP BY keys
diff --git a/content/en/altinity-kb-queries-and-syntax/joins/_index.md b/content/en/altinity-kb-queries-and-syntax/joins/_index.md
index 7868e67f86..6f6267594a 100644
--- a/content/en/altinity-kb-queries-and-syntax/joins/_index.md
+++ b/content/en/altinity-kb-queries-and-syntax/joins/_index.md
@@ -3,9 +3,70 @@ title: "JOINs"
linkTitle: "JOINs"
description: >
JOINs
+aliases:
+ - /altinity-kb-queries-and-syntax/joins/join-table-engine/
---
-See presentation:
+Resources:
-[https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/join.pdf](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/join.pdf)
+* [Overview of JOINs (Russian)](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/join.pdf) - Presentation from Meetup 38 in 2019
+* [Notes on JOIN options](https://excalidraw.com/#json=xX_heZcCu0whsDmC2Mdvo,ppbUVFpPz-flJu5ZDnwIPw)
-https://excalidraw.com/#json=xX_heZcCu0whsDmC2Mdvo,ppbUVFpPz-flJu5ZDnwIPw
+## Join Table Engine
+
+The main purpose of JOIN table engine is to avoid building the right table for joining on each query execution. So it's usually used when you have a high amount of fast queries which share the same right table for joining.
+
+### Updates
+
+It's possible to update rows with setting `join_any_take_last_row` enabled.
+
+```sql
+CREATE TABLE id_val_join
+(
+ `id` UInt32,
+ `val` UInt8
+)
+ENGINE = Join(ANY, LEFT, id)
+SETTINGS join_any_take_last_row = 1
+
+Ok.
+
+INSERT INTO id_val_join VALUES (1,21)(1,22)(3,23);
+
+Ok.
+
+SELECT *
+FROM
+(
+ SELECT toUInt32(number) AS id
+ FROM numbers(4)
+) AS n
+ANY LEFT JOIN id_val_join USING (id)
+
+┌─id─┬─val─┐
+│ 0 │ 0 │
+│ 1 │ 22 │
+│ 2 │ 0 │
+│ 3 │ 23 │
+└────┴─────┘
+
+INSERT INTO id_val_join VALUES (1,40)(2,24);
+
+Ok.
+
+SELECT *
+FROM
+(
+ SELECT toUInt32(number) AS id
+ FROM numbers(4)
+) AS n
+ANY LEFT JOIN id_val_join USING (id)
+
+┌─id─┬─val─┐
+│ 0 │ 0 │
+│ 1 │ 40 │
+│ 2 │ 24 │
+│ 3 │ 23 │
+└────┴─────┘
+```
+
+[Join table engine documentation](https://clickhouse.com/docs/en/engines/table-engines/special/join/)
diff --git a/content/en/altinity-kb-queries-and-syntax/joins/join-table-engine.md b/content/en/altinity-kb-queries-and-syntax/joins/join-table-engine.md
index 86a4453fad..1b0a6fb757 100644
--- a/content/en/altinity-kb-queries-and-syntax/joins/join-table-engine.md
+++ b/content/en/altinity-kb-queries-and-syntax/joins/join-table-engine.md
@@ -3,6 +3,7 @@ title: "JOIN table engine"
linkTitle: "JOIN table engine"
description: >
JOIN table engine
+draft: true
---
The main purpose of JOIN table engine is to avoid building the right table for joining on each query execution. So it's usually used when you have a high amount of fast queries which share the same right table for joining.
@@ -60,4 +61,4 @@ ANY LEFT JOIN id_val_join USING (id)
└────┴─────┘
```
-[https://clickhouse.tech/docs/en/engines/table-engines/special/join/](https://clickhouse.tech/docs/en/engines/table-engines/special/join/)
+[Join table engine documentation](https://clickhouse.com/docs/en/engines/table-engines/special/join/)
diff --git a/content/en/altinity-kb-queries-and-syntax/joins/joins-tricks.md b/content/en/altinity-kb-queries-and-syntax/joins/joins-tricks.md
new file mode 100644
index 0000000000..26468cee0b
--- /dev/null
+++ b/content/en/altinity-kb-queries-and-syntax/joins/joins-tricks.md
@@ -0,0 +1,400 @@
+---
+title: "JOIN optimization tricks"
+linkTitle: "JOIN optimization tricks"
+---
+
+All tests below were done with default `hash` join. ClickHouse joins are evolving rapidly and behavior varies with other join types.
+
+# Data
+
+For our exercise, we will use two tables from a well known TPS-DS benchmark: store_sales and customer. Table sizes are the following:
+
+store_sales = 2 billion rows
+customer = 12 millions rows
+
+So there are 200 rows in store_sales table per each customer on average. Also 90% of customers made 1-10 purchases.
+
+Schema example:
+
+```sql
+CREATE TABLE store_sales
+(
+ `ss_sold_time_sk` DateTime,
+ `ss_sold_date_sk` Date,
+ `ss_ship_date_sk` Date,
+ `ss_item_sk` UInt32,
+ `ss_customer_sk` UInt32,
+ `ss_cdemo_sk` UInt32,
+ `ss_hdemo_sk` UInt32,
+ `ss_addr_sk` UInt32,
+ `ss_store_sk` UInt32,
+ `ss_promo_sk` UInt32,
+ `ss_ticket_number` UInt32,
+ `ss_quantity` UInt32,
+ `ss_wholesale_cost` Float64,
+ `ss_list_price` Float64,
+ `ss_sales_price` Float64,
+ `ss_ext_discount_amt` Float64,
+ `ss_ext_sales_price` Float64,
+ `ss_ext_wholesale_cost` Float64,
+ `ss_ext_list_price` Float64,
+ `ss_ext_tax` Float64,
+ `ss_coupon_amt` Float64,
+ `ss_net_paid` Float64,
+ `ss_net_paid_inc_tax` Float64,
+ `ss_net_profit` Float64
+)
+ENGINE = MergeTree
+ORDER BY ss_ticket_number
+
+CREATE TABLE customer
+(
+ `c_customer_sk` UInt32,
+ `c_current_addr_sk` UInt32,
+ `c_first_shipto_date_sk` Date,
+ `c_first_sales_date_sk` Date,
+ `c_salutation` String,
+ `c_c_first_name` String,
+ `c_last_name` String,
+ `c_preferred_cust_flag` String,
+ `c_birth_date` Date,
+ `c_birth_country` String,
+ `c_login` String,
+ `c_email_address` String,
+ `c_last_review_date` Date
+)
+ENGINE = MergeTree
+ORDER BY c_customer_id
+```
+
+# Target query
+
+```sql
+SELECT
+ sumIf(ss_sales_price, customer.c_first_name = 'James') AS sum_James,
+ sumIf(ss_sales_price, customer.c_first_name = 'Lisa') AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+INNER JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk
+```
+
+## Baseline performance
+
+```sql
+SELECT
+ sumIf(ss_sales_price, customer.c_first_name = 'James') AS sum_James,
+ sumIf(ss_sales_price, customer.c_first_name = 'Lisa') AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+INNER JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk
+
+0 rows in set. Elapsed: 188.384 sec. Processed 2.89 billion rows, 40.60 GB (15.37 million rows/s., 216.92 MB/s.)
+```
+
+## Manual pushdown of conditions
+
+If we look at our query, we only care if sale belongs to customer named `James` or `Lisa` and dont care for rest of cases. We can use that.
+
+Usually, ClickHouse is able to pushdown conditions, but not in that case, when conditions itself part of function expression, so you can manually help in those cases.
+
+```sql
+SELECT
+ sumIf(ss_sales_price, customer.c_first_name = 'James') as sum_James,
+ sumIf(ss_sales_price, customer.c_first_name = 'Lisa') as sum_Lisa,
+ sum(ss_sales_price) as sum_total
+FROM store_sales LEFT JOIN (SELECT * FROM customer WHERE c_first_name = 'James' OR c_first_name = 'Lisa') as customer ON store_sales.ss_customer_sk = customer.c_customer_sk
+
+1 row in set. Elapsed: 35.370 sec. Processed 2.89 billion rows, 40.60 GB (81.76 million rows/s., 1.15 GB/s.)
+```
+
+
+## Reduce right table row size
+
+### Reduce attribute columns (push expression before JOIN step)
+
+Our row from the right table consists of 2 fields: customer_sk and c_first_name.
+First one is needed to JOIN by it, so it's not much we can do here, but we can transform a bit of the second column.
+
+Again, let's look in how we use this column in main query:
+
+customer.c_first_name = 'James'
+customer.c_first_name = 'Lisa'
+
+We calculate 2 simple conditions(which don't have any dependency on data from the left table) and nothing more.
+It does mean that we can move this calculation to the right table, it will make 3 improvements!
+
+1. Right table will be smaller -> smaller RAM usage -> better cache hits
+2. We will calculate our conditions over a smaller data set. In the right table we have only 10 million rows and after joining because of the left table we have 2 billion rows -> 200 times improvement!
+3. Our resulting table after JOIN will not have an expensive String column, only 1 byte UInt8 instead -> less copy of data in memory.
+
+Let's do it:
+
+There are several ways to rewrite that query, let's not bother with simple once and go straight to most optimized:
+
+Put our 2 conditions in hand-made bitmask:
+
+In order to do that we will take our conditions and multiply them by
+
+```
+(c_first_name = 'James') + (2 * (c_first_name = 'Lisa')
+
+C_first_name | (c_first_name = 'James') + (2 * (c_first_name = 'Lisa')
+ James | 00000001
+ Lisa | 00000010
+```
+
+As you can see, if you do it in that way, your conditions will not interfere with each other!
+But we need to be careful with the wideness of the resulting numeric type.
+Let's write our calculations in type notation:
+`UInt8 + UInt8*2 -> UInt8 + UInt16 -> UInt32`
+
+But we actually do not use more than first 2 bits, so we need to cast this expression back to UInt8
+
+Last thing to do is use the bitTest function in order to get the result of our condition by its position.
+
+And resulting query is:
+
+```sql
+SELECT
+ sumIf(ss_sales_price, bitTest(customer.cond, 0)) AS sum_James,
+ sumIf(ss_sales_price, bitTest(customer.cond, 1)) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+LEFT JOIN
+(
+ SELECT
+ c_customer_sk,
+ ((c_first_name = 'James') + (2 * (c_first_name = 'Lisa')))::UInt8 AS cond FROM customer
+ WHERE (c_first_name = 'James') OR (c_first_name = 'Lisa')
+) AS customer ON store_sales.ss_customer_sk = customer.c_customer_sk
+
+1 row in set. Elapsed: 31.699 sec. Processed 2.89 billion rows, 40.60 GB (91.23 million rows/s., 1.28 GB/s.)
+```
+
+### Reduce key column size
+
+But can we make something with our JOIN key column?
+
+It's type is Nullable(UInt64)
+
+Let's check if we really need to have a 0…18446744073709551615 range for our customer id, it sure looks like that we have much less people on earth than this number. The same about Nullable trait, we don’t care about Nulls in customer_id
+
+SELECT max(c_customer_sk) FROM customer
+
+For sure, we don't need that wide type.
+Lets remove Nullable trait and cast column to UInt32, twice smaller in byte size compared to UInt64.
+
+```sql
+SELECT
+ sumIf(ss_sales_price, bitTest(customer.cond, 0)) AS sum_James,
+ sumIf(ss_sales_price, bitTest(customer.cond, 1)) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+LEFT JOIN
+(
+ SELECT
+ CAST(c_customer_sk, 'UInt32') AS c_customer_sk,
+ (c_first_name = 'James') + (2 * (c_first_name = 'Lisa')) AS cond
+ FROM customer
+ WHERE (c_first_name = 'James') OR (c_first_name = 'Lisa')
+) AS customer ON store_sales.ss_customer_sk_nn = customer.c_customer_sk
+
+1 row in set. Elapsed: 27.093 sec. Processed 2.89 billion rows, 26.20 GB (106.74 million rows/s., 967.16 MB/s.)
+```
+
+Another 10% perf improvement from using UInt32 key instead of Nullable(Int64)
+Looks pretty neat, we almost got 10 times improvement over our initial query.
+Can we do better?
+
+Probably, but it does mean that we need to get rid of JOIN.
+
+## Use IN clause instead of JOIN
+
+Despite that all DBMS support ~ similar feature set, feature performance on different database are different:
+
+Small example, for PostgreSQL, is recommended to replace big IN clauses with JOINs, because IN clauses have bad performance.
+But for ClickHouse it's the opposite!, IN works faster than JOIN, because it only checks key existence in HashSet and doesn't need to extract any data from the right table in IN.
+
+Let's test that:
+
+```sql
+SELECT
+ sumIf(ss_sales_price, ss_customer_sk IN (
+ SELECT c_customer_sk
+ FROM customer
+ WHERE c_first_name = 'James'
+ )) AS sum_James,
+ sumIf(ss_sales_price, ss_customer_sk IN (
+ SELECT c_customer_sk
+ FROM customer
+ WHERE c_first_name = 'Lisa'
+ )) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+1 row in set. Elapsed: 16.546 sec. Processed 2.90 billion rows, 40.89 GB (175.52 million rows/s., 2.47 GB/s.)
+```
+
+Almost 2 times faster than our previous record with JOIN, what if we will improve the same hint with c_customer_sk key like in JOIN?
+
+```sql
+SELECT
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_first_name = 'James'
+ )) AS sum_James,
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_first_name = 'Lisa'
+ )) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+1 row in set. Elapsed: 12.355 sec. Processed 2.90 billion rows, 26.49 GB (235.06 million rows/s., 2.14 GB/s.)
+```
+
+Another 25% performance!
+
+But, there is one big limitation with IN approach, what if we have more than just 2 conditions?
+
+```sql
+SELECT
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_first_name = 'James'
+ )) AS sum_James,
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_first_name = 'Lisa'
+ )) AS sum_Lisa,
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_last_name = 'Smith'
+ )) AS sum_Smith,
+ sumIf(ss_sales_price, ss_customer_sk_nn IN (
+ SELECT c_customer_sk::UInt32
+ FROM customer
+ WHERE c_last_name = 'Williams'
+ )) AS sum_Williams,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+1 row in set. Elapsed: 23.690 sec. Processed 2.93 billion rows, 27.06 GB (123.60 million rows/s., 1.14 GB/s.)
+```
+
+## Adhoc alternative to Dictionary with FLAT layout
+
+But first is a short introduction. What the hell is a Dictionary with a FLAT layout?
+
+Basically, it's just a set of Array's for each attribute where the value position in the attribute array is just a dictionary key
+For sure it put heavy limitation about what dictionary key could be, but it gives really good advantages:
+
+`['Alice','James', 'Robert','John', ...].length = 12mil, Memory usage ~ N*sum(sizeOf(String(N)) + 1)`
+
+It's really small memory usage (good cache hit rate) & really fast key lookups (no complex hash calculation)
+
+So, if it's that great what are the caveats?
+First one is that your keys should be ideally autoincremental (with small number of gaps)
+And for second, lets look in that simple query and write down all calculations:
+
+```sql
+SELECT sumIf(ss_sales_price, dictGet(...) = 'James')
+```
+
+1. Dictionary call (2 billion times)
+2. String equality check (2 billion times)
+
+Although it's really efficient in terms of dictGet call and memory usage by Dictionary, it still materializes the String column (memcpy) and we pay a penalty of execution condition on top of such a string column for each row.
+
+But what if we could first calculate our required condition and create such a "Dictionary" ad hoc in query time?
+
+And we can actually do that!
+But let's repeat our analysis again:
+
+```sql
+SELECT sumIf(ss_sales_price, here_lives_unicorns(dictGet(...) = 'James'))
+```
+
+`['Alice','James', 'Lisa','James', ...].map(x -> multiIf(x = 'James', 1, x = 'Lisa', 2, 0)) => [0,1,2,1,...].length` = 12mil, Memory usage ~ `N*sizeOf(UInt8)` <- It's event smaller than FLAT dictionary
+
+And actions:
+
+1. String equality check (12 million times)
+2. Create Array (12 million elements)
+3. Array call (2 billion times)
+4. UInt8 equality check (2 billion times)
+
+But what is `here_lives_unicorns` function, does it exist in ClickHouse?
+
+No, but we can hack it with some array manipulation:
+
+```sql
+SELECT sumIf(ss_sales_price, arr[customer_id] = 2)
+
+WITH (
+ SELECT groupArray(assumeNotNull((c_first_name = 'James') + (2 * (c_first_name = 'Lisa')))::UInt8)
+ FROM
+ (
+ SELECT *
+ FROM customer
+ ORDER BY c_customer_sk ASC
+ )
+ ) AS cond
+SELECT
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk], 0)) AS sum_James,
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk], 1)) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+1 row in set. Elapsed: 13.006 sec. Processed 2.89 billion rows, 40.60 GB (222.36 million rows/s., 3.12 GB/s.)
+
+WITH (
+ SELECT groupArray(assumeNotNull((c_first_name = 'James') + (2 * (c_first_name = 'Lisa')))::UInt8)
+ FROM
+ (
+ SELECT *
+ FROM customer
+ ORDER BY c_customer_sk ASC
+ )
+ ) AS cond,
+ bitTest(cond[ss_customer_sk_nn], 0) AS cond_james,
+ bitTest(cond[ss_customer_sk_nn], 1) AS cond_lisa
+SELECT
+ sumIf(ss_sales_price, cond_james) AS sum_James,
+ sumIf(ss_sales_price, cond_lisa) AS sum_Lisa,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+
+1 row in set. Elapsed: 10.054 sec. Processed 2.89 billion rows, 26.20 GB (287.64 million rows/s., 2.61 GB/s.)
+```
+
+20% faster than the IN approach, what if we will have not 2 but 4 such conditions:
+
+```sql
+WITH (
+ SELECT groupArray(assumeNotNull((((c_first_name = 'James') + (2 * (c_first_name = 'Lisa'))) + (4 * (c_last_name = 'Smith'))) + (8 * (c_last_name = 'Williams')))::UInt8)
+ FROM
+ (
+ SELECT *
+ FROM customer
+ ORDER BY c_customer_sk ASC
+ )
+ ) AS cond
+SELECT
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk_nn], 0)) AS sum_James,
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk_nn], 1)) AS sum_Lisa,
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk_nn], 2)) AS sum_Smith,
+ sumIf(ss_sales_price, bitTest(cond[ss_customer_sk_nn], 3)) AS sum_Williams,
+ sum(ss_sales_price) AS sum_total
+FROM store_sales
+
+1 row in set. Elapsed: 11.454 sec. Processed 2.89 billion rows, 26.39 GB (252.49 million rows/s., 2.30 GB/s.)
+```
+
+As we can see, that Array approach doesn't even notice that we increased the amount of conditions by 2 times.
diff --git a/content/en/altinity-kb-queries-and-syntax/jsonextract-to-parse-many-attributes-at-a-time.md b/content/en/altinity-kb-queries-and-syntax/jsonextract-to-parse-many-attributes-at-a-time.md
index db5348511a..38fb071d87 100644
--- a/content/en/altinity-kb-queries-and-syntax/jsonextract-to-parse-many-attributes-at-a-time.md
+++ b/content/en/altinity-kb-queries-and-syntax/jsonextract-to-parse-many-attributes-at-a-time.md
@@ -4,6 +4,9 @@ linkTitle: "JSONExtract to parse many attributes at a time"
description: >
JSONExtract to parse many attributes at a time
---
+
+Don't use several JSONExtract for parsing big JSON. It's very ineffective, slow, and consumes CPU. Try to use one JSONExtract to parse String to Tupes and next get the needed elements:
+
```sql
WITH JSONExtract(json, 'Tuple(name String, id String, resources Nested(description String, format String, tracking_summary Tuple(total UInt32, recent UInt32)), extras Nested(key String, value String))') AS parsed_json
SELECT
@@ -15,3 +18,83 @@ SELECT
tupleElement(tupleElement(tupleElement(parsed_json, 'resources'), 'tracking_summary'), 'recent') AS `resources.tracking_summary.recent`
FROM url('https://raw.githubusercontent.com/jsonlines/guide/master/datagov100.json', 'JSONAsString', 'json String')
```
+However, such parsing requires static schema - all keys should be presented in every row, or you will get an empty structure. More dynamic parsing requires several JSONExtract invocations, but still - try not to scan the same data several times:
+
+```sql
+WITH
+ '{"timestamp":"2024-06-12T14:30:00.001Z","functionality":"DOCUMENT","flowId":"210abdee-6de5-474a-83da-748def0facc1","step":"BEGIN","env":"dev","successful":true,"data":{"action":"initiate_view","stats":{"total":1,"success":1,"failed":0},"client_ip":"192.168.1.100","client_port":"8080"}}' AS json,
+ JSONExtractKeysAndValues(json, 'String') AS m,
+ mapFromArrays(m.1, m.2) AS p
+SELECT
+ extractKeyValuePairs(p['data'])['action'] AS data,
+ (p['successful']) = 'true' AS successful
+FORMAT Vertical
+
+/*
+Row 1:
+──────
+data: initiate_view
+successful: 1
+*/
+
+```
+
+A good approach to get a proper schema from a json message is to let `clickhouse-local` schema inference do the job:
+
+```bash
+$ ls example_message.json
+example_message.json
+
+$ clickhouse-local --query="DESCRIBE file('example_message.json', 'JSONEachRow')" --format="Vertical";
+
+Row 1:
+──────
+name: resourceLogs
+type: Array(Tuple(
+ resource Nullable(String),
+ scopeLogs Array(Tuple(
+ logRecords Array(Tuple(
+ attributes Array(Tuple(
+ key Nullable(String),
+ value Tuple(
+ stringValue Nullable(String)))),
+ body Tuple(
+ stringValue Nullable(String)),
+ observedTimeUnixNano Nullable(String),
+ spanId Nullable(String),
+ traceId Nullable(String))),
+ scope Nullable(String)))))
+```
+
+For very subnested dynamic JSON files, if you don't need all the keys, you could parse sublevels specifically. Still this will require several JSONExtract calls but each call will have less data to parse so complexity will be reduced for each pass: O(log n)
+
+```sql
+CREATE TABLE better_parsing (json String) ENGINE = Memory;
+INSERT INTO better_parsing FORMAT JSONAsString {"timestamp":"2024-06-12T14:30:00.001Z","functionality":"DOCUMENT","flowId":"210abdee-6de5-474a-83da-748def0facc1","step":"BEGIN","env":"dev","successful":true,"data":{"action":"initiate_view","stats":{"total":1,"success":1,"failed":0},"client_ip":"192.168.1.100","client_port":"8080"}}
+
+WITH parsed_content AS
+ (
+ SELECT
+ JSONExtractKeysAndValues(json, 'String') AS 1st_level_arr,
+ mapFromArrays(1st_level_arr.1, 1st_level_arr.2) AS 1st_level_map,
+ JSONExtractKeysAndValues(1st_level_map['data'], 'String') AS 2nd_level_arr,
+ mapFromArrays(2nd_level_arr.1, 2nd_level_arr.2) AS 2nd_level_map,
+ JSONExtractKeysAndValues(2nd_level_map['stats'], 'String') AS 3rd_level_arr,
+ mapFromArrays(3rd_level_arr.1, 3rd_level_arr.2) AS 3rd_level_map
+ FROM json_tests.better_parsing
+ )
+SELECT
+ 1st_level_map['timestamp'] AS timestamp,
+ 2nd_level_map['action'] AS action,
+ 3rd_level_map['total'] AS total
+ 3rd_level_map['nokey'] AS no_key_empty
+FROM parsed_content
+
+/*
+ ┌─timestamp────────────────┬─action────────┬─total─┬─no_key_empty─┐
+1. │ 2024-06-12T14:30:00.001Z │ initiate_view │ 1 │ │
+ └──────────────────────────┴───────────────┴───────┴──────────────┘
+
+1 row in set. Elapsed: 0.003 sec.
+*/
+```
diff --git a/content/en/altinity-kb-queries-and-syntax/lag-lead.md b/content/en/altinity-kb-queries-and-syntax/lag-lead.md
index 8db5c6f649..9ca10840ea 100644
--- a/content/en/altinity-kb-queries-and-syntax/lag-lead.md
+++ b/content/en/altinity-kb-queries-and-syntax/lag-lead.md
@@ -59,7 +59,7 @@ order by g, a;
└───┴────────────┴────────────┴────────────┘
```
-## Using window functions (starting from Clickhouse 21.3)
+## Using window functions (starting from ClickHouse® 21.3)
```sql
SET allow_experimental_window_functions = 1;
diff --git a/content/en/altinity-kb-queries-and-syntax/literal-decimal-or-float.md b/content/en/altinity-kb-queries-and-syntax/literal-decimal-or-float.md
index b2246cfe47..cb8cad57ec 100644
--- a/content/en/altinity-kb-queries-and-syntax/literal-decimal-or-float.md
+++ b/content/en/altinity-kb-queries-and-syntax/literal-decimal-or-float.md
@@ -20,6 +20,21 @@ SELECT
└─────────────────┴─────────────┴────────────────────┴─────────────────────┘
```
+
+> When we try to type cast 64.32 to Decimal128(2) the resulted value is 64.31.
+
+When it sees a number with a decimal separator it interprets as `Float64` literal (where `64.32` have no accurate representation, and actually you get something like `64.319999999999999999`) and later that Float is casted to Decimal by removing the extra precision.
+
+Workaround is very simple - wrap the number in quotes (and it will be considered as a string literal by query parser, and will be transformed to Decimal directly), or use postgres-alike casting syntax:
+
+```sql
+select cast(64.32,'Decimal128(2)') a, cast('64.32','Decimal128(2)') b, 64.32::Decimal128(2) c;
+
+┌─────a─┬─────b─┬─────c─┐
+│ 64.31 │ 64.32 │ 64.32 │
+└───────┴───────┴───────┘
+```
+
## Float64
```sql
diff --git a/content/en/altinity-kb-queries-and-syntax/machine-learning-in-clickhouse.md b/content/en/altinity-kb-queries-and-syntax/machine-learning-in-clickhouse.md
index 06b52f0340..0fafdb833e 100644
--- a/content/en/altinity-kb-queries-and-syntax/machine-learning-in-clickhouse.md
+++ b/content/en/altinity-kb-queries-and-syntax/machine-learning-in-clickhouse.md
@@ -4,8 +4,9 @@ linkTitle: "Machine learning in ClickHouse"
description: >
Machine learning in ClickHouse
---
-[https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup31/ml.pdf](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup31/ml.pdf)
-[CatBoost / MindsDB / Fast.ai]({{}})
+Resources
-[https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/forecast.pdf](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/forecast.pdf)
+* [Machine Learning in ClickHouse](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup31/ml.pdf) - Presentation from 2019 (Meetup 31)
+* [ML discussion: CatBoost / MindsDB / Fast.ai](../../altinity-kb-integrations/catboost-mindsdb-fast.ai) - Brief article from 2021
+* [Machine Learning Forecase (Russian)](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup38/forecast.pdf) - Presentation from 2019 (Meetup 38)
diff --git a/content/en/altinity-kb-queries-and-syntax/multiple-date-column-in-partition-key.md b/content/en/altinity-kb-queries-and-syntax/multiple-date-column-in-partition-key.md
index 86ef6e0efd..9719b0a2e0 100644
--- a/content/en/altinity-kb-queries-and-syntax/multiple-date-column-in-partition-key.md
+++ b/content/en/altinity-kb-queries-and-syntax/multiple-date-column-in-partition-key.md
@@ -18,7 +18,7 @@ CREATE TABLE part_key_multiple_dates
`inserted_at` DateTime
)
ENGINE = MergeTree
-PARTITION BY (toYYYYMM(date), ignore(created_at), ignore(inserted_at))
+PARTITION BY (toYYYYMM(date), ignore(created_at, inserted_at))
ORDER BY (key, time);
diff --git a/content/en/altinity-kb-queries-and-syntax/mutations.md b/content/en/altinity-kb-queries-and-syntax/mutations.md
index 9b5093eef1..448e698933 100644
--- a/content/en/altinity-kb-queries-and-syntax/mutations.md
+++ b/content/en/altinity-kb-queries-and-syntax/mutations.md
@@ -4,7 +4,7 @@ linkTitle: "Mutations"
description: >
ALTER UPDATE / DELETE
---
-Q. How to know if `ALTER TABLE … DELETE/UPDATE mutation ON CLUSTER` was finished successfully on all the nodes?
+## How to know if `ALTER TABLE … DELETE/UPDATE mutation ON CLUSTER` was finished successfully on all the nodes?
A. mutation status in system.mutations is local to each replica, so use
@@ -14,3 +14,27 @@ SELECT hostname(), * FROM clusterAllReplicas('your_cluster_name', system.mutatio
```
Look on `is_done` and `latest_fail_reason` columns
+
+## Are mutations being run in parallel or they are sequential in ClickHouse® (in scope of one table)
+
+
+
+ClickHouse runs mutations sequentially, but it can combine several mutations in a single and apply all of them in one merge.
+Sometimes, it can lead to problems, when a combined expression which ClickHouse needs to execute becomes really big. (If ClickHouse combined thousands of mutations in one)
+
+
+Because ClickHouse stores data in independent parts, ClickHouse is able to run mutation(s) merges for each part independently and in parallel.
+It also can lead to high resource utilization, especially memory usage if you use `x IN (SELECT ... FROM big_table)` statements in mutation, because each merge will run and keep in memory its own HashSet. You can avoid this problem, if you will use [Dictionary approach](../update-via-dictionary) for such mutations.
+
+Parallelism of mutations controlled by settings:
+
+```sql
+SELECT *
+FROM system.merge_tree_settings
+WHERE name LIKE '%mutation%'
+
+┌─name───────────────────────────────────────────────┬─value─┬─changed─┬─description──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─type───┐
+│ max_replicated_mutations_in_queue │ 8 │ 0 │ How many tasks of mutating parts are allowed simultaneously in ReplicatedMergeTree queue. │ UInt64 │
+│ number_of_free_entries_in_pool_to_execute_mutation │ 20 │ 0 │ When there is less than specified number of free entries in pool, do not execute part mutations. This is to leave free threads for regular merges and avoid "Too many parts" │ UInt64 │
+└────────────────────────────────────────────────────┴───────┴─────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┘
+```
diff --git a/content/en/altinity-kb-queries-and-syntax/pivot-unpivot.md b/content/en/altinity-kb-queries-and-syntax/pivot-unpivot.md
index 52d5996f6e..a2847e045c 100644
--- a/content/en/altinity-kb-queries-and-syntax/pivot-unpivot.md
+++ b/content/en/altinity-kb-queries-and-syntax/pivot-unpivot.md
@@ -3,6 +3,9 @@ title: "PIVOT / UNPIVOT"
linkTitle: "PIVOT / UNPIVOT"
description: >
PIVOT / UNPIVOT
+keywords:
+ - clickhouse pivot
+ - clickhouse unpivot
---
## PIVOT
@@ -12,7 +15,7 @@ CREATE TABLE sales(suppkey UInt8, category String, quantity UInt32) ENGINE=Memor
INSERT INTO sales VALUES (2, 'AA' ,7500),(1, 'AB' , 4000),(1, 'AA' , 6900),(1, 'AB', 8900), (1, 'AC', 8300), (1, 'AA', 7000), (1, 'AC', 9000), (2,'AA', 9800), (2,'AB', 9600), (1,'AC', 8900),(1, 'AD', 400), (2,'AD', 900), (2,'AD', 1200), (1,'AD', 2600), (2, 'AC', 9600),(1, 'AC', 6200);
```
-### Using Map data type (starting from Clickhouse 21.1)
+### Using Map data type (starting from ClickHouse® 21.1)
```sql
WITH CAST(sumMap([category], [quantity]), 'Map(String, UInt32)') AS map
@@ -122,18 +125,14 @@ ORDER BY suppkey ASC
│ 3 │ BRAND_C │ AC │ 6900 │
│ 3 │ BRAND_C │ AD │ 3400 │
└─────────┴─────────┴──────────┴──────────┘
-```
-
-### Using tupleToNameValuePairs (starting from ClickHouse 21.9)
-```sql
SELECT
suppkey,
brand,
tpl.1 AS category,
tpl.2 AS quantity
FROM sales_w
-ARRAY JOIN tupleToNameValuePairs((AA, AB, AC, AD)) AS tpl
+ARRAY JOIN tupleToNameValuePairs(CAST((AA, AB, AC, AD), 'Tuple(AA UInt32, AB UInt32, AC UInt32, AD UInt32)')) AS tpl
ORDER BY suppkey ASC
┌─suppkey─┬─brand───┬─category─┬─quantity─┐
@@ -151,3 +150,4 @@ ORDER BY suppkey ASC
│ 3 │ BRAND_C │ AD │ 3400 │
└─────────┴─────────┴──────────┴──────────┘
```
+
diff --git a/content/en/altinity-kb-queries-and-syntax/projections-examples.md b/content/en/altinity-kb-queries-and-syntax/projections-examples.md
index 9ce1da7a05..76803bdedf 100644
--- a/content/en/altinity-kb-queries-and-syntax/projections-examples.md
+++ b/content/en/altinity-kb-queries-and-syntax/projections-examples.md
@@ -1,10 +1,203 @@
---
-title: "Projections examples"
-linkTitle: "Projections examples"
+title: "ClickHouse® Projections"
+linkTitle: "ClickHouse Projections"
description: >
- Projections examples
+ Using this ClickHouse feature to optimize queries
+keywords:
+ - clickhouse projections
+ - clickhouse projection vs materialized view
---
-## Aggregating projections
+
+Projections in ClickHouse act as inner tables within a main table, functioning as a mechanism to optimize queries by using these inner tables when only specific columns are needed. Essentially, a projection is similar to a [Materialized View](/altinity-kb-schema-design/materialized-views/) with an [AggregatingMergeTree engine](/engines/mergetree-table-engine-family/aggregatingmergetree/), designed to be automatically populated with relevant data.
+
+However, too many projections can lead to excess storage, much like overusing Materialized Views. Projections share the same lifecycle as the main table, meaning they are automatically backfilled and don’t require query rewrites, which is particularly advantageous when integrating with BI tools.
+
+Projection parts are stored within the main table parts, and their merges occur simultaneously as the main table merges, ensuring data consistency without additional maintenance.
+
+compared to a separate table+MV setup:
+- A separate table gives you more freedom (like partitioning, granularity, etc), but projections - more consistency (parts managed as a whole)
+- Projections do not support many features (like indexes and FINAL). That becomes better with recent versions, but still a drawback
+
+The design approach for projections is the same as for indexes. Create a table and give it to users. If you encounter a slower query, add a projection for that particular query (or set of similar queries). You can create 10+ projections per table, materialize, drop, etc - the very same as indexes. You exchange query speed for disk space/IO and CPU needed to build and rebuild projections on merges.
+
+## Links
+
+* Amos Bird - kuaishou.com - Projections in ClickHouse. [slides](https://github.com/ClickHouse/clickhouse-presentations/blob/master/percona2021/projections.pdf). [video](https://youtu.be/jJ5VuLr2k5k?list=PLWhC0zeznqkkNYzcvHEfZ8hly3Cu9ojKk)
+* [Documentation](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/mergetree/#projections)
+* [tinybird blog article](https://blog.tinybird.co/2021/07/09/projections/)
+* ClickHouse presentation on Projections https://www.youtube.com/watch?v=QDAJTKZT8y4
+* Blog video https://clickhouse.com/videos/how-to-a-clickhouse-query-using-projections
+
+
+## Why is a ClickHouse projection not used?
+
+A query analyzer should have a reason for using a projection and should not have any limitation to do so.
+
+- the query should use ONLY the columns defined in the projection.
+- There should be a lot of data to read from the main table (gigabytes)
+- for ORDER BY projection WHERE statement referring to a column should be in the query
+- FINAL queries do not work with projections.
+- tables with DELETEd rows do not work with projections. This is because rows in a projection may be affected by a DELETE operation. But there is a MergeTree setting lightweight_mutation_projection_mode to change the behavior (Since 24.7)
+- Projection is used only if it is cheaper to read from it than from the table (expected amount of rows and GBs read is smaller)
+- Projection should be materialized. Verify that all parts have the needed projection by comparing system.parts and system.projection_parts (see query below)
+- a bug in a Clickhouse version. Look at [changelog](https://clickhouse.com/docs/whats-new/changelog) and search for projection.
+- If there are many projections per table, the analyzer can select any of them. If you think that it is better, use settings `preferred_optimize_projection_name` or `force_optimize_projection_name`
+- If expressions are used instead of plain column names, the query should use the exact expression as defined in the projection with the same functions and modifiers. Use column aliases to make the query the very same as in the projection definition:
+
+```sql
+CREATE TABLE test
+(
+ a Int64,
+ ts DateTime,
+ week alias toStartOfWeek(ts),
+ PROJECTION weekly_projection
+ (
+ SELECT week, sum(a) group by week
+ )
+)
+ENGINE = MergeTree ORDER BY a;
+
+insert into test
+select number, now()-number*100
+from numbers(1e7);
+
+--explain indexes=1
+select week, sum(a) from test group by week
+settings force_optimize_projection=1;
+```
+
+https://fiddle.clickhouse.com/7f331eb2-9408-4813-9c67-caef4cdd227d
+
+Explain result: ReadFromMergeTree (weekly_projection)
+
+```
+Expression ((Project names + Projection))
+ Aggregating
+ Expression
+ ReadFromMergeTree (weekly_projection)
+ Indexes:
+ PrimaryKey
+ Condition: true
+ Parts: 9/9
+ Granules: 9/1223
+```
+
+## check parts
+
+- has the projection materialized
+- does not have lightweight deletes
+
+```
+SELECT
+ p.database AS base_database,
+ p.table AS base_table,
+ p.name AS base_part_name, -- Name of the part in the base table
+ p.has_lightweight_delete,
+ pp.active
+FROM system.parts AS p -- Alias for the base table's parts
+LEFT JOIN system.projection_parts AS pp -- Alias for the projection's parts
+ON p.database = pp.database AND p.table = pp.table
+ AND p.name = pp.parent_name
+ AND pp.name = 'projection'
+WHERE
+ p.database = 'database'
+ AND p.table = 'table'
+ AND p.active -- Consider only active parts of the base table
+ -- and not pp.active -- see only missed in the list
+ORDER BY p.database, p.table, p.name;
+
+```
+
+## Recalculate on Merge
+
+What happens in the case of non-trivial background merges in ReplacingMergeTree, AggregatingMergeTree and similar, and OPTIMIZE table DEDUPLICATE queries?
+
+* Before version 24.8, projections became out of sync with the main data.
+* Since version 24.8, it is controlled by a new table-level setting: [deduplicate_merge_projection_mode](https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#deduplicate_merge_projection_mode) = `throw`/`drop`/`rebuild`
+* Somewhere later (before 25.3) `ignore` option was introduced. It can be helpful for cases when SummingMergeTree is used with Projections and no DELETE operation in any flavor (Replacing/Collapsing/DELETE/ALTER DELETE) is executed over the table.
+
+However, projection usage is still disabled for FINAL queries. So, you have to use OPTIMIZE FINAL or SELECT ...GROUP BY instead of FINAL for fighting duplicates between parts
+
+```
+CREATE TABLE users (uid Int16, name String, version Int16,
+ projection xx (
+ select name,uid,version order by name
+ )
+) ENGINE=ReplacingMergeTree order by uid
+settings deduplicate_merge_projection_mode='rebuild'
+ ;
+
+INSERT INTO users
+SELECT
+ number AS uid,
+ concat('User_', toString(uid)) AS name,
+ 1 AS version
+FROM numbers(100000);
+
+INSERT INTO users
+SELECT
+ number AS uid,
+ concat('User_', toString(uid)) AS name,
+ 2 AS version
+FROM numbers(100000);
+
+SELECT 'duplicate',name,uid,version FROM users
+where name ='User_98304'
+settings force_optimize_projection=1 ;
+
+SELECT 'dedup by group by/limit 1 by',name,uid,version FROM users
+where name ='User_98304'
+order by version DESC
+limit 1 by uid
+settings force_optimize_projection=1
+;
+
+optimize table users final ;
+
+SELECT 'dedup after optimize',name,uid,version FROM users
+where name ='User_98304'
+settings force_optimize_projection=1 ;
+
+```
+https://fiddle.clickhouse.com/e1977a66-09ce-43c4-aabc-508c957d44d7
+
+
+## System tables
+
+- system.projections
+- system.projection_parts
+- system.projection_parts_columns
+
+```
+SELECT
+ database,
+ table,
+ name,
+ formatReadableSize(sum(data_compressed_bytes) AS size) AS compressed,
+ formatReadableSize(sum(data_uncompressed_bytes) AS usize) AS uncompressed,
+ round(usize / size, 2) AS compr_rate,
+ sum(rows) AS rows,
+ count() AS part_count
+FROM system.projection_parts
+WHERE active
+GROUP BY
+ database,
+ table,
+ name
+ORDER BY size DESC;
+```
+
+## How to receive a list of tables with projections?
+
+```
+select database, table from system.tables
+where create_table_query ilike '%projection%'
+ and database <> 'system'
+```
+
+## Examples
+
+### Aggregating ClickHouse projections
```sql
create table z(Browser String, Country UInt8, F Float64)
@@ -61,9 +254,9 @@ group by Browser,Country format Null;
Elapsed: 0.005 sec. Processed 22.43 thousand rows
```
-## Emulation of an inverted index using orderby projection
+### Emulation of an inverted index using orderby projection
-You can create an `orderby projection` and include all columns of a table, but if a table is very wide it will double of stored data. This expample demonstrate a trick, we create an `orderby projection` and include primary key columns and the target column and sort by the target column. This allows using subquery to find primary key values and after that to query the table using the primary key.
+You can create an `orderby projection` and include all columns of a table, but if a table is very wide it will double the amount of stored data. This example demonstrate a trick, we create an `orderby projection` and include primary key columns and the target column and sort by the target column. This allows using subquery to find [primary key values](../../engines/mergetree-table-engine-family/pick-keys/) and after that to query the table using the primary key.
```sql
CREATE TABLE test_a
@@ -112,8 +305,4 @@ VS
**Elapsed: 0.013 sec. Processed 32.77 thousand rows** -- optimized
-## See also
-* Amos Bird - kuaishou.com - Projections in ClickHouse. [slides](https://github.com/ClickHouse/clickhouse-presentations/blob/master/percona2021/projections.pdf). [video](https://youtu.be/jJ5VuLr2k5k?list=PLWhC0zeznqkkNYzcvHEfZ8hly3Cu9ojKk)
-* [Documentation](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/mergetree/#projections)
-* [tinybird blog article](https://blog.tinybird.co/2021/07/09/projections/)
diff --git a/content/en/altinity-kb-queries-and-syntax/roaring-bitmaps-for-calculating-retention.md b/content/en/altinity-kb-queries-and-syntax/roaring-bitmaps-for-calculating-retention.md
index 394bb5e16e..074e25ac4c 100644
--- a/content/en/altinity-kb-queries-and-syntax/roaring-bitmaps-for-calculating-retention.md
+++ b/content/en/altinity-kb-queries-and-syntax/roaring-bitmaps-for-calculating-retention.md
@@ -41,4 +41,4 @@ WHERE h IN (0, 1)
└──────┴───────┘
```
-See also [https://cdmana.com/2021/01/20210109005922716t.html](https://cdmana.com/2021/01/20210109005922716t.html)
+See also [A primer on roaring bitmaps](https://vikramoberoi.com/a-primer-on-roaring-bitmaps-what-they-are-and-how-they-work/)
diff --git a/content/en/altinity-kb-queries-and-syntax/row_policy_using_dictionary.md b/content/en/altinity-kb-queries-and-syntax/row_policy_using_dictionary.md
new file mode 100644
index 0000000000..1c945c79ae
--- /dev/null
+++ b/content/en/altinity-kb-queries-and-syntax/row_policy_using_dictionary.md
@@ -0,0 +1,346 @@
+---
+title: "Row policies overhead (hiding 'removed' tenants)"
+linkTitle: "Row policies overhead"
+weight: 100
+description: >
+ One more approach to hide (delete) rows in ClickHouse®
+---
+
+## No row policy
+
+```sql
+CREATE TABLE test_delete
+(
+ tenant Int64,
+ key Int64,
+ ts DateTime,
+ value_a String
+)
+ENGINE = MergeTree
+PARTITION BY toYYYYMM(ts)
+ORDER BY (tenant, key, ts);
+
+INSERT INTO test_delete
+SELECT
+ number%5,
+ number,
+ toDateTime('2020-01-01')+number/10,
+ concat('some_looong_string', toString(number)),
+FROM numbers(1e8);
+
+INSERT INTO test_delete -- multiple small tenants
+SELECT
+ number%5000,
+ number,
+ toDateTime('2020-01-01')+number/10,
+ concat('some_looong_string', toString(number)),
+FROM numbers(1e8);
+```
+
+```sql
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 1 │ 20020000 │
+│ 2 │ 20020000 │
+│ 3 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.285 sec. Processed 200.00 million rows, 1.60 GB (702.60 million rows/s., 5.62 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.265 sec. Processed 20.23 million rows, 863.93 MB (76.33 million rows/s., 3.26 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.062 sec. Processed 20.23 million rows, 242.31 MB (324.83 million rows/s., 3.89 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.009 sec. Processed 212.99 thousand rows, 1.80 MB (24.39 million rows/s., 206.36 MB/s.)
+```
+
+## row policy using expression
+
+```sql
+CREATE ROW POLICY pol1 ON test_delete USING tenant not in (1,2,3) TO all;
+
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+│ 6 │ 20000 │
+│ 7 │ 20000 │
+│ 8 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.333 sec. Processed 140.08 million rows, 1.12 GB (420.59 million rows/s., 3.36 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.287 sec. Processed 20.23 million rows, 863.93 MB (70.48 million rows/s., 3.01 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.080 sec. Processed 20.23 million rows, 242.31 MB (254.20 million rows/s., 3.05 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.011 sec. Processed 212.99 thousand rows, 3.44 MB (19.53 million rows/s., 315.46 MB/s.)
+
+Q5) SELECT uniq(value_a) FROM test_delete where tenant = 1;
+┌─uniq(value_a)─┐
+│ 0 │
+└───────────────┘
+1 row in set. Elapsed: 0.008 sec. Processed 180.22 thousand rows, 1.44 MB (23.69 million rows/s., 189.54 MB/s.)
+
+DROP ROW POLICY pol1 ON test_delete;
+```
+
+## row policy using table subquery
+
+```sql
+create table deleted_tenants(tenant Int64) ENGINE=MergeTree order by tenant;
+
+CREATE ROW POLICY pol1 ON test_delete USING tenant not in deleted_tenants TO all;
+
+SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 1 │ 20020000 │
+│ 2 │ 20020000 │
+│ 3 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.455 sec. Processed 200.00 million rows, 1.60 GB (439.11 million rows/s., 3.51 GB/s.)
+
+insert into deleted_tenants values(1),(2),(3);
+
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+│ 6 │ 20000 │
+│ 7 │ 20000 │
+│ 8 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.329 sec. Processed 140.08 million rows, 1.12 GB (426.34 million rows/s., 3.41 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.287 sec. Processed 20.23 million rows, 863.93 MB (70.56 million rows/s., 3.01 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.080 sec. Processed 20.23 million rows, 242.31 MB (251.39 million rows/s., 3.01 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.010 sec. Processed 213.00 thousand rows, 3.44 MB (20.33 million rows/s., 328.44 MB/s.)
+
+Q5) SELECT uniq(value_a) FROM test_delete where tenant = 1;
+┌─uniq(value_a)─┐
+│ 0 │
+└───────────────┘
+1 row in set. Elapsed: 0.008 sec. Processed 180.23 thousand rows, 1.44 MB (22.11 million rows/s., 176.90 MB/s.)
+
+DROP ROW POLICY pol1 ON test_delete;
+DROP TABLE deleted_tenants;
+```
+
+## row policy using external dictionary (NOT dictHas)
+
+```sql
+create table deleted_tenants(tenant Int64, deleted UInt8 default 1) ENGINE=MergeTree order by tenant;
+
+insert into deleted_tenants(tenant) values(1),(2),(3);
+
+CREATE DICTIONARY deleted_tenants_dict (tenant UInt64, deleted UInt8)
+PRIMARY KEY tenant SOURCE(CLICKHOUSE(TABLE deleted_tenants))
+LIFETIME(600) LAYOUT(FLAT());
+
+CREATE ROW POLICY pol1 ON test_delete USING NOT dictHas('deleted_tenants_dict', tenant) TO all;
+
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+│ 6 │ 20000 │
+│ 7 │ 20000 │
+│ 8 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.388 sec. Processed 200.00 million rows, 1.60 GB (515.79 million rows/s., 4.13 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.291 sec. Processed 20.23 million rows, 863.93 MB (69.47 million rows/s., 2.97 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.084 sec. Processed 20.23 million rows, 242.31 MB (240.07 million rows/s., 2.88 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.010 sec. Processed 212.99 thousand rows, 3.44 MB (21.45 million rows/s., 346.56 MB/s.)
+
+Q5) SELECT uniq(value_a) FROM test_delete where tenant = 1;
+┌─uniq(value_a)─┐
+│ 0 │
+└───────────────┘
+1 row in set. Elapsed: 0.046 sec. Processed 20.22 million rows, 161.74 MB (440.26 million rows/s., 3.52 GB/s.)
+
+DROP ROW POLICY pol1 ON test_delete;
+DROP DICTIONARY deleted_tenants_dict;
+DROP TABLE deleted_tenants;
+```
+
+## row policy using external dictionary (dictHas)
+
+```sql
+create table deleted_tenants(tenant Int64, deleted UInt8 default 1) ENGINE=MergeTree order by tenant;
+
+insert into deleted_tenants(tenant) select distinct tenant from test_delete where tenant not in (1,2,3);
+
+CREATE DICTIONARY deleted_tenants_dict (tenant UInt64, deleted UInt8)
+PRIMARY KEY tenant SOURCE(CLICKHOUSE(TABLE deleted_tenants))
+LIFETIME(600) LAYOUT(FLAT());
+
+CREATE ROW POLICY pol1 ON test_delete USING dictHas('deleted_tenants_dict', tenant) TO all;
+
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+│ 6 │ 20000 │
+│ 7 │ 20000 │
+│ 8 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.399 sec. Processed 200.00 million rows, 1.60 GB (501.18 million rows/s., 4.01 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.284 sec. Processed 20.23 million rows, 863.93 MB (71.30 million rows/s., 3.05 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.080 sec. Processed 20.23 million rows, 242.31 MB (251.88 million rows/s., 3.02 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.010 sec. Processed 212.99 thousand rows, 3.44 MB (22.01 million rows/s., 355.50 MB/s.)
+
+Q5) SELECT uniq(value_a) FROM test_delete where tenant = 1;
+┌─uniq(value_a)─┐
+│ 0 │
+└───────────────┘
+1 row in set. Elapsed: 0.034 sec. Processed 20.22 million rows, 161.74 MB (589.90 million rows/s., 4.72 GB/s.)
+
+DROP ROW POLICY pol1 ON test_delete;
+DROP DICTIONARY deleted_tenants_dict;
+DROP TABLE deleted_tenants;
+```
+
+## row policy using engine=Set
+```sql
+create table deleted_tenants(tenant Int64) ENGINE=Set;
+
+insert into deleted_tenants(tenant) values(1),(2),(3);
+
+CREATE ROW POLICY pol1 ON test_delete USING tenant not in deleted_tenants TO all;
+
+Q1) SELECT tenant, count() FROM test_delete GROUP BY tenant ORDER BY tenant LIMIT 6;
+┌─tenant─┬──count()─┐
+│ 0 │ 20020000 │
+│ 4 │ 20020000 │
+│ 5 │ 20000 │
+│ 6 │ 20000 │
+│ 7 │ 20000 │
+│ 8 │ 20000 │
+└────────┴──────────┘
+6 rows in set. Elapsed: 0.322 sec. Processed 200.00 million rows, 1.60 GB (621.38 million rows/s., 4.97 GB/s.)
+
+Q2) SELECT uniq(value_a) FROM test_delete where tenant = 4;
+┌─uniq(value_a)─┐
+│ 20016427 │
+└───────────────┘
+1 row in set. Elapsed: 0.275 sec. Processed 20.23 million rows, 863.93 MB (73.56 million rows/s., 3.14 GB/s.)
+
+Q3) SELECT max(ts) FROM test_delete where tenant = 4;
+┌─────────────max(ts)─┐
+│ 2020-04-25 17:46:39 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.084 sec. Processed 20.23 million rows, 242.31 MB (240.07 million rows/s., 2.88 GB/s.)
+
+Q4) SELECT max(ts) FROM test_delete where tenant = 4 and key = 444;
+┌─────────────max(ts)─┐
+│ 2020-01-01 00:00:44 │
+└─────────────────────┘
+1 row in set. Elapsed: 0.010 sec. Processed 212.99 thousand rows, 3.44 MB (20.69 million rows/s., 334.18 MB/s.)
+
+Q5) SELECT uniq(value_a) FROM test_delete where tenant = 1;
+┌─uniq(value_a)─┐
+│ 0 │
+└───────────────┘
+1 row in set. Elapsed: 0.030 sec. Processed 20.22 million rows, 161.74 MB (667.06 million rows/s., 5.34 GB/s.)
+
+DROP ROW POLICY pol1 ON test_delete;
+DROP TABLE deleted_tenants;
+```
+
+
+
+## results
+
+expression: `CREATE ROW POLICY pol1 ON test_delete USING tenant not in (1,2,3) TO all;`
+
+table subq: `CREATE ROW POLICY pol1 ON test_delete USING tenant not in deleted_tenants TO all;`
+
+ext. dict. NOT dictHas : `CREATE ROW POLICY pol1 ON test_delete USING NOT dictHas('deleted_tenants_dict', tenant) TO all;`
+
+ext. dict. dictHas :
+
+| Q | no policy | expression | table subq | ext. dict. NOT | ext. dict. | engine=Set |
+|----|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
+| Q1 | 0.285 / 200.00m | 0.333 / 140.08m | 0.329 / 140.08m | 0.388 / 200.00m | 0.399 / 200.00m | 0.322 / 200.00m |
+| Q2 | 0.265 / 20.23m | 0.287 / 20.23m | 0.287 / 20.23m | 0.291 / 20.23m | 0.284 / 20.23m | 0.275 / 20.23m |
+| Q3 | 0.062 / 20.23m | 0.080 / 20.23m | 0.080 / 20.23m | 0.084 / 20.23m | 0.080 / 20.23m | 0.084 / 20.23m |
+| Q4 | 0.009 / 212.99t | 0.011 / 212.99t | 0.010 / 213.00t | 0.010 / 212.99t | 0.010 / 212.99t | 0.010 / 212.99t |
+| Q5 | | 0.008 / 180.22t | 0.008 / 180.23t | 0.046 / 20.22m | 0.034 / 20.22m | 0.030 / 20.22m |
+
+Expression in row policy seems to be fastest way (Q1, Q5).
diff --git a/content/en/altinity-kb-queries-and-syntax/sampling-example.md b/content/en/altinity-kb-queries-and-syntax/sampling-example.md
index 4c28707bd6..b270199107 100644
--- a/content/en/altinity-kb-queries-and-syntax/sampling-example.md
+++ b/content/en/altinity-kb-queries-and-syntax/sampling-example.md
@@ -1,10 +1,11 @@
---
title: "Sampling Example"
linkTitle: "Sampling Example"
-description: >
- Clickhouse table sampling example
---
-The most important idea about sampling that the primary index must have **low cardinality**. The following example demonstrates how sampling can be setup correctly, and an example if it being set up incorrectly as a comparison.
+
+The most important idea about sampling that the primary index must have **LowCardinality**. (For more information, see [the Altinity Knowledge Base article on LowCardinality](../../altinity-kb-schema-design/lowcardinality) or [a ClickHouse® user\'s lessons learned from LowCardinality](https://altinity.com/blog/2020-5-20-reducing-clickhouse-storage-cost-with-the-low-cardinality-type-lessons-from-an-instana-engineer)).
+
+The following example demonstrates how sampling can be setup correctly, and an example if it being set up incorrectly as a comparison.
Sampling requires `sample by expression` . This ensures a range of sampled column types fit within a specified range, which ensures the requirement of low cardinality. In this example, I cannot use `transaction_id` because I can not ensure that the min value of `transaction_id = 0` and `max value = MAX_UINT64`. Instead, I used `cityHash64(transaction_id)`to expand the range within the minimum and maximum values.
diff --git a/content/en/altinity-kb-queries-and-syntax/simplestateif-or-ifstate-for-simple-aggregate-functions.md b/content/en/altinity-kb-queries-and-syntax/simplestateif-or-ifstate-for-simple-aggregate-functions.md
index f7bbb99eb3..06f81fb7b3 100644
--- a/content/en/altinity-kb-queries-and-syntax/simplestateif-or-ifstate-for-simple-aggregate-functions.md
+++ b/content/en/altinity-kb-queries-and-syntax/simplestateif-or-ifstate-for-simple-aggregate-functions.md
@@ -6,7 +6,7 @@ description: >
---
### Q. What is SimpleAggregateFunction? Are there advantages to use it instead of AggregateFunction in AggregatingMergeTree?
-SimpleAggregateFunction can be used for those aggregations when the function state is exactly the same as the resulting function value. Typical example is `max` function: it only requires storing the single value which is already maximum, and no extra steps needed to get the final value. In contrast `avg` need to store two numbers - sum & count, which should be divided to get the final value of aggregation (done by the `-Merge` step at the very end).
+The ClickHouse® SimpleAggregateFunction can be used for those aggregations when the function state is exactly the same as the resulting function value. Typical example is `max` function: it only requires storing the single value which is already maximum, and no extra steps needed to get the final value. In contrast `avg` need to store two numbers - sum & count, which should be divided to get the final value of aggregation (done by the `-Merge` step at the very end).
@@ -47,7 +47,7 @@ SimpleAggregateFunction can be used for those aggregations when the function sta
reading raw value per row
you can access it directly
-
you need to use finalizeAgggregation function
+
you need to use finalizeAggregation function
using aggregated value
@@ -77,9 +77,11 @@ SimpleAggregateFunction can be used for those aggregations when the function sta