Skip to content

Mount state to non-temporary directory - #474

Draft
philipp94831 wants to merge 5 commits into
masterfrom
feature/state-dir
Draft

Mount state to non-temporary directory#474
philipp94831 wants to merge 5 commits into
masterfrom
feature/state-dir

Conversation

@philipp94831

Copy link
Copy Markdown
Member

No description provided.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Helm charts to configure Kafka app state directories under a non-temporary path and to mount the persistent volume at that same directory when running as a StatefulSet with persistence enabled.

Changes:

  • Add kafka.stateDir defaults for streams, consumerproducer, and consumer charts.
  • Inject KAFKA_STATE_DIR env var from kafka.stateDir into each chart’s Deployment/StatefulSet template.
  • Mount the datadir volume at kafka.stateDir (instead of /tmp/...) and document the new value in chart READMEs.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
charts/streams-app/values.yaml Adds default kafka.stateDir for streams state.
charts/streams-app/templates/deployment.yaml Sets KAFKA_STATE_DIR and mounts datadir at the configured state dir.
charts/streams-app/README.md Documents the new kafka.stateDir value.
charts/consumerproducer-app/values.yaml Adds default kafka.stateDir for consumerproducer chart.
charts/consumerproducer-app/templates/deployment.yaml Sets KAFKA_STATE_DIR and mounts datadir at the configured state dir.
charts/consumerproducer-app/README.md Documents the new kafka.stateDir value.
charts/consumer-app/values.yaml Adds default kafka.stateDir for consumer chart.
charts/consumer-app/templates/deployment.yaml Sets KAFKA_STATE_DIR and mounts datadir at the configured state dir.
charts/consumer-app/README.md Documents the new kafka.stateDir value.
Suppressed comments (3)

charts/streams-app/templates/deployment.yaml:42

  • volumeMounts.mountPath must be a string; rendering .Values.kafka.stateDir without quoting can produce invalid manifests if the value is not a plain path scalar. Quote it to force a string.
              mountPath: {{ .Values.kafka.stateDir }}

charts/consumerproducer-app/templates/deployment.yaml:42

  • volumeMounts.mountPath must be a string; rendering .Values.kafka.stateDir without quoting can produce invalid manifests if the value is not a plain path scalar. Quote it to force a string.
              mountPath: {{ .Values.kafka.stateDir }}

charts/consumer-app/templates/deployment.yaml:30

  • volumeMounts.mountPath must be a string; rendering .Values.kafka.stateDir without quoting can produce invalid manifests if the value is not a plain path scalar. Quote it to force a string.
              mountPath: {{ .Values.kafka.stateDir }}

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread charts/streams-app/templates/deployment.yaml Outdated
Comment thread charts/consumerproducer-app/templates/deployment.yaml Outdated
Comment thread charts/consumer-app/templates/deployment.yaml Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

charts/streams-app/templates/deployment.yaml:29

  • KAFKA_STATE_DIR (i.e., Kafka config state.dir) is set unconditionally, but the datadir volume is only mounted when persistence.enabled && statefulSet. This changes behavior for non-persistent/non-stateful deployments and can point Kafka Streams at a directory that isn’t backed by any volume.

Consider only setting KAFKA_STATE_DIR when the datadir mount is present (same condition as the volumeMount), so state continues to use Kafka Streams’ default when not running statefully with persistence.

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

charts/consumerproducer-app/templates/deployment.yaml:29

  • KAFKA_STATE_DIR (Kafka config state.dir) is set unconditionally, but the datadir volume is only mounted when persistence.enabled && statefulSet. This changes behavior for non-persistent/non-stateful deployments and can point the app at a state directory that isn’t backed by any volume.

Consider only setting KAFKA_STATE_DIR when the datadir mount is present (same condition as the volumeMount).

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

charts/consumer-app/templates/deployment.yaml:19

  • KAFKA_STATE_DIR (Kafka config state.dir) is set unconditionally, but the datadir volume is only mounted when persistence.enabled && statefulSet. This changes behavior for non-persistent/non-stateful deployments and can point the app at a directory that isn’t backed by any volume.

Consider only setting KAFKA_STATE_DIR when the datadir mount is present (same condition as the volumeMount).

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

charts/streams-app/README.md:60

  • kafka.stateDir is documented as a general setting, but the chart only applies it when persistence.enabled and statefulSet are both true (it’s not passed as KAFKA_STATE_DIR otherwise). This can confuse users who set kafka.stateDir without enabling persistence/statefulset and see no effect.
| `kafka.stateDir`             | Directory for Kafka Streams state.                                                                                                                                               | `/state/kafka-streams` |

charts/consumerproducer-app/README.md:60

  • kafka.stateDir is documented as a general setting, but the chart only applies it when persistence.enabled and statefulSet are both true (it’s not passed as KAFKA_STATE_DIR otherwise). Clarifying this avoids confusion for users running a plain Deployment.
| `kafka.stateDir`             | Directory for app state.                                                                                                                                                         | `/state/kafka-consumer-producer` |

charts/consumer-app/README.md:60

  • kafka.stateDir is documented as a general setting, but the chart only applies it when persistence.enabled and statefulSet are both true (it’s not passed as KAFKA_STATE_DIR otherwise). Consider documenting this condition explicitly.
| `kafka.stateDir`             | Directory for app state.                                                                                                                                                         | `/state/kafka-consumer` |

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Suppressed comments (3)

charts/streams-app/templates/deployment.yaml:45

  • mountPath is derived from kafka.stateDir, but users can also set kafka.config["state.dir"] (which becomes KAFKA_STATE_DIR) via the shared common-env helper. If those diverge, the app will write state somewhere that isn't on the mounted volume. Consider making the mountPath follow kafka.config["state.dir"] when provided.
            - name: datadir
              mountPath: {{ .Values.kafka.stateDir | quote }}
            {{- end }}

charts/consumerproducer-app/templates/deployment.yaml:44

  • mountPath is derived from kafka.stateDir, but users can also set kafka.config["state.dir"] (which becomes KAFKA_STATE_DIR) via the shared common-env helper. If those diverge, the app may write state somewhere that isn't on the mounted volume. Consider making the mountPath follow kafka.config["state.dir"] when provided.
            - name: datadir
              mountPath: {{ .Values.kafka.stateDir | quote }}

charts/consumer-app/templates/deployment.yaml:32

  • mountPath is derived from kafka.stateDir, but users can also set kafka.config["state.dir"] (which becomes KAFKA_STATE_DIR) via the shared common-env helper. If those diverge, the app may write state somewhere that isn't on the mounted volume. Consider making the mountPath follow kafka.config["state.dir"] when provided.
            - name: datadir
              mountPath: {{ .Values.kafka.stateDir | quote }}

Comment thread charts/streams-app/templates/deployment.yaml Outdated
Comment thread charts/consumer-app/templates/deployment.yaml Outdated
Comment thread charts/consumerproducer-app/templates/deployment.yaml Outdated
@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (6)

charts/streams-app/templates/deployment.yaml:42

  • mountPath is derived from .Values.kafka.stateDir, but users can also override the effective Kafka Streams state directory via kafka.config["state.dir"] (which becomes KAFKA_STATE_DIR). If those diverge, the app will write state outside the mounted volume.
              mountPath: {{ .Values.kafka.stateDir | quote }}

charts/consumerproducer-app/templates/deployment.yaml:42

  • mountPath is derived from .Values.kafka.stateDir, but users can also override state.dir via kafka.config["state.dir"] (rendered into KAFKA_STATE_DIR by kafka-app.common-env). If these diverge, the app will write state outside the mounted volume.
              mountPath: {{ .Values.kafka.stateDir | quote }}

charts/consumer-app/templates/deployment.yaml:30

  • mountPath is derived from .Values.kafka.stateDir, but users can also override state.dir via kafka.config["state.dir"] (rendered into KAFKA_STATE_DIR by kafka-app.common-env). If these diverge, the app will write state outside the mounted volume.
              mountPath: {{ .Values.kafka.stateDir | quote }}

charts/streams-app/templates/deployment.yaml:29

  • kafka-app.common-env already renders KAFKA_* env vars from .Values.kafka.config (including state.dir -> KAFKA_STATE_DIR). Adding KAFKA_STATE_DIR here unconditionally can create duplicate env var entries when users set kafka.config["state.dir"], making the effective value unclear.

This issue also appears on line 42 of the same file.

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

charts/consumerproducer-app/templates/deployment.yaml:29

  • kafka-app.common-env already renders KAFKA_* env vars from .Values.kafka.config (including state.dir -> KAFKA_STATE_DIR). Adding KAFKA_STATE_DIR here unconditionally can create duplicate env var entries when users set kafka.config["state.dir"], making the effective value unclear.

This issue also appears on line 42 of the same file.

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

charts/consumer-app/templates/deployment.yaml:19

  • kafka-app.common-env already renders KAFKA_* env vars from .Values.kafka.config (including state.dir -> KAFKA_STATE_DIR). Adding KAFKA_STATE_DIR here unconditionally can create duplicate env var entries when users set kafka.config["state.dir"], making the effective value unclear.

This issue also appears on line 30 of the same file.

            - name: KAFKA_STATE_DIR
              value: {{ .Values.kafka.stateDir | quote }}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants