Issue origin
After migrating our java apps in cf from the Ruby-based 4.x buildpack to 5.0.4 go rewrite, we observed a steadily growing number of memory consumed in comparison to the old buildpack.
We tried different things and the behaviour hinted at a missing setting of MALLOC_ARENA_MAX. We have added this manually in the manifest and it solved the issues for us. That was surprising, because the buildpack's own start command
already contains MALLOC_ARENA_MAX=2 — which is what led to the investigation and this issue.
Investigation
We compared the start commands and notice, that in the Ruby-based 4.x buildpack, the assignments are a prefix of the command, so they are available in the java process:
... && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec .../bin/java $JAVA_OPTS ...
5.0.x — adds an && separates the assignment from the command, making it a standalone shell assignment:
... && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 && eval exec $JAVA_HOME/bin/java $JAVA_OPTS ...
On a running app one can verify this against the last ruby and the 5.0.4 go buildpack with:
cf ssh APP -c 'pid=$(pgrep -f "bin/java" | head -1); tr "\0" "\n" < /proc/$pid/environ | grep -i malloc'
which returned nothing under 5.0.4 while the start command reported by
cf curl "/v3/apps/$(cf app APP --guid)/processes" contains MALLOC_ARENA_MAX=2.
And vice versa, in the 4.77.0 it is present in the acutal JVM and the start commands.
Fix
I prepared a small fix here: #1434
Please note: I haven't fully understand why there is still the code path for shell script generation of memory_calculator.sh, but this path does the "right thing" already with exporting the value.
Suspected root cause
The memory_calculator.go ends the snippet with && MALLOC_ARENA_MAX=2 and finalize.go:268 appends
" && " + containerCommand after it. Due to this, MALLOC_ARENA_MAX remains a plain shell variable and
is not exported for the java process.
Expected behavior
MALLOC_ARENA_MAX=2 is part of the environment of the JVM process.
Actual behavior
MALLOC_ARENA_MAX=2 appears in the start command, but is absent from the JVM
Environment
CF Java buildpack 5.0.4
Issue origin
After migrating our java apps in cf from the Ruby-based 4.x buildpack to 5.0.4 go rewrite, we observed a steadily growing number of memory consumed in comparison to the old buildpack.
We tried different things and the behaviour hinted at a missing setting of
MALLOC_ARENA_MAX. We have added this manually in the manifest and it solved the issues for us. That was surprising, because the buildpack's own start commandalready contains
MALLOC_ARENA_MAX=2— which is what led to the investigation and this issue.Investigation
We compared the start commands and notice, that in the Ruby-based 4.x buildpack, the assignments are a prefix of the command, so they are available in the java process:
5.0.x — adds an
&&separates the assignment from the command, making it a standalone shell assignment:On a running app one can verify this against the last ruby and the 5.0.4 go buildpack with:
cf ssh APP -c 'pid=$(pgrep -f "bin/java" | head -1); tr "\0" "\n" < /proc/$pid/environ | grep -i malloc'which returned nothing under 5.0.4 while the start command reported by
cf curl "/v3/apps/$(cf app APP --guid)/processes"containsMALLOC_ARENA_MAX=2.And vice versa, in the 4.77.0 it is present in the acutal JVM and the start commands.
Fix
I prepared a small fix here: #1434
Please note: I haven't fully understand why there is still the code path for shell script generation of
memory_calculator.sh, but this path does the "right thing" already with exporting the value.Suspected root cause
The
memory_calculator.goends the snippet with&& MALLOC_ARENA_MAX=2andfinalize.go:268appends" && " + containerCommandafter it. Due to this,MALLOC_ARENA_MAXremains a plain shell variable andis not exported for the java process.
Expected behavior
MALLOC_ARENA_MAX=2is part of the environment of the JVM process.Actual behavior
MALLOC_ARENA_MAX=2appears in the start command, but is absent from the JVMEnvironment
CF Java buildpack 5.0.4