Slow but constant memory increase rendering xlsx templates

[l-dalleaste_lectra1]

Re: Slow but constant memory leak with chrome-pdf recipe on Docker (jsreport 4.12.0)

Hello, following the discussion about chrome-pdf, I investigated similar memory behavior with large XLSX reports after a customer experienced OOM errors during repeated report generation. Memory appeared to remain elevated even after rendering had completed.

Below is the analysis, carried out with the help of GPT 6 Astra, along with the patch it helped me develop. I'd like to share it for your review and possible inclusion in a future release.

Cached async Handlebars request-data retention in XLSX reports

Summary

We identified and reproduced JavaScript object retention caused by the async
Handlebars adapter mutating a cached compiled template specification.

The adapter attaches a decorator that closes over a per-render Handlebars
instance. That instance holds registered helper closures, which can retain
request context and report data. Handlebars also attaches that decorator to
the compiled program function. The cached specification therefore keeps
request-specific objects reachable after rendering has finished.

The proposed fix creates a private specification and private program-function
wrappers for each invocation. It preserves compilation caching without letting
the cached objects point back to the request.

This correction was applied downstream and exercised with real XLSX workloads
in a production-like Kubernetes clone. The minimal reproduction below does
not require XLSX, customer data, or the custom extension.

This finding concerns async Handlebars, as used by our XLSX transformation
path. It does not establish the same root cause for every Chrome PDF memory
issue.

Environment and scope

Component	Investigated configuration
jsreport	4.12.0
jsreport core	4.9.0
@jsreport/jsreport-handlebars	4.1.0
Handlebars	4.7.7
Node.js	22.23.1
Operating system / allocator	Debian Bookworm / glibc 2.36
XLSX extension	Custom extension declaring version 4.6.1
Initial isolated investigation	One persistent rendering worker
Subsequent clone deployment	Three replicas, four rendering workers per replica

The custom XLSX extension is relevant to interpreting the release versions:
its lib/generation/processXlsx.js is byte-for-byte identical to the file
published in jsreport 4.13.0. This was checked both locally and in the installed
extension in the clone. It does not imply that the entire custom extension is
identical to upstream 4.13.0.

Rendering flow

1. A rendering worker receives the request, loads the template and helpers,
   and prepares the report data.
2. jsreport compiles the template specification and caches it in that worker.
3. A per-render Handlebars instance is created and request-specific helper
   functions are registered on it.
4. The XLSX transformation sets req.context.asyncHandlebars = true and
   evaluates the original report content against req.data, including parsed
   XLSX structures and the report dataset.
5. The async adapter creates the runtime template from the cached specification.
6. After rendering, request-specific objects should become collectible while
   the compiled specification remains cached for reuse.

The relevant implementation locations are:

• packages/jsreport-core/lib/worker/render/executeEngine.js
• packages/jsreport-handlebars/lib/handlebarsEngine.js
• packages/jsreport-handlebars/async-helpers/index.js
• packages/jsreport-xlsx/lib/transformation/index.js
• Handlebars' lib/handlebars/runtime.js

The Handlebars engine's compilation code explicitly intends to return a
specification that is independent of the current Handlebars instance so it
can be cached safely.

Root cause

Mutation of the cached specification

The original async adapter assigns spec.main_d inside
handlebars.template(spec). Its decorator references the per-render
handlebars instance, including handlebars.wrapHelperResult.

The specification passed to this function is the same object retained by
jsreport's template cache.

Mutation of shared program functions

Handlebars' runtime also performs:

templateSpec.main.decorator = templateSpec.main_d

For numbered program functions, it similarly attaches the corresponding
decorator to the program function object.

Consequently, copying only the specification object would not be enough:
its main and numbered program functions would still be shared with the
cached specification.

Effective retention path

Worker template cache
  -> compiled template specification / program function
  -> async decorator
  -> per-render Handlebars instance
  -> registered helper closures
  -> request context and report data

This path was established through source inspection and a standalone
reachability reproduction. It is not a claim based on an exported heap
snapshot.

As long as this path remains reachable, GC cannot collect the corresponding
objects. This is different from an allocator delaying the return of already
freed memory to the operating system.

Retention does not necessarily accumulate once per render

Repeated executions of the same cached specification replaced the reference
to the previous runtime in our reproduction. They retained the latest
payload, rather than retaining every historical payload.

However, different cached specifications and different workers can each
retain a large payload. Cache warming, additional templates, or changing
payload sizes can therefore make the application footprint grow.

Observed evidence

Isolated real XLSX workload

With one rendering worker and the original adapter, a completed heavy XLSX
render left approximately 707 MiB of worker heap after spontaneous major
GC. Repeating the same report returned to approximately the same retained
level.

After applying the adapter correction and restarting the diagnostic process,
two executions of the same heavy report returned the worker heap to
approximately 39-40 MiB after spontaneous GC.

The patched process RSS after those executions was approximately
269-292 MiB. These RSS numbers are not equivalent to JavaScript heap usage.

No application GC was forced for these real-report measurements.

Standalone reachability checks

A synthetic reproduction kept the compiled specification strongly reachable,
registered a helper closing over a payload, and held only WeakRef references
to the payload and per-render Handlebars instance.

• Ordinary Handlebars released the payload and runtime after GC.
• The original async adapter retained them while the specification was cached.
• A second execution released the first payload but retained the new one.
• The original adapter also retained the payload after a helper threw.
• The patched async adapter released the payload and runtime.

Forced GC was used only in disposable synthetic test processes.

Clone workload checks

The correction was also applied to the three normal application replicas in
the clone, retaining four rendering workers per replica.

In a later controlled workload, all twelve worker heaps returned to around
40 MiB after twelve heavy XLSX renders, including two concurrent reports per
replica. XLSX outputs were readable archives and retained the expected workbook,
worksheet, row, and cell counts.

These are structural output checks, not a claim that every cell was manually
reviewed or that the generated files were byte-identical.

Subsequent user-triggered report workloads also returned all twelve worker
heaps to approximately 39-42 MiB after the final rendering activity and idle
period. There were no observed OOM events or container restarts in these runs.

The observations support the correction for the reproduced retention path.
They do not prove the absence of every other memory issue or arbitrary
long-term stability.

Exact adapter patch applied

Target: packages/jsreport-handlebars/async-helpers/index.js.

The diff below uses normalized LF line endings. Our downstream installer
normalizes CRLF before checking and patching the source.

diff --git a/packages/jsreport-handlebars/async-helpers/index.js b/packages/jsreport-handlebars/async-helpers/index.js
--- a/packages/jsreport-handlebars/async-helpers/index.js
+++ b/packages/jsreport-handlebars/async-helpers/index.js
@@ -47,7 +47,19 @@
   const _template = handlebars.VM.template
   handlebars.wrapHelperResult = (p) => p
   handlebars.template = function (spec) {
-    spec.main_d = (prog, props, container, depth, data, blockParams, depths) => async (context) => {
+    const renderSpec = { ...spec }
+
+    // The runtime attaches decorators to program functions; keep cached functions untouched.
+    for (const key of Object.keys(renderSpec)) {
+      if ((key === 'main' || /^\d+$/.test(key)) && typeof renderSpec[key] === 'function') {
+        const program = renderSpec[key]
+        renderSpec[key] = function (...args) {
+          return program.apply(this, args)
+        }
+      }
+    }
+
+    renderSpec.main_d = (prog, props, container, depth, data, blockParams, depths) => async (context) => {
       const originalFn = container.fn
       container.fn = (...args) => {
         const rf = originalFn(...args)
@@ -58,11 +70,11 @@
       }
 
       // here I've changed the last param from `depths` to `[context]`. This was needed to make the ../gotoparent working
-      const v = spec.main(container, context, container.helpers, container.partials, data, blockParams, [context])
+      const v = renderSpec.main(container, context, container.helpers, container.partials, data, blockParams, [context])
       // result can be actually SafeString
       return v.then((r) => r.toString())
     }
-    return _template(spec, handlebars)
+    return _template(renderSpec, handlebars)
   }
 
   handlebars.compile = function (template, options) {

Why this fixes the reproduced path

• renderSpec is private to the runtime invocation.
• Its main and numbered program functions are new wrapper objects, so runtime
  decorator assignments do not modify the cached functions.
• The wrappers delegate to the original compiled code with the same arguments
  and this.
• The per-render decorator is attached only to the private specification.
• The cache continues to retain compiled code, not the request-specific runtime.

This does not deep-copy the request data, disable compilation caching, force
GC, or recycle workers.

The fix should be applied before workers start, or followed by a restart.
Replacing the adapter file does not retroactively clean already-mutated
specifications or reload modules in a running process.

Minimal reproduction

Save the following as repro.cjs in a project with
@jsreport/jsreport-handlebars 4.1.0 and Handlebars 4.7.7 installed.
It invokes the engine directly and does not require a running jsreport server.

const assert = require('node:assert/strict');
const path = require('node:path');
const { setImmediate: nextTurn } = require('node:timers/promises');

const engine = require('@jsreport/jsreport-handlebars/lib/handlebarsEngine')({
  handlebarsModulePath: path.dirname(require.resolve('handlebars/package.json')),
});
const cache = new Map();

function runtime() {
  const context = engine.createContext({
    context: { asyncHandlebars: true },
  });
  return {
    context,
    require(name) {
      assert.equal(name, 'handlebars');
      return context.handlebars;
    },
  };
}

async function execute(spec) {
  const current = runtime();
  const payload = { rows: new Array(10000).fill(123) };
  const references = {
    payload: new WeakRef(payload),
    handlebars: new WeakRef(current.context.handlebars),
  };
  const output = await engine.execute(
    spec,
    { value: () => payload.rows.length },
    {},
    { require: current.require },
  );
  assert.equal(output, '10000');
  return references;
}

async function main() {
  assert.equal(typeof global.gc, 'function', 'Run with --expose-gc');
  const compiler = runtime();
  const spec = engine.compile('{{value}}', { require: compiler.require });
  cache.set('template', spec);
  const references = await execute(spec);

  for (let i = 0; i < 4; i++) {
    await nextTurn();
    global.gc();
  }

  assert.equal(cache.get('template'), spec);
  const retainedPayload = references.payload.deref() !== undefined;
  const retainedHandlebars = references.handlebars.deref() !== undefined;
  const expected = process.env.HANDLEBARS_EXPECT_RETENTION === 'true';
  assert.equal(retainedPayload, expected);
  assert.equal(retainedHandlebars, expected);
  console.log(JSON.stringify({
    retainedPayload,
    retainedHandlebars,
    cachedSpecifications: cache.size,
  }));
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

On an unpatched installation:

HANDLEBARS_EXPECT_RETENTION=true node --expose-gc repro.cjs

Expected result:

{"retainedPayload":true,"retainedHandlebars":true,"cachedSpecifications":1}

After applying the adapter correction:

HANDLEBARS_EXPECT_RETENTION=false node --expose-gc repro.cjs

Expected result:

{"retainedPayload":false,"retainedHandlebars":false,"cachedSpecifications":1}

Additional regression coverage

Our broader synthetic harness exercised:

• Ordinary Handlebars as a non-retaining control.
• Repeated execution of the same cached async specification.
• Cleanup after a helper throws.
• Two concurrent executions sharing the same compiled specification.
• Async helpers inside each, parent-context lookup, and block parameters.
• Partials, escaped output, and unescaped output.
• Execution with the specification and compiled program functions frozen.
• Absence of main_d and main.decorator mutations on the cached objects.

The frozen-specification case is particularly useful for protecting the
cache-immutability invariant.

Upstream review should also consider supported custom decorators and any
function metadata expectations not represented by these cases.

Relation to the XLSX improvements in jsreport 4.13.0

The 4.13.0 release includes two relevant generation improvements:

1. Skip generation work when the input workbook has no Handlebars tags.
2. Separate dynamic data evaluation from final XML construction, excluding
   static XML and sharedStrings.xml from unnecessary evaluation.

These changes improve the generation path. They do not replace the async
Handlebars retention correction.

The async adapter source is identical between the 4.12.0 and 4.13.0 tags.
In 4.13.0 it still assigns spec.main_d and passes the shared specification
to _template(spec, handlebars). The legacy XLSX transformation still enables
async Handlebars.

The release updates @jsreport/jsreport-handlebars from 4.1.0 to 4.1.1 and
Handlebars from 4.7.7 to 4.7.9, but Handlebars' runtime still attaches
decorators to the compiled program functions.

This conclusion is based on tagged-source comparison. We did not perform a
full application upgrade or an end-to-end benchmark of jsreport 4.13.0.

Separate issue: native allocator RSS retention

After fixing the JavaScript retention path, process RSS can still remain
higher than its cold baseline.

In a separate controlled experiment on an idle clone replica, glibc reported
approximately 583 MiB of free blocks in its arenas. A single explicitly
authorized malloc_trim(0) reduced RSS from approximately 912 MiB to
461 MiB, while the measured worker heap stayed around 40 MiB.

This demonstrates a substantial contribution from already-freed native
pages. Allocator-free bytes are not necessarily all resident or reclaimable,
and this experiment does not prove the absence of other native leaks.

We separately applied the following startup mitigation to the clone and to
the downstream runtime Dockerfile:

ENV MALLOC_ARENA_MAX="2"

It limits glibc arena proliferation and can reduce native-memory retention,
but may increase allocation lock contention. It is platform-specific and
workload-dependent, and should not be treated as the fix for the
Handlebars object-retention bug.

No native addon or periodic malloc_trim was added to the application image.
The native addon used for the one-off investigation was diagnostic only.

Neither correction guarantees that RSS returns to the cold baseline or that
arbitrarily heavy concurrent reports fit the container's memory limit.
The adapter patch does not change worker concurrency or V8 heap limits.

Suggested upstream action

Treat cached compiled async Handlebars specifications and their program
functions as immutable with respect to per-render state.

Consider incorporating the private-specification/private-function approach
above, together with reachability and frozen-specification regression tests.
In particular, do not rely on a shallow specification copy alone.

Public references

• Original memory discussion
• Async adapter in jsreport 4.12.0
• Async adapter in jsreport 4.13.0
• Handlebars engine in jsreport 4.13.0
• XLSX transformation in jsreport 4.13.0
• Handlebars 4.7.9 runtime
• jsreport 4.13.0 release
• Two-phase XLSX generation change
• Skip generation without dynamic tags
• Node.js memoryUsage documentation