Large CSV Generation

Evan Furbeyre

We are currently trying to generate a large CSV (via the Handlebars engine, as a text file) asynchronously and save it to S3, but are running into this error when polling the status of the report: Report generation failed. Error: Cannot create a string longer than 0x1fffffe8 characters

Does anyone know of an alternate solution to this?
Is it possible to stream the report to S3 rather than build the entire thing in memory?
Would an XLSX file work any better?

Thanks

jan_blaha

Does anyone know of an alternate solution to this?

Unfortunately, the handlebars need to hold results in the memory so you can't generate CSV bigger than ~500mb this way. We have a solution we want to implement mentioned here:
https://github.com/jsreport/jsreport/issues/839

Is it possible to stream the report to S3 rather than build the entire thing in memory?

You could make the template "empty" and not use the handlebars to concatenate the csv. Instead, you implement the afterRender script. This script would "batch loop" your data, create csv text lines and stream the result to the s3 using aws s3 nodejs SDK.

Would an XLSX file work any better?

Yes, the xlsxAdd helper batches the XML into files which are later streamed and compressed to the xlsx. So the memory consumption there is better.