Posts made by matthewa-smith

Sorry, it seems my issue has not been resolved entirely; it's still having issues when trying to append a merged PDF, except now with EOL errors instead of xref errors. This error didn't present itself earlier, I'm assuming, because I was only doing a merge on one PDF using pdftk, which it seems was just doing a simple copy. Now that I'm including merging multiple PDFs, it's giving a different output that is erroring.

Looking at the location it's erroring in, it seems to expect there to be a line ending directly after endstream in the PDF, but my output PDF has a space first, then the LF.

Not sure if there is any way to reconcile this easily.

A sample of this file can be found here.

https://playground.jsreport.net/w/anon/ZL774wiw

Thanks for your help, it seems it was indeed the way I was fetching the data. I opted to use the 'http' module instead of 'request' and buffer everything into a Uint8Array first before trying to append, and that seems to have worked.

That leads me to my next question:

Is there any way to set the $pdf report data in the afterRender function? My assumption is no, but I'm hoping there is a less cumbersome way than having to do:

async function afterRender(req, res, done) {
    --snip--

    // append merged doc
    res.content = await jsreport.pdfUtils.append(res.content, Buffer.from(buf));

    // get $pdf object
    let $pdf = await jsreport.pdfUtils.parse(res.content);

    // re-render this template with the $pdf object in data
    let actual = await jsreport.render({
        template: {
            shortid: 'id', // same template that is currently being rendered
        },
        data: {
            ...req.data,
            $pdf: $pdf,
        },
    });

    // append merged doc again
    res.content = await jsreport.pdfUtils.append(actual.content, Buffer.from(buf));
    done();
}

Sure, I've uploaded a sample of the output from pdftk here.

I should also note, after a bit of testing, it seems that even PDFs created by jsreport itself have this issue as well. If I create a report via jsreport and then try to fetch and append that PDF to my report, I also get the same xref error. Is this behavior expected? The reports I am creating through jsreport are using the chrome-pdf recipe with handlebars engine.

A sample of the jsreport output is here.

Hello,

I am trying to append a merged PDF document created by pdftk into a new report but am receiving an xref error:

error: Report render failed
(because) invalid xref: xref expected but not found
Error: Invalid xref: xref expected but not found
    at PDFXref.parseXrefObject (node_modules\@jsreport\pdfjs\lib\object\xref.js:107:13)
    at PDFXref.parse (node_modules\@jsreport\pdfjs\lib\object\xref.js:64:19)
    at Parser.parse (node_modules\@jsreport\pdfjs\lib\parser\parser.js:39:26)
    at module.exports.parseBuffer (node_modules\@jsreport\pdfjs\lib\parser\parser.js:99:10)
    at new ExternalDocument (node_modules\@jsreport\pdfjs\lib\external.js:7:20)
    at Object.append (node_modules\@jsreport\jsreport-pdf-utils\lib\pdfManipulator.js:29:23)
    at Object.append (node_modules\@jsreport\jsreport-pdf-utils\lib\proxyExtend.js:25:25)
    at Request._callback (sandbox.js:15:47)
    at self.callback (node_modules\request\request.js:185:22)
    at Request.emit (node:events:518:28) rootId=2emrsywhkmxopvw, id=2emrsywhkmxopvw

2024-04-30T22:04:29.879Z - error: Error during processing request at http://localhost:3000/reporting/api/report, details: Invalid xref: xref expected but not found, stack: Error: Invalid xref: xref expected but not found
    at PDFXref.parseXrefObject (node_modules\@jsreport\pdfjs\lib\object\xref.js:107:13)
    at PDFXref.parse (node_modules\@jsreport\pdfjs\lib\object\xref.js:64:19)
    at Parser.parse (node_modules\@jsreport\pdfjs\lib\parser\parser.js:39:26)
    at module.exports.parseBuffer (node_modules\@jsreport\pdfjs\lib\parser\parser.js:99:10)
    at new ExternalDocument (node_modules\@jsreport\pdfjs\lib\external.js:7:20)
    at Object.append (node_modules\@jsreport\jsreport-pdf-utils\lib\pdfManipulator.js:29:23)
    at Object.append (node_modules\@jsreport\jsreport-pdf-utils\lib\proxyExtend.js:25:25)
    at Request._callback (sandbox.js:15:47)
    at self.callback (node_modules\request\request.js:185:22)
    at Request.emit (node:events:518:28)

The pdf I am trying to append is a merge of multiple pdfs that are also merged pdfs. Let me illustrate what I mean. The below shows the folder structure of my report staging area:

root
|
---- subfolder_1
|    |
|    ---- merged_1.pdf <--- created by pdftk by merging report_1.pdf and external.pdf
|    |
|    ---- report_1.pdf <-- gets created by jsreport
|    |
|    ---- external.pdf <-- uploaded by user, gets appended to report_1.pdf
|
---- subfolder_2
|    |
|    ---- merged_2.pdf <-- created by pdftk by merging report_2.pdf, in this case, with nothing
|    |
|    ---- report_2.pdf <-- gets created by jsreport
|
---- merged.pdf <-- created by pdftk by merging merged_1.pdf and merged_2.pdf

The issue I'm running into is that the merged.pdf file does not seem to be able to properly be appended.

The report I am trying to append to is doing a simple fetch for the pdf and then merging it via the pdfUtils.append function:

const request = require('request');
const jsreport = require('jsreport-proxy');

async function afterRender (req, res, done) {
    // fetch pdf content
    request.get({
        url: /*endpoint to fetch merged.pdf*/,
    }, async (err, response, body) => {
        let buf = Buffer.from(body);
        res.content = await jsreport.pdfUtils.append(res.content, buf);
        done();
    });
}

The endpoint it hits is defined as:

let getMergedReport = (req, res, next) => {
    let pdf = fs.readFileSync(`path\\to\\merged.pdf`);
    res.send(pdf);
}

The reason for doing multiple merges is because I need to be able to append attachments to each subreport before merging them all together.

The report I'm trying to generate is a simple cover page that I would like to be able to show the page count of the entire report after merging it all together.

Is this due to a general incompatibility with pdftk or something else?

Any help or suggestions are appreciated, thanks!

Hello,

That seems to have worked! After disabling the console-level logging (but still having error-level logging enabled), we had another instance of this issue popping up. However, nothing is logged in our error log. Thus, we're assuming that this issue probably doesn't have anything to do with jsReport (possibly a locked file issue instead). We'll continue to monitor the issue but I get the feeling this may be being caused by stale filehandles, since we have also been getting EPERM errors as well related to this issue.

Thank you for your help!

Hello,

Sorry for the late reply! We made the suggested change in the jsreport.config.json file, however, when we restart the jsReport server so that it can grab the updated configuration information, we get a new error during jsReport initialization:

2023-07-26T17:37:11.263Z - error: Error occurred during reporter init: DataCloneError: function Format(options = {}) {
    this.options = options;
  } could not be cloned.
    at new Worker (node:internal/worker:235:17)
    at createWorker (****\node_modules\@jsreport\advanced-workers\lib\workersManager.js:14:20)
    at createWorker (****\node_modules\@jsreport\advanced-workers\lib\workersManager.js:33:29)
    at ****\node_modules\@jsreport\advanced-workers\lib\pool.js:16:26
    at ****\node_modules\@jsreport\advanced-workers\lib\pool.js:21:52
    at Array.map (<anonymous>)
    at Object.init (****\node_modules\@jsreport\advanced-workers\lib\pool.js:21:42)
    at Object.init (****\node_modules\@jsreport\advanced-workers\lib\workersManager.js:37:24)
    at MainReporter.init (****\node_modules\@jsreport\jsreport-core\lib\main\reporter.js:256:34)

We changed the logging back to how it was initially for the time being until we can pin the issue down.

Hello!

jsReport version: ^3.1.1

Recently we have been seeing a peculiar issue. We currently have a web page that gathers some parameters to use as report parameters later on. We then create a file stream on the server using fs.createWriteStream with all default options. We then call our jsReport server's /api/report endpoint via a POST request to render a report and .pipe that response output to our WriteStream object we created earlier. See code snippet below:

let exportReport = (req, res, next) => {
    let dataObj = req.body || {};
    let fName = `${req.body['options']['title']}-${new Date().toDateString()}.${req.body['options']['extn'] ||'xlsx'}`;
    let ws = fs.createWriteStream(`./temp/${fName}`); // writeable stream
    ws.on('error', (err) => {
        next(err);
    });
    ws.on('finish', () => {
        setTimeout(function() {
            fs.unlink(ws.path, (delErr) => {});
        }, 120000); //delete file after 2 mins
        return res.send(`/tmp/${fName}`);
    });

    request.post({
        headers: {'Content-Type': 'application/json'},
        url: `${jsReportUrl}/api/report`,
        body: JSON.stringify(dataObj)
    }).on('error', (err) => {
        next(err);
    }).pipe(ws);
};

This all works well and good most of the time. The problem we've been seeing is that sometimes this will fail. The file that gets created on disk will have 0 bytes written to it and the response for this exportRpt function never gets sent. There seems to be a consistent error that is popping up when it does happen: error: uncaughtException: EPIPE: broken pipe, write. We've also noticed it has most recently been happening on Mondays, though checking the error logs reveals that it's not always Mondays, sometimes it was happening on just Tuesdays, other times on just Wednesdays.

At the same time as this EPIPE error, the jsReport log will look something like this:

...
2023-07-23T20:06:40.552Z - info: reporter initialized
2023-07-23T20:06:53.876Z - info: Starting rendering request 1 (user: null) rootId=o130d53vua1o29w, id=o130d53vua1o29w
2023-07-23T20:06:53.877Z - info: Rendering template { name: PortalReportsXlsx, recipe: xlsx, engine: handlebars, preview: false } rootId=o130d53vua1o29w, id=o130d53vua1o29w
2023-07-23T20:07:00.647Z - info: Rendering request 1 finished in 6857 ms rootId=o130d53vua1o29w, id=o130d53vua1o29w
2023-07-24T14:59:32.510Z - info: Starting rendering request 2 (user: null) rootId=eoovg79j2wcrvh3, id=eoovg79j2wcrvh3
2023-07-24T15:00:40.596Z - info: Initializing jsreport (version: 3.2.0, configuration file: jsreport.config.json, nodejs: 16.13.1)
2023-07-24T15:00:40.599Z - info: Searching for available extensions in ****
...

Rendering request 2 is the one that fails in this instance. Notice the timestamps of rendering request 1 and 2, and also notice that right after rendering request 2 is made, jsReport begins to re-initialize itself. Once jsReport has re-initialized itself, the report will generate correctly, write to the file on disk, and send the response.

We've been trying to figure this issue out for the past couple of weeks, but since it seems to be happening so infrequently, it's hard to identify the steps to reproduce the failure. Any help with this issue would be greatly appreciated!

I'll post the relevant section of the error log below for reference:

...
2023-07-24T14:59:32.524Z - error: uncaughtException: EPIPE: broken pipe, write
Error: EPIPE: broken pipe, write
    at Socket._write (node:internal/net:55:25)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Socket.Writable.write (node:internal/streams/writable:334:10)
    at Console.log (****\node_modules\@jsreport\jsreport-core\node_modules\winston\lib\winston\transports\console.js:79:23)
    at Console._write (****\node_modules\winston-transport\index.js:103:17)
    at doWrite (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:428:64)
    at writeOrBuffer (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:417:5)
    at Console.Writable.write (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:334:11)
    at DerivedLogger.ondata (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:681:20)
    at DerivedLogger.emit (node:events:402:35)
    at addChunk (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:298:12)
    at readableAddChunk (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:280:11)
    at DerivedLogger.Readable.push (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:241:10)
    at DerivedLogger.Transform.push (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_transform.js:139:32)
    at DerivedLogger._transform (****\node_modules\@jsreport\jsreport-core\node_modules\winston\lib\winston\logger.js:305:12) error=Error: EPIPE: broken pipe, write, stack=Error: EPIPE: broken pipe, write
    at Socket._write (node:internal/net:55:25)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Socket.Writable.write (node:internal/streams/writable:334:10)
    at Console.log (****\node_modules\@jsreport\jsreport-core\node_modules\winston\lib\winston\transports\console.js:79:23)
    at Console._write (****\node_modules\winston-transport\index.js:103:17)
    at doWrite (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:428:64)
    at writeOrBuffer (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:417:5)
    at Console.Writable.write (****\node_modules\winston-transport\node_modules\readable-stream\lib\_stream_writable.js:334:11)
    at DerivedLogger.ondata (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:681:20)
    at DerivedLogger.emit (node:events:402:35)
    at addChunk (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:298:12)
    at readableAddChunk (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:280:11)
    at DerivedLogger.Readable.push (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_readable.js:241:10)
    at DerivedLogger.Transform.push (****\node_modules\@jsreport\jsreport-core\node_modules\readable-stream\lib\_stream_transform.js:139:32)
    at DerivedLogger._transform (****\node_modules\@jsreport\jsreport-core\node_modules\winston\lib\winston\logger.js:305:12), exception=true, date=Mon Jul 24 2023 07:59:32 GMT-0700 (Pacific Daylight Time), process=[object Object], os=[object Object], trace=[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
...