Generating same report for different parameters automatically

palphonse

Thanks again,
Sorry I'm new to JSReport and node.js, technically what would be the best way to render for example 100000 returned records that we are looping over.
Would we need to use "await" if the calls to render are split into workers or will JSreport handle this.

Thanks

jan_blaha

There are many ways how to reach parallel processing of your records with a concurrency limit.
I would personally use bluebird module and map function with concurrency limit set to something reasonable like 5.
http://bluebirdjs.com/docs/api/promise.map.html

I would recommend checking some tutorials on javascript async/await and promises.

palphonse

Hello again,

Just to recap my problem.We are using chrome-pdf to run a scheduled report, when it's time for the report to run, we need to generate a report for a large number of users (could reach 100,000) So we would need one scheduled template (template 1) to render another template (template 2) with a specific parameter. So template 1 would need to render template 2 100,000 times.
From what I understood from your help and the documentation, we would need to adjust some settings in the config file and use a module to split the requests made to template 2 from template 1. Is this correct?
Could you please advise on what settings would be needed, and how to go about this?
Thank again

jan_blaha

From what I understood from your help and the documentation, we would need to adjust some settings in the config file

Setting "timeout" to a very high number should be enough for the start.

use a module to split the requests made to template 2 from template 1

The bluebirdjs I mentioned is a common utility library. You can fulfill the task also without it with few lines of code.

In case you want to have multiple jsreport physical servers to load balance the report generation, you should consider a different architecture. Maybe have the scheduler running inside your app and triggering the 100 000 requests from your app.

palphonse

Hello again,

I set the numberofworkers to 4, and set concurrency to 4 in bluebirdjs and trying to render 1000 requests for the purpose of testing. It is working, but I didn't notice a difference in performance if i changed the numberofworkers to lets say 2 or 1.
Why could this be?

Thanks

jan_blaha

Great! Glad you are moving forward.

Please try config like this. And modify the numbers. You should see the difference. At least for the chrome-pdf recipe.

{
  "templatingEngines": {
    "numberOfWorkers": 2,
    "strategy": "http-server"
  },
  "chrome": {
     "numberOfWorkers": 2,
     "strategy": "chrome-pool"
  }
}

The numberOfWorkers property is currently being used only when you use http-server and chrome-pool strategy.

palphonse

Hello, thank you, these are the settings i had, i also used 2 for both

jan_blaha

I tried this simple test script using the sample invoice template.

const jsreport = require('jsreport-proxy')
const Promise = require('bluebird')

function beforeRender(req, res) {
    const reportsToRun = []
    for (let i = 0; i < 100; i++) {
        reportsToRun.push(i)
    }

    return Promise.map(reportsToRun, (i) => jsreport.render({
        template: {
            name: 'invoice-main'
        }        
    }), {
        concurrency: 4
    })
}

Results
numberOfWorkers: 2
23s

numberOfWorkers: 1
37s

Maybe try the same with the invoice template the first, to see if it makes difference or not.

palphonse

Ok thanks I will try it. The difference is that the second template that is being rendered uses mssql to read data could this be a reason?

palphonse

Hello again, I tried the sample, and i noticed a few seconds difference, however when I run my template, no difference is noticed.Could this be because we are using mssql to get data from a db?

jan_blaha

few seconds difference

It should be close to 2x performance.

Could this be because we are using mssql to get data from a db?

Hm, please try to share the output logs from jsreport. Maybe there is a "blocking" operation that is consuming the most of the time and the performance improvement with parallelization is then minimal. Let's say data query to the db takes 90% of the report rendering time. And the query consumes whole db time so the second query from the second report just waits until the first one is finished. In this case, the parallelization has no effect.

palphonse

thanks,

so from what i understand jsreport will not process multiple sql queries in parallel?

palphonse

Hello again,
Please note that we are using a server with 16 CPUs.
We tested by doubling the numberOfWorkers in the jsReport config starting at 1 up to 16, with concurrency = numberOfWorkers x 4 ( from 4 till 64 respectively)
Using this setup, we didn't get any noticeable performance boost after numberOfWorkers = 2 and concurrency = 8.
However, executing the same setup without fetching data from DB resulted in better performance all the way till numberOfWorkers = 8 and concurrency = 32.
During the report generation, we monitored the SQL server and it was running under 20% of its maximum CPU usage since we used a very simple SELECT query.

Below is a link to the requested log (I was not able to attach the file to the message).
The log was for numberOfWorkers =16 concuurency = 64

https://we.tl/t-ZHCwh4PF8F

Thanks

jan_blaha

You should enable the debug logs, it is typically done using jsreport configure command, or by adding to the config

"logger": {
    "console": {
      "transport": "console",
      "level": "debug"
    },
    "file": {
      "transport": "file",
      "level": "info",
      "filename": "logs/reporter.log"
    },
    "error": {
      "transport": "file",
      "level": "error",
      "filename": "logs/error.log"
    }
}

However, the 20% CPU doesn't mean your scripts aren't waiting for the database. They still likely do. It can be because of data IO or something else.

palphonse

Hello we already enabled debug logs in the config file, please find the link to the reporter.log file (we were not able to attach it here)The error.log file is empty.

https://we.tl/t-ZHCwh4PF8F

Thanks