Generating same report for different parameters automatically

[palphonse]

Hello,

I am trying to find a way to use jsreport to achieve the following.

I would like to schedule a report to run and send an email once done to the appropriate recipient.
This means that the report would need to run multiple times with different parameters each time, once for each recipient based on the appropriate parameter.

Thank you for your help.

[jan_blaha]

Please check this part of the documentation
https://jsreport.net/learn/scripts#rendering-another-template-from-script

The idea would be to schedule a template with a custom script attached. The script would render another template multiple times based on the parameters you wish.

const jsreport = require('jsreport-proxy')

async function afterRender(req, res) {  
  // the template xxxxx can have another script emailing the result attached or you can email directly here
  await jsreport.render({ template: { name: 'xxxxxx' }, data: {} })  
  await jsreport.render({ template: { name: 'xxxxxx' }, data: {} })  
  await jsreport.render({ template: { name: 'xxxxxx' }, data: {} })  
}

[palphonse]

Thanks for the reply, so I guess if the parameters are fetched from a DB, then we can loop over them and call the render function accordingly?

[jan_blaha]

Yes, you have the full power and flexibility in the custom script hook.

[palphonse]

Thanks again.
Lets say there are 1000 records that we are looping over, so we will have to call render 1000 times. How will JsReport handle these requests. Will they be divided among different workers or just 1 worker.

Thanks

[jan_blaha]

jsreport server with default config runs templating engine evaluation and chrome rendering in the dedicated workers.
This applies as well in case of calling jsreport.render from the script.
So yes, the load of 1000 calls will get splitter into workers.
However, I would anyway recommend limiting the parallel processing to couple of reports.

[palphonse]

Thanks again,
Sorry I'm new to JSReport and node.js, technically what would be the best way to render for example 100000 returned records that we are looping over.
Would we need to use "await" if the calls to render are split into workers or will JSreport handle this.

Thanks

[jan_blaha]

There are many ways how to reach parallel processing of your records with a concurrency limit.
I would personally use bluebird module and map function with concurrency limit set to something reasonable like 5.
http://bluebirdjs.com/docs/api/promise.map.html

I would recommend checking some tutorials on javascript async/await and promises.

[palphonse]

Hello again,

Just to recap my problem.We are using chrome-pdf to run a scheduled report, when it's time for the report to run, we need to generate a report for a large number of users (could reach 100,000) So we would need one scheduled template (template 1) to render another template (template 2) with a specific parameter. So template 1 would need to render template 2 100,000 times.
From what I understood from your help and the documentation, we would need to adjust some settings in the config file and use a module to split the requests made to template 2 from template 1. Is this correct?
Could you please advise on what settings would be needed, and how to go about this?
Thank again

[jan_blaha]

From what I understood from your help and the documentation, we would need to adjust some settings in the config file

Setting "timeout" to a very high number should be enough for the start.

use a module to split the requests made to template 2 from template 1

The bluebirdjs I mentioned is a common utility library. You can fulfill the task also without it with few lines of code.

In case you want to have multiple jsreport physical servers to load balance the report generation, you should consider a different architecture. Maybe have the scheduler running inside your app and triggering the 100 000 requests from your app.

[palphonse]

Hello again,

I set the numberofworkers to 4, and set concurrency to 4 in bluebirdjs and trying to render 1000 requests for the purpose of testing. It is working, but I didn't notice a difference in performance if i changed the numberofworkers to lets say 2 or 1.
Why could this be?

Thanks

[jan_blaha]

Great! Glad you are moving forward.

Please try config like this. And modify the numbers. You should see the difference. At least for the chrome-pdf recipe.

{
  "templatingEngines": {
    "numberOfWorkers": 2,
    "strategy": "http-server"
  },
  "chrome": {
     "numberOfWorkers": 2,
     "strategy": "chrome-pool"
  }
}

The numberOfWorkers property is currently being used only when you use http-server and chrome-pool strategy.

[palphonse]

Hello, thank you, these are the settings i had, i also used 2 for both

[jan_blaha]

I tried this simple test script using the sample invoice template.

const jsreport = require('jsreport-proxy')
const Promise = require('bluebird')

function beforeRender(req, res) {
    const reportsToRun = []
    for (let i = 0; i < 100; i++) {
        reportsToRun.push(i)
    }

    return Promise.map(reportsToRun, (i) => jsreport.render({
        template: {
            name: 'invoice-main'
        }        
    }), {
        concurrency: 4
    })
}

Results
numberOfWorkers: 2
23s

numberOfWorkers: 1
37s

Maybe try the same with the invoice template the first, to see if it makes difference or not.

[palphonse]

Ok thanks I will try it. The difference is that the second template that is being rendered uses mssql to read data could this be a reason?

[palphonse]

Hello again, I tried the sample, and i noticed a few seconds difference, however when I run my template, no difference is noticed.Could this be because we are using mssql to get data from a db?

[jan_blaha]

few seconds difference

It should be close to 2x performance.

Could this be because we are using mssql to get data from a db?

Hm, please try to share the output logs from jsreport. Maybe there is a "blocking" operation that is consuming the most of the time and the performance improvement with parallelization is then minimal. Let's say data query to the db takes 90% of the report rendering time. And the query consumes whole db time so the second query from the second report just waits until the first one is finished. In this case, the parallelization has no effect.

[palphonse]

thanks,

so from what i understand jsreport will not process multiple sql queries in parallel?

[palphonse]

Hello again,
Please note that we are using a server with 16 CPUs.
We tested by doubling the numberOfWorkers in the jsReport config starting at 1 up to 16, with concurrency = numberOfWorkers x 4 ( from 4 till 64 respectively)
Using this setup, we didn't get any noticeable performance boost after numberOfWorkers = 2 and concurrency = 8.
However, executing the same setup without fetching data from DB resulted in better performance all the way till numberOfWorkers = 8 and concurrency = 32.
During the report generation, we monitored the SQL server and it was running under 20% of its maximum CPU usage since we used a very simple SELECT query.

Below is a link to the requested log (I was not able to attach the file to the message).
The log was for numberOfWorkers =16 concuurency = 64

https://we.tl/t-ZHCwh4PF8F

Thanks

[jan_blaha]

You should enable the debug logs, it is typically done using jsreport configure command, or by adding to the config

"logger": {
    "console": {
      "transport": "console",
      "level": "debug"
    },
    "file": {
      "transport": "file",
      "level": "info",
      "filename": "logs/reporter.log"
    },
    "error": {
      "transport": "file",
      "level": "error",
      "filename": "logs/error.log"
    }
}

However, the 20% CPU doesn't mean your scripts aren't waiting for the database. They still likely do. It can be because of data IO or something else.

[palphonse]

Hello we already enabled debug logs in the config file, please find the link to the reporter.log file (we were not able to attach it here)The error.log file is empty.

https://we.tl/t-ZHCwh4PF8F

Thanks