Generating same report for different parameters automatically

palphonse

Hello again,

I set the numberofworkers to 4, and set concurrency to 4 in bluebirdjs and trying to render 1000 requests for the purpose of testing. It is working, but I didn't notice a difference in performance if i changed the numberofworkers to lets say 2 or 1.
Why could this be?

Thanks

jan_blaha

Great! Glad you are moving forward.

Please try config like this. And modify the numbers. You should see the difference. At least for the chrome-pdf recipe.

{
  "templatingEngines": {
    "numberOfWorkers": 2,
    "strategy": "http-server"
  },
  "chrome": {
     "numberOfWorkers": 2,
     "strategy": "chrome-pool"
  }
}

The numberOfWorkers property is currently being used only when you use http-server and chrome-pool strategy.

palphonse

Hello, thank you, these are the settings i had, i also used 2 for both

jan_blaha

I tried this simple test script using the sample invoice template.

const jsreport = require('jsreport-proxy')
const Promise = require('bluebird')

function beforeRender(req, res) {
    const reportsToRun = []
    for (let i = 0; i < 100; i++) {
        reportsToRun.push(i)
    }

    return Promise.map(reportsToRun, (i) => jsreport.render({
        template: {
            name: 'invoice-main'
        }        
    }), {
        concurrency: 4
    })
}

Results
numberOfWorkers: 2
23s

numberOfWorkers: 1
37s

Maybe try the same with the invoice template the first, to see if it makes difference or not.

palphonse

Ok thanks I will try it. The difference is that the second template that is being rendered uses mssql to read data could this be a reason?

palphonse

Hello again, I tried the sample, and i noticed a few seconds difference, however when I run my template, no difference is noticed.Could this be because we are using mssql to get data from a db?

jan_blaha

few seconds difference

It should be close to 2x performance.

Could this be because we are using mssql to get data from a db?

Hm, please try to share the output logs from jsreport. Maybe there is a "blocking" operation that is consuming the most of the time and the performance improvement with parallelization is then minimal. Let's say data query to the db takes 90% of the report rendering time. And the query consumes whole db time so the second query from the second report just waits until the first one is finished. In this case, the parallelization has no effect.

palphonse

thanks,

so from what i understand jsreport will not process multiple sql queries in parallel?

palphonse

Hello again,
Please note that we are using a server with 16 CPUs.
We tested by doubling the numberOfWorkers in the jsReport config starting at 1 up to 16, with concurrency = numberOfWorkers x 4 ( from 4 till 64 respectively)
Using this setup, we didn't get any noticeable performance boost after numberOfWorkers = 2 and concurrency = 8.
However, executing the same setup without fetching data from DB resulted in better performance all the way till numberOfWorkers = 8 and concurrency = 32.
During the report generation, we monitored the SQL server and it was running under 20% of its maximum CPU usage since we used a very simple SELECT query.

Below is a link to the requested log (I was not able to attach the file to the message).
The log was for numberOfWorkers =16 concuurency = 64

https://we.tl/t-ZHCwh4PF8F

Thanks

jan_blaha

You should enable the debug logs, it is typically done using jsreport configure command, or by adding to the config

"logger": {
    "console": {
      "transport": "console",
      "level": "debug"
    },
    "file": {
      "transport": "file",
      "level": "info",
      "filename": "logs/reporter.log"
    },
    "error": {
      "transport": "file",
      "level": "error",
      "filename": "logs/error.log"
    }
}

However, the 20% CPU doesn't mean your scripts aren't waiting for the database. They still likely do. It can be because of data IO or something else.

palphonse

Hello we already enabled debug logs in the config file, please find the link to the reporter.log file (we were not able to attach it here)The error.log file is empty.

https://we.tl/t-ZHCwh4PF8F

Thanks