Nested Each slow performing

zewar96

We have a report that is building data in to an Excel file. The data is about 28MB, so it's a large amount of records. Some of those records have nested data. If we do an #each in handlebars and then nest another #each underneath it, it is SIGNIFICANTLY slower than if we create a JS function called {{{ CreateLookupColumns }}} and pass in that data and have it build the records and return a single string back. I am wondering if there are known performance issue with nesting #each statements because of context switching and if as a standard practice we should avoid the nested construct with large data sets.

Unfortunately I cannot publicly post this example, but if you need to see a working example, I can show you where it takes 45+seconds when using nested each statements and 9 seconds when it's using the single JS function call that does the exact same code. We have narrowed everything down to it being the nested Each statement that is the issue and i can only think of the context switching as being the slowness.

Just trying to understand the recommended way that we should build the reports as it works correctly both ways, just seems to be a lot slower one way over the other.

-Thanks

zewar96

I should also mention that I am still on 3.11.3, but hope to move to 3.13 soon. I am just having some issue with our Docker and permissions on the new version of Node

jan_blaha

Do you mean you work with xlsx recipe?
https://jsreport.net/learn/xlsx
I am asking because there are also other recipes producing excel in jsreport.

Do you type handlebars directly to the xlsx template in excel or use transformation xlsxAdd helpers?
This is distinguished as generation vs transformation in the documentation.
What do you do in the loop actually? Perhaps a simple minimal abstract playground demo with no real data would help us the best.

zewar96

Yes, this is the xlsx recipe.

We use the xslxAdd helper

Here is an example of the From and the To

{{#each data}}
    {{#xlsxAdd "xl/worksheets/sheet1.xml" "worksheet.sheetData[0].row"}}
        <row>
            <c t="inlineStr"><is><t>{{field1}}</t></is></c>
            <c t="inlineStr"><is><t>{{field2}}</t></is></c>
            <c t="inlineStr"><is><t>{{field3}}</t></is></c>
            <c t="inlineStr"><is><t>{{field4}}</t></is></c>
            <c t="inlineStr"><is><t>{{field5}}</t></is></c>
            <c s="11"><v>{{getDayDif startDate}}</v></c>
            <c s="11"><v>{{getDayDif endDate}}</v></c>
            <c t="inlineStr"><is><t>{{field6}}</t></is></c>
            <c t="inlineStr"><is><t>{{field7}}</t></is></c>
            <c t="inlineStr" s="9"><is><t>{{field8}}</t></is></c>
            {{#each ../customAttributes}}
              <c t="inlineStr"><is><t>{{getAttributeValue ../attributes id 0}}</t></is></c>
            {{/each}}
              </row>
          {{/xlsxAdd}}
      {{/each}}

This was changed to:

{{#each data}}
    {{#xlsxAdd "xl/worksheets/sheet1.xml" "worksheet.sheetData[0].row"}}
        <row>
            <c t="inlineStr"><is><t>{{field1}}</t></is></c>
            <c t="inlineStr"><is><t>{{field2}}</t></is></c>
            <c t="inlineStr"><is><t>{{field3}}</t></is></c>
            <c t="inlineStr"><is><t>{{field4}}</t></is></c>
            <c t="inlineStr"><is><t>{{field5}}</t></is></c>
            <c s="11"><v>{{getDayDif startDate}}</v></c>
            <c s="11"><v>{{getDayDif endDate}}</v></c>
            <c t="inlineStr"><is><t>{{field6}}</t></is></c>
            <c t="inlineStr"><is><t>{{field7}}</t></is></c>
            <c t="inlineStr" s="9"><is><t>{{field8}}</t></is></c>
            {{{outputCustomAttributes ../customAttributes attributes}}}
        </row>
    {{/xlsxAdd}}
{{/each}}

The outputCustomAttributes JS function is an exact copy of the code that happens within the #each above.

I will try to work up a playground example with fake Data that can show a working example of the difference in performance.

zewar96

a sample workspace was created
https://playground.jsreport.net/w/zewar96/ORg9CbKr

if you run the top section only (the fast one), according to profile, "Successfully zipped now." takes 2751ms
if you run the bottom section only (the slow one), according to profile, "Successfully zipped now." takes 9622ms.

The only difference between these 2 are that one uses the #each in handlebars and the other uses the same loop and same process, but does it all in JS and returns a single value. I played with this a bunch and even if i remove any of the lookups that i do an put in static values, the #each is still significantly slower and i don't know why.

jan_blaha

Thank you for the demo. I see the difference and analyze if there is something to do about it. I post back with updates.

zewar96

any ideas on what is causing this slowness? i don't want to have to move to this new reporting style as it's very hard to read/maintain

admin

I am still trying to improve this. I will update this week.

zewar96

Checking on status. Have you found the underlying cause of the slowness?

admin

I am still working on it and still don't know if we will be able to do something about it or not.

The problem is that there is some overhead in handlebars to call a helper function and this overhead gets much bigger when we use tweaked handlebars with support for async helpers. I will update you when I have some resolution about this.

admin

We finally identified the problematic point and the next jsreport update should solve this.

The problem is that we use nodejs domains to provide better errors to the users when evaluating user code.
The nodejs itself has issues in domains implementation causing bad performance on async functions evaluation.
We talk about 10x worse performance when running an empty async function in domain vs not in domain.
We will ship the next release without using domains because we think it isn't now necessary.

zewar96

That's fantastic news. Thanks for digging in to this as i know it wasn't easy to find and fix. Hopefully the performance improvements will be a benefit to everyone. I know they will be huge for us and save us from having to rewrite a bunch of reports using a non-standard way.