Slow Export

  • The more templates/folders/etc we add, the slower the export function becomes. I wouldn't think that exporting a single folder (with maybe 5 items) would get slower as we add other folders and other templates. There may be some inefficiencies in the export code that need to be reviewed when you guys have some time. For a small example, I just exported 3 folders (a total of 17 items) and it took it almost 2 full minutes to generate a 34Kb .jsrexport file. It never used to take this long, but as we have grown the product and number of reports in our system it has continue to get slower and slower. Another example of why i think it's inefficient is that if i export a single template, it also takes the exact amount of time to export but i end up with a 3Kb file.

  • I forgot to mention that i am on 3.10.0

  • just exported 3 folders (a total of 17 items) and it took it almost 2 full minutes

    That is strange. What template store do you use? I mean what you have in config store.provider.

  • "store": {
    "provider": "fs"

    This mounts to a folder that is a EFS mount within AWS which should have no issues with speed. Just to be sure i checked EFS metrics and you can see 2 small blips which were the 2 exports that i tested with. This caused a 2% usage of the IO limit.

  • What if you try the same export with the same data locally, on your dev machine, with a local disk?

  • I will load a clean instance and export/import the entire structure to something running local and let you know. Unfortunately i will not be able to get to that until tomorrow morning. Thank you for the suggestion. I will let you know the results

  • I was able to do a quick test. There is definitely something weird with our deployed version that I am going to have to figure out. Here are the results from running locally

    Export all folders and files 13.3 seconds, 11.3MB file with 1403 total items
    (same test as before) Export 3 folders 8.696 seconds, 34Kb file, 17 items
    (same test as before) Export 1 temlate 8.673 seconds, 2Kb file

    There are MANY differences in my test though. The deployed version is a docker instance and has shared CPU/RAM and is mounted to an EFS instance. This test was running directly on my laptop with nothing else running. When running this export from the deployed server i see a spike in CPU and RAM, but only to 30% of allocated CPU and 88% of allocated RAM. It's possible the RAM is being throttled and it's using disk swap space, but I don't think so

    Do you suspect the issue is CPU related, Memory related, or disk related?

  • The fs store keeps data in memory. So it needs just CPU and Memory for export.

    Export 3 folders 8.696 seconds, 34Kb file, 17 items

    This should be ~300ms typically.

    Could you try to export the showcase workspace? How long that takes for you?
    For me its 300ms.

  • Yes that is about right...~300ms

    What is the backend file store for this?

  • The idea was that you export from the playground, import to your local and try the export there.

    The playground uses mongodb as template store, but the fs store should be the fastest anyway.

  • ok...sorry i misunderstood. I took your export, added it to my dev environment (clean setup from with 1 change to add the license to the config). I then added all of my templates from our test server and did the export of your stuff.

    When I exported just your items it took almost 11 seconds. This makes me think that it is doing something disk related to find what it needs and the iteration of the items to get what it needs is causing the slowdown.

    When I watch what is happening on my machine in Resource Monitor, i can see that it's processing files on disk and spikes CPU and disk, but doesn't do much at all with memory.

    I changed the backend to be PostgreSQL for templates and all of the speed issues have been fixed on the export.

  • Strange, maybe its some antivirus thing?
    I don't see another reason why it should be 30x slower on your machine.
    Don't you have authorization enabled?

  • No authorization used. This is just a clean load using your windows instructions, running locally via powershell to load Node.js. The only thing i did was to add 1400+ templates to the system before exporting.

Log in to reply

Looks like your connection to jsreport forum was lost, please wait while we try to reconnect.