Slow Export
-
just exported 3 folders (a total of 17 items) and it took it almost 2 full minutes
That is strange. What template store do you use? I mean what you have in config
store.provider.
-
"store": {
"provider": "fs"
},This mounts to a folder that is a EFS mount within AWS which should have no issues with speed. Just to be sure i checked EFS metrics and you can see 2 small blips which were the 2 exports that i tested with. This caused a 2% usage of the IO limit.
-
What if you try the same export with the same data locally, on your dev machine, with a local disk?
-
I will load a clean instance and export/import the entire structure to something running local and let you know. Unfortunately i will not be able to get to that until tomorrow morning. Thank you for the suggestion. I will let you know the results
-
I was able to do a quick test. There is definitely something weird with our deployed version that I am going to have to figure out. Here are the results from running locally
Export all folders and files 13.3 seconds, 11.3MB file with 1403 total items
(same test as before) Export 3 folders 8.696 seconds, 34Kb file, 17 items
(same test as before) Export 1 temlate 8.673 seconds, 2Kb fileThere are MANY differences in my test though. The deployed version is a docker instance and has shared CPU/RAM and is mounted to an EFS instance. This test was running directly on my laptop with nothing else running. When running this export from the deployed server i see a spike in CPU and RAM, but only to 30% of allocated CPU and 88% of allocated RAM. It's possible the RAM is being throttled and it's using disk swap space, but I don't think so
Do you suspect the issue is CPU related, Memory related, or disk related?
-
The fs store keeps data in memory. So it needs just CPU and Memory for export.
Export 3 folders 8.696 seconds, 34Kb file, 17 items
This should be ~300ms typically.
Could you try to export the showcase workspace? How long that takes for you?
https://playground.jsreport.net/w/admin/S3xqZ0Zc
For me its 300ms.
-
Yes that is about right...~300ms
What is the backend file store for this?
-
The idea was that you export from the playground, import to your local and try the export there.
The playground uses mongodb as template store, but the fs store should be the fastest anyway.
-
ok...sorry i misunderstood. I took your export, added it to my dev environment (clean setup from https://jsreport.net/learn/windows with 1 change to add the license to the config). I then added all of my templates from our test server and did the export of your stuff.
When I exported just your items it took almost 11 seconds. This makes me think that it is doing something disk related to find what it needs and the iteration of the items to get what it needs is causing the slowdown.
When I watch what is happening on my machine in Resource Monitor, i can see that it's processing files on disk and spikes CPU and disk, but doesn't do much at all with memory.
I changed the backend to be PostgreSQL for templates and all of the speed issues have been fixed on the export.
-
Strange, maybe its some antivirus thing?
I don't see another reason why it should be 30x slower on your machine.
Don't you have authorization enabled?
-
No authorization used. This is just a clean load using your windows instructions, running locally via powershell to load Node.js. The only thing i did was to add 1400+ templates to the system before exporting.
