Import of exported files takes too long using AWS S3
-
Hello dear jsReport team,
currently we are using jsReport in the version 2.11.0 with:
jsreport-fs-store-aws-s3-persistence@1.4.1 (with the fix for umlauts and non ascii file/folder names) jsreport-fs-store-aws-sns-sync@1.2.1After/during the import a
~.tran/folder is created
After a while, under 1 Minute all the needed files are displayed in the GUI and can be used. As soon as jsReport is restarted e.g. the next day all the files which are in the
.~tranfolder are not visible or accessible anymore via the GUI.Also it seems that the synchronization of the import takes too long to respond in the GUI although the file itself is only around
600kbin size.Can you give me advice on how to improve the speed of the import of the zip file or is this a knonw issue that it takes too long using AWS S3 for importing files?
I tried the same thing to export and import the current templates without AWS S3 and this actions takes only seconds. With AWS S3 it takes around 3 minutes.
-
Hi,
the import uses a transactional approach to assure consistency.
The~tranfolder contains the dirty changes from the transaction which you shouldn't see in the studio.
The~tranfolder should get deleted as soon as the transactions commit.It's strange you see the changes in the studio when the
~tranfolder is still there.The transactional approach requires many operations to assure consistency and can be slow with s3.
Especially when there isn't a low network latency between the server and s3. Do you run the server in AWS?I tried importing hundreds of entities from my local home to s3 and it can take easily 3 minutes. However, I don't see such issues with persistence you do. When possible, please email me your current workspace and what you are importing. I would like to try it out and see if I can replicate the problem.
-
Hi Mr. @jan_blaha,
did you also tried a merge import after a full import? In this case it is super often the case that although the import is done (regarding the logs) the
~.tranfile is still there and not deleted.
-
Yes, I tried a couple of times.
-
Mr. Blaha,
do you have an import/export file you could send me which was working for you so I can try it out with another data sample on our environment? So I can also verify your outcome.
Importing partially (less data) from one instance to another in our environment leads to more correct behaviour. This means the data is imported and the ~.tran folder is deleted afterwards. But this is not our normal use case. Usually, we simply want to merge the data from e.g. Staging to Production.
-
You can try this one with 200 entities
-
Dear @jan_blaha,
unfortunately, we still have the issue that we can not reliably work with jsReport because after re-starts of jsReport the imported files disappear from the Application and we can not generate files.
Is there a way how I can send you our sample export files in different versions so you can take a look at why it is it not working?
We already tried a look at different things, like importing in different jsReport versions etc.
Also your provided sample was working on our side without an issue. But as far as I can see your sample is around 103 kb in size and our samples are already around 600 to 700 kb in size.
-
Please email it to me jan.blaha@jsreport.net
The import performance was significantly improved in jsreport v3.
If its an option for you, you can try the update.
-
Hi @jan_blaha,
I can verify that the import in version 3.0.1 is way faster. The other good thing is also that is visible now in the logs when the import starts and when it is finished. We are thinking about an upgrade. Congratulations on the new version.
Unfortunately, we also tried the import of our templates on the new version 3.0.1 but with the same issues.
-
Please email me the workspace zip and config file and describe the steps to reproduce the issue. Thank you.
Maybe you would have better results with mapping normal file system disk and using the default fs store instead of the s3.
