Import of exported files takes too long using AWS S3



  • Hello dear jsReport team,

    currently we are using jsReport in the version 2.11.0 with:

    jsreport-fs-store-aws-s3-persistence@1.4.1 (with the fix for umlauts and non ascii file/folder names)
    jsreport-fs-store-aws-sns-sync@1.2.1 
    

    After/during the import a ~.tran/ folder is created

    0_1629706395667_upload-4997215b-6ced-4a97-b7a8-3dd7617ab23d

    After a while, under 1 Minute all the needed files are displayed in the GUI and can be used. As soon as jsReport is restarted e.g. the next day all the files which are in the .~tran folder are not visible or accessible anymore via the GUI.

    Also it seems that the synchronization of the import takes too long to respond in the GUI although the file itself is only around 600kb in size.

    Can you give me advice on how to improve the speed of the import of the zip file or is this a knonw issue that it takes too long using AWS S3 for importing files?

    I tried the same thing to export and import the current templates without AWS S3 and this actions takes only seconds. With AWS S3 it takes around 3 minutes.



  • Hi,

    the import uses a transactional approach to assure consistency.
    The ~tran folder contains the dirty changes from the transaction which you shouldn't see in the studio.
    The ~tran folder should get deleted as soon as the transactions commit.

    It's strange you see the changes in the studio when the ~tran folder is still there.

    The transactional approach requires many operations to assure consistency and can be slow with s3.
    Especially when there isn't a low network latency between the server and s3. Do you run the server in AWS?

    I tried importing hundreds of entities from my local home to s3 and it can take easily 3 minutes. However, I don't see such issues with persistence you do. When possible, please email me your current workspace and what you are importing. I would like to try it out and see if I can replicate the problem.



  • Hi Mr. @jan_blaha,

    did you also tried a merge import after a full import? In this case it is super often the case that although the import is done (regarding the logs) the ~.tran file is still there and not deleted.



  • Yes, I tried a couple of times.



  • Mr. Blaha,

    do you have an import/export file you could send me which was working for you so I can try it out with another data sample on our environment? So I can also verify your outcome.

    Importing partially (less data) from one instance to another in our environment leads to more correct behaviour. This means the data is imported and the ~.tran folder is deleted afterwards. But this is not our normal use case. Usually, we simply want to merge the data from e.g. Staging to Production.



  • You can try this one with 200 entities

    https://1drv.ms/u/s!AogtKyPn-GjbhY4veDB5Nqg8mKuRqQ?e=1Goz7w


Log in to reply
 

Looks like your connection to jsreport forum was lost, please wait while we try to reconnect.