slow import and export with azure blob storage fs persistence

[sammeel]

We have the jsreport docker full running on an azure app service (P1V3) with a connection to an azure storage.

Lately we have been having a lot of issues with timeouts when doing an import through both powershell and the jsreport cli.

An app service has a timeout of about 200seconds which we seem to hit each and every time.
I've been able to reproduce this behavior locally where imports get slower depending on the amount of files already known by jsreport. My guess is the azure-storage implementation that seems to be overloading calls to the storage account.
We've already made our blob storage into a premium one (which uses SSDs instead of magnetic disks), but this does not give a difference.

These are the timings I get when binding my local environment to the storage account and doing imports for 4 projects with the same files (206 entities) in it:

import	import time storage account
1	1,1minute
2	2,1minutes
3	3.2minutes
4	4.3minutes

In the monitoring we see a huge amount of calls going towards the storage account:

Not sure if related, but during this long import/export, we see many of these messages in the console:

When doing an export of a single file after the 4 imports above (so 4 times 205 entities), it took me 4.5 minutes:

These are the requests that have been made to the storage account during that 4 minute export (no other load than me):

Do we have some mistake in our configuration? Or is there something else going on here?

This is our configuration:

{
  "httpPort": 5488,
  "store": {
    "provider": "fs"
  },
  "blobStorage": {
    "provider": "fs"
  },
  "trustUserCode": true,
  "reportTimeout": 180000,
  "workers": {
    "numberOfWorkers": 8
  },
  "extensions": {
    "fs-store": {
      "externalModificationsSync": false,
      "persistence": {
        "provider": "azure-storage"
      }
    },
    "chrome-pdf": {
      "launchOptions": {
        "timeout": 180000,
        "args": ["--no-sandbox", "--disable-setuid-sandbox"],
        "ignoreDefaultArgs": ["--disable-extensions"]
      }
    },
    "fs-store-azure-storage-persistence": {
      "accountName": "<account name>",
      "accountKey": "<secret>",
      "container": "<container>"
    },
    "authentication": {
      "cookieSession": {},
      "admin": {
        "username": "admin",
        "password": "password"
      },
      "enabled": false
    },
    "scripts": {
      "strategy": "http-server"
    }
  },
  "templatingEngines": {
    "strategy": "http-server"
  },
  "chrome": {
    "timeout": 180000,
    "strategy": "chrome-pool",
    "numberOfWorkers": 8
  },
  "sandbox": {
    "allowedModules": "*"
  }
}

[admin]

Are you using the latest jsreport? There were some optimizations added in v4.
Do you have the Azure app service in the same region as the blob storage account?

The import runs in the transaction which needs to guarantee consistency for accidental server failures.
This requires more writes and low latencies from the file system. We will check if there are some optimization options.
However, you may have better results with a mounted Azure disk. Or a full-blown db like mongo.

[sammeel]

The above test was tried with version 4.3.1 as the 4.4.0 was giving us issues.

The azure app service and blob storage are running in the same region and zone.

I'll try with the mounted disk and mongodb (on cosmosdb) and let you know the results. Might be a couple of days before I managed to test it though.