JSReport as Azure Function Timeout when waiting worker error.

GoldenGuillaume

Use case

I developed an typescript azure function in consumption plan with scale out limited to 15 instances, hosted on Linux to generate reports with the help of jsreport library based on this tutorial: https://jsreport.net/learn/azure-functions-serverless.

Currently the use case is the following:

A Web APP API is using as proxy between external services and the JSReport function, the api is called then call the function endpoint to return the report.

Problem

Currently when executing some JMeter tests with more or less 120 requests the azure function throw the following error:

Error: Timeout when waiting for worker
at Timeout._onTimeout (/home/site/wwwroot/node_modules/@jsreport/advanced-workers/lib/pool.js:78:27)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)

My production config file is the following:

{
    "httpPort": 5488,
    "allowLocalFilesAccess": true,
    "trustUserCode": true,
    "licenseKey": "#############",
    "reportTimeout": 100000,
    "workers": {
        "numberOfWorkers": 1
    },
    "store": {
        "provider": "fs"
    },
    "blobStorage": {
        "provider": "fs",
        "dataDirectory": "/tmp/storage"
    },
    "logger": {
        "file": {
            "silent": true
        },
        "error": {
            "silent": true
        }
    },
    "extensions": {
        "chrome-pdf": {
            "launchOptions": {
                "args": ["--no-sandbox"]
            }
        },
        "authentication": {
            "enabled": false
        },
        "authorization": {
            "enabled": false
        },
        "cli": {
            "enabled": false
        },
        "express": {
            "enabled": false
        },
        "freeze": {
            "enabled": false
        },
        "fs-store": {
            "dataDirectory": "/tmp/data"
        },
        "import-export": {
            "enabled": false
        },
        "public-templates": {
            "enabled": false
        },
        "sample-template": {
            "enabled": false
        },
        "scheduling": {
            "enabled": false
        },
        "studio": {
            "enabled": false
        },
        "studio-theme-dark": {
            "enabled": false
        },
        "tags": {
            "enabled": false
        },
        "version-control": {
            "enabled": false
        }
    }
}

dependencies versions:

"dependencies": {
    "bwip-js": "^4.1.1",
    "fs-extra": "^10.0.1",
    "jsreport": "3.10.0",
    "zod": "^3.22.2"
  },
  "devDependencies": {
    "@azure/functions": "^3.0.0",
    "@types/fs-extra": "^11.0.2",
    "@types/jsreport": "^2.9.1",
    "@types/node": "16.x",
    "@typescript-eslint/eslint-plugin": "^6.7.0",
    "@typescript-eslint/parser": "^6.7.0",
    "eslint": "^8.49.0",
    "rimraf": "^5.0.0",
    "typescript": "^4.0.0"
  }

From now i tried several thinks as decreasing/increasing the number of workers, without any effect.
Increasing the Node heap size with --max-old-space-size=1024 to avoid a previously error which was:

[WORKER_CRASHED]: Worker terminated due to reaching memory limit: JS heap out of memory

The template who is tested is a file containing a qr code generated by a script.

admin

The workers.numberOfWorkers config limits the parallelism. Every worker can process only one parallel request.
If you send 120 requests at once, the requests that won't have any free workers will wait in the queue and eventually fail on Error: Timeout when waiting for worker.

Increasing workers.numberOfWorkers should help but if you increase it too much, you will reach the server limits. A good value for the start can be for example 2x the number of CPUs.

You mentioned you have a scaling limit of 15 instances. However this typically doesn't have immediate effect. If you send 120 requests, it will reach just one instance and overload it resulting in the mentioned timeout.