Azure app service with docker, timeout when waiting for worker
-
We're running jsreport in a docker container in Azure App service. At the moment we have to restart the app service at least daily because of these errors:
Timeout when waiting for worker Error: Timeout when waiting for worker at Timeout._onTimeout (/app/node_modules/@jsreport/advanced-workers/lib/pool.js:78:27) at listOnTimeout (node:internal/timers:557:17) at processTimers (node:internal/timers:500:7)The only logentry is:
info +0 Render request 1226 queued for execution and waiting for availible workerI can't determine what report is being requested because all report names are listed as 'anonymous'
Do you have any suggestions on where to start investigating the issue?
-
I also noticed that CPU usage is always high, even when no reports are being requested. It seems that the chrome render processes aren't quit after rendering. See this process list:
Some use 20 hours of CPU time and more..
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19177 231172 30 10 1394296 322008 13172 R 8.0 9.2 25:14.49 node 26387 231172 30 10 20.334g 51856 27696 S 8.0 1.5 37:29.21 chrome 1536 231172 30 10 20.326g 44028 27800 S 4.0 1.3 54:34.24 chrome 1558 231172 30 10 20.335g 54224 27704 S 4.0 1.5 58:36.88 chrome 9108 231172 30 10 20.355g 61560 24520 S 4.0 1.8 247:45.07 chrome 10141 231172 30 10 397568 101692 86240 S 4.0 2.9 0:25.14 chrome 10739 231172 30 10 397424 104532 89084 S 4.0 3.0 0:20.02 chrome 20189 231172 30 10 381264 36480 21268 S 4.0 1.0 16:52.08 chrome 23784 231172 30 10 20.335g 53404 28296 R 4.0 1.5 52:21.07 chrome 1149 231172 30 10 20.337g 50460 32684 S 0.0 1.4 0:00.45 chrome 1271 231172 30 10 20.337g 51848 34092 S 0.0 1.5 0:00.43 chrome 1480 231172 30 10 381004 35728 20576 S 0.0 1.0 17:44.48 chrome 1481 231172 30 10 389740 35988 20876 S 0.0 1.0 22:33.08 chrome 1489 231172 30 10 262832 23620 11040 S 0.0 0.7 0:00.07 chrome 1490 231172 30 10 262832 22772 10188 S 0.0 0.6 0:00.11 chrome 1492 231172 30 10 262832 23624 11044 S 0.0 0.7 0:00.08 chrome 1493 231172 30 10 262832 23428 10844 S 0.0 0.7 0:00.12 chrome 1571 231172 30 10 298220 31764 16896 S 0.0 0.9 0:00.65 chrome 1591 231172 30 10 303204 33828 17044 S 0.0 1.0 0:01.14 chrome 4811 231072 20 0 587332 23680 2108 S 0.0 0.7 1:03.10 node 9076 231172 30 10 381120 27444 18932 S 0.0 0.8 95:34.62 chrome 9079 231172 30 10 262832 14472 9204 S 0.0 0.4 0:00.07 chrome 9080 231172 30 10 262832 14540 9248 S 0.0 0.4 0:00.17 chrome 9124 231172 30 10 298212 22912 15128 S 0.0 0.7 0:01.27 chrome 10185 231172 30 10 262832 83484 70904 S 0.0 2.4 0:00.08 chrome 10187 231172 30 10 262832 83272 70692 S 0.0 2.4 0:00.13 chrome 10210 231172 30 10 0 0 0 Z 0.0 0.0 0:00.03 chrome 10225 231172 30 10 20.325g 109968 94364 R 0.0 3.1 1:01.95 chrome 10240 231172 30 10 298224 94128 79256 S 0.0 2.7 0:01.16 chrome 10246 231172 30 10 381260 34500 19388 S 0.0 1.0 74:13.79 chrome 10249 231172 30 10 262832 22388 9812 S 0.0 0.6 0:00.08 chrome 10250 231172 30 10 262832 21988 9408 S 0.0 0.6 0:00.36 chrome 10286 231172 30 10 20.355g 68200 24264 R 0.0 1.9 192:19.25 chrome 10305 231172 30 10 298200 30816 15956 S 0.0 0.9 0:03.12 chrome 10330 231172 30 10 0 0 0 Z 0.0 0.0 0:00.08 chrome 10333 231172 30 10 0 0 0 Z 0.0 0.0 0:01.23 chrome 10391 231172 30 10 0 0 0 Z 0.0 0.0 0:00.10 chrome 10410 231172 30 10 0 0 0 Z 0.0 0.0 0:10.95 chrome 10751 231172 30 10 262832 85512 72928 S 0.0 2.4 0:00.08 chrome 10752 231172 30 10 262832 85608 73024 S 0.0 2.4 0:00.10 chrome 10785 231172 30 10 20.325g 111908 96224 R 0.0 3.2 0:53.58 chrome 10833 231172 30 10 298220 95904 81016 S 0.0 2.7 0:00.66 chrome 16953 231072 20 0 5936 640 576 S 0.0 0.0 0:00.00 sleep 19123 231072 20 0 19776 88 4 S 0.0 0.0 0:00.05 startup.sh 19436 231072 20 0 69960 96 0 S 0.0 0.0 0:00.00 sshd 19441 231072 20 0 19772 1812 1604 S 0.0 0.1 0:05.69 webssh-watc+ 19443 231072 20 0 48868 60 0 S 0.0 0.0 0:00.00 runuser 19453 232075 20 0 3023696 37284 412 S 0.0 1.1 0:59.86 Kudu.Servic+ 19518 231072 20 0 10.585g 20644 20 S 0.0 0.6 0:00.69 node 19958 231072 30 10 1044 4 0 S 0.0 0.0 0:00.38 dummy 20002 231072 30 10 1044 4 0 S 0.0 0.0 0:00.36 dummy 20196 231172 30 10 262832 22720 10140 S 0.0 0.6 0:00.08 chrome 20197 231172 30 10 262840 22856 10276 S 0.0 0.7 0:00.42 chrome 20227 231172 30 10 20.343g 59248 27276 R 0.0 1.7 48:46.61 chrome 20248 231172 30 10 298248 30704 15868 S 0.0 0.9 0:09.12 chrome 20389 231172 30 10 20.339g 65200 33196 S 0.0 1.9 0:01.76 chrome 20527 231072 30 10 1044 4 0 S 0.0 0.0 0:00.63 dummy 20593 231072 30 10 1044 4 0 S 0.0 0.0 0:00.62 dummy 22701 231072 30 10 1044 4 0 S 0.0 0.0 0:01.60 dummy 23708 231172 30 10 380992 35088 19860 S 0.0 1.0 19:53.58 chrome 23718 231172 30 10 262832 23256 10672 S 0.0 0.7 0:00.07 chrome 23720 231172 30 10 262832 22716 10136 S 0.0 0.6 0:00.16 chrome 23803 231172 30 10 298220 31960 17084 S 0.0 0.9 0:01.58 chrome 23861 231172 30 10 0 0 0 Z 0.0 0.0 0:00.08 chrome 23862 231172 30 10 0 0 0 Z 0.0 0.0 0:00.71 chrome 23892 231172 30 10 0 0 0 Z 0.0 0.0 0:04.48 chrome 25437 231172 30 10 0 0 0 Z 0.0 0.0 0:00.14 chrome 25442 231172 30 10 0 0 0 Z 0.0 0.0 0:00.11 chrome 25495 231172 30 10 0 0 0 Z 0.0 0.0 0:00.07 chrome 25498 231172 30 10 0 0 0 Z 0.0 0.0 0:00.07 chrome 25500 231172 30 10 0 0 0 Z 0.0 0.0 0:00.07 chrome 25503 231172 30 10 0 0 0 Z 0.0 0.0 0:00.07 chrome 26248 231172 30 10 381244 35588 20408 S 0.0 1.0 14:11.32 chrome 26280 231172 30 10 381196 35860 20508 S 0.0 1.0 14:00.60 chrome 26283 231172 30 10 262832 23308 10728 S 0.0 0.7 0:00.08 chrome 26284 231172 30 10 262832 22748 10168 S 0.0 0.6 0:00.16 chrome 26291 231172 30 10 262832 23308 10728 S 0.0 0.7 0:00.08 chrome 26295 231172 30 10 262832 22764 10184 S 0.0 0.6 0:00.19 chrome 26318 231172 30 10 20.332g 50740 28256 S 0.0 1.4 37:18.04 chrome 26334 231172 30 10 298200 31688 16816 S 0.0 0.9 0:04.20 chrome 26406 231172 30 10 298244 30708 15864 S 0.0 0.9 0:04.72 chrome 29391 231172 30 10 380940 35788 20624 S 0.0 1.0 0:01.28 chrome 29397 231172 30 10 262832 23432 10848 S 0.0 0.7 0:00.08 chrome 29398 231172 30 10 262832 23412 10832 S 0.0 0.7 0:00.11 chrome 29429 231172 30 10 20.325g 45712 29704 S 0.0 1.3 0:00.08 chrome 29450 231172 30 10 298296 32040 17128 S 0.0 0.9 0:00.84 chrome 29763 231172 30 10 386672 35956 20684 S 0.0 1.0 27:47.29 chrome 29768 231172 30 10 262832 22732 10148 S 0.0 0.6 0:00.07 chrome 29769 231172 30 10 262832 23704 11124 S 0.0 0.7 0:00.28 chrome 29798 231172 30 10 20.338g 55520 27452 R 0.0 1.6 73:23.36 chrome 29814 231172 30 10 303208 33656 16884 S 0.0 1.0 0:02.39 chrome 29852 231172 30 10 386796 35936 20516 S 0.0 1.0 28:35.51 chrome 29855 231172 30 10 262832 23044 10468 S 0.0 0.7 0:00.08 chrome 29856 231172 30 10 262832 22720 10136 S 0.0 0.6 0:00.21 chrome 29885 231172 30 10 20.338g 55364 28152 R 0.0 1.6 73:16.82 chrome 29903 231172 30 10 303196 33548 16772 S 0.0 1.0 0:01.67 chrome
-
I can't determine what report is being requested because all report names are listed as 'anonymous'
Are your templates persisted? I mean does your request looks like
{ "template": { "name": "xxx" } }If you open the profiler from the startup page, you should see what templates are currently running
Note the user you are logged in with needs to have permissions to the templates.
It seems that the chrome render processes aren't quit after rendering. See this process list:
By default, every worker keeps a single instance of chrome running, to avoid chrome startup time.
https://jsreport.net/learn/chrome-pdf#chrome-process-allocationDo you apply some special configs?
-
According to the profiler nothing is still running, but the container is still using 100% CPU. There's no special config, apart from license key and admin users and such
-
I have done some load tests and there seems to be some problem in the chrome version used in the 3.7.1. We will release today jsreport 3.8.0 with the updated chrome which should eventually fix this.
-
The 3.8.0 is published, please try it out.
-
Number of processes seems much more normal right now. Also the reports are faster and cpu during processing is lower.
Is this just an update chrome version?
-
The 3.8.0 contains all kinds of things...
https://jsreport.net/blog/release-380
https://github.com/jsreport/jsreport/releasesThe 3.8.0 default image includes updated chrome and the updated alpine base image.
In our experience chrome in some versions isn't working as it should. Sometimes its performance degrades for example.
