Azure app service with docker, timeout when waiting for worker



  • We're running jsreport in a docker container in Azure App service. At the moment we have to restart the app service at least daily because of these errors:

    Timeout when waiting for worker
    Error: Timeout when waiting for worker
        at Timeout._onTimeout (/app/node_modules/@jsreport/advanced-workers/lib/pool.js:78:27)
        at listOnTimeout (node:internal/timers:557:17)
        at processTimers (node:internal/timers:500:7)
    

    The only logentry is:
    info +0 Render request 1226 queued for execution and waiting for availible worker

    I can't determine what report is being requested because all report names are listed as 'anonymous'

    Do you have any suggestions on where to start investigating the issue?



  • I also noticed that CPU usage is always high, even when no reports are being requested. It seems that the chrome render processes aren't quit after rendering. See this process list:

    Some use 20 hours of CPU time and more..

      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    19177 231172    30  10 1394296 322008  13172 R  8.0  9.2  25:14.49 node
    26387 231172    30  10 20.334g  51856  27696 S  8.0  1.5  37:29.21 chrome
     1536 231172    30  10 20.326g  44028  27800 S  4.0  1.3  54:34.24 chrome
     1558 231172    30  10 20.335g  54224  27704 S  4.0  1.5  58:36.88 chrome
     9108 231172    30  10 20.355g  61560  24520 S  4.0  1.8 247:45.07 chrome
    10141 231172    30  10  397568 101692  86240 S  4.0  2.9   0:25.14 chrome
    10739 231172    30  10  397424 104532  89084 S  4.0  3.0   0:20.02 chrome
    20189 231172    30  10  381264  36480  21268 S  4.0  1.0  16:52.08 chrome
    23784 231172    30  10 20.335g  53404  28296 R  4.0  1.5  52:21.07 chrome
     1149 231172    30  10 20.337g  50460  32684 S  0.0  1.4   0:00.45 chrome
     1271 231172    30  10 20.337g  51848  34092 S  0.0  1.5   0:00.43 chrome
     1480 231172    30  10  381004  35728  20576 S  0.0  1.0  17:44.48 chrome
     1481 231172    30  10  389740  35988  20876 S  0.0  1.0  22:33.08 chrome
     1489 231172    30  10  262832  23620  11040 S  0.0  0.7   0:00.07 chrome
     1490 231172    30  10  262832  22772  10188 S  0.0  0.6   0:00.11 chrome
     1492 231172    30  10  262832  23624  11044 S  0.0  0.7   0:00.08 chrome
     1493 231172    30  10  262832  23428  10844 S  0.0  0.7   0:00.12 chrome
     1571 231172    30  10  298220  31764  16896 S  0.0  0.9   0:00.65 chrome
     1591 231172    30  10  303204  33828  17044 S  0.0  1.0   0:01.14 chrome
     4811 231072    20   0  587332  23680   2108 S  0.0  0.7   1:03.10 node
     9076 231172    30  10  381120  27444  18932 S  0.0  0.8  95:34.62 chrome
     9079 231172    30  10  262832  14472   9204 S  0.0  0.4   0:00.07 chrome
     9080 231172    30  10  262832  14540   9248 S  0.0  0.4   0:00.17 chrome
     9124 231172    30  10  298212  22912  15128 S  0.0  0.7   0:01.27 chrome
    10185 231172    30  10  262832  83484  70904 S  0.0  2.4   0:00.08 chrome
    10187 231172    30  10  262832  83272  70692 S  0.0  2.4   0:00.13 chrome
    10210 231172    30  10       0      0      0 Z  0.0  0.0   0:00.03 chrome
    10225 231172    30  10 20.325g 109968  94364 R  0.0  3.1   1:01.95 chrome
    10240 231172    30  10  298224  94128  79256 S  0.0  2.7   0:01.16 chrome
    10246 231172    30  10  381260  34500  19388 S  0.0  1.0  74:13.79 chrome
    10249 231172    30  10  262832  22388   9812 S  0.0  0.6   0:00.08 chrome
    10250 231172    30  10  262832  21988   9408 S  0.0  0.6   0:00.36 chrome
    10286 231172    30  10 20.355g  68200  24264 R  0.0  1.9 192:19.25 chrome
    10305 231172    30  10  298200  30816  15956 S  0.0  0.9   0:03.12 chrome
    10330 231172    30  10       0      0      0 Z  0.0  0.0   0:00.08 chrome
    10333 231172    30  10       0      0      0 Z  0.0  0.0   0:01.23 chrome
    10391 231172    30  10       0      0      0 Z  0.0  0.0   0:00.10 chrome
    10410 231172    30  10       0      0      0 Z  0.0  0.0   0:10.95 chrome
    10751 231172    30  10  262832  85512  72928 S  0.0  2.4   0:00.08 chrome
    10752 231172    30  10  262832  85608  73024 S  0.0  2.4   0:00.10 chrome
    10785 231172    30  10 20.325g 111908  96224 R  0.0  3.2   0:53.58 chrome
    10833 231172    30  10  298220  95904  81016 S  0.0  2.7   0:00.66 chrome
    16953 231072    20   0    5936    640    576 S  0.0  0.0   0:00.00 sleep
    19123 231072    20   0   19776     88      4 S  0.0  0.0   0:00.05 startup.sh
    19436 231072    20   0   69960     96      0 S  0.0  0.0   0:00.00 sshd
    19441 231072    20   0   19772   1812   1604 S  0.0  0.1   0:05.69 webssh-watc+
    19443 231072    20   0   48868     60      0 S  0.0  0.0   0:00.00 runuser
    19453 232075    20   0 3023696  37284    412 S  0.0  1.1   0:59.86 Kudu.Servic+
    19518 231072    20   0 10.585g  20644     20 S  0.0  0.6   0:00.69 node
    19958 231072    30  10    1044      4      0 S  0.0  0.0   0:00.38 dummy
    20002 231072    30  10    1044      4      0 S  0.0  0.0   0:00.36 dummy
    20196 231172    30  10  262832  22720  10140 S  0.0  0.6   0:00.08 chrome
    20197 231172    30  10  262840  22856  10276 S  0.0  0.7   0:00.42 chrome
    20227 231172    30  10 20.343g  59248  27276 R  0.0  1.7  48:46.61 chrome
    20248 231172    30  10  298248  30704  15868 S  0.0  0.9   0:09.12 chrome
    20389 231172    30  10 20.339g  65200  33196 S  0.0  1.9   0:01.76 chrome
    20527 231072    30  10    1044      4      0 S  0.0  0.0   0:00.63 dummy
    20593 231072    30  10    1044      4      0 S  0.0  0.0   0:00.62 dummy
    22701 231072    30  10    1044      4      0 S  0.0  0.0   0:01.60 dummy
    23708 231172    30  10  380992  35088  19860 S  0.0  1.0  19:53.58 chrome
    23718 231172    30  10  262832  23256  10672 S  0.0  0.7   0:00.07 chrome
    23720 231172    30  10  262832  22716  10136 S  0.0  0.6   0:00.16 chrome
    23803 231172    30  10  298220  31960  17084 S  0.0  0.9   0:01.58 chrome
    23861 231172    30  10       0      0      0 Z  0.0  0.0   0:00.08 chrome
    23862 231172    30  10       0      0      0 Z  0.0  0.0   0:00.71 chrome
    23892 231172    30  10       0      0      0 Z  0.0  0.0   0:04.48 chrome
    25437 231172    30  10       0      0      0 Z  0.0  0.0   0:00.14 chrome
    25442 231172    30  10       0      0      0 Z  0.0  0.0   0:00.11 chrome
    25495 231172    30  10       0      0      0 Z  0.0  0.0   0:00.07 chrome
    25498 231172    30  10       0      0      0 Z  0.0  0.0   0:00.07 chrome
    25500 231172    30  10       0      0      0 Z  0.0  0.0   0:00.07 chrome
    25503 231172    30  10       0      0      0 Z  0.0  0.0   0:00.07 chrome
    26248 231172    30  10  381244  35588  20408 S  0.0  1.0  14:11.32 chrome
    26280 231172    30  10  381196  35860  20508 S  0.0  1.0  14:00.60 chrome
    26283 231172    30  10  262832  23308  10728 S  0.0  0.7   0:00.08 chrome
    26284 231172    30  10  262832  22748  10168 S  0.0  0.6   0:00.16 chrome
    26291 231172    30  10  262832  23308  10728 S  0.0  0.7   0:00.08 chrome
    26295 231172    30  10  262832  22764  10184 S  0.0  0.6   0:00.19 chrome
    26318 231172    30  10 20.332g  50740  28256 S  0.0  1.4  37:18.04 chrome
    26334 231172    30  10  298200  31688  16816 S  0.0  0.9   0:04.20 chrome
    26406 231172    30  10  298244  30708  15864 S  0.0  0.9   0:04.72 chrome
    29391 231172    30  10  380940  35788  20624 S  0.0  1.0   0:01.28 chrome
    29397 231172    30  10  262832  23432  10848 S  0.0  0.7   0:00.08 chrome
    29398 231172    30  10  262832  23412  10832 S  0.0  0.7   0:00.11 chrome
    29429 231172    30  10 20.325g  45712  29704 S  0.0  1.3   0:00.08 chrome
    29450 231172    30  10  298296  32040  17128 S  0.0  0.9   0:00.84 chrome
    29763 231172    30  10  386672  35956  20684 S  0.0  1.0  27:47.29 chrome
    29768 231172    30  10  262832  22732  10148 S  0.0  0.6   0:00.07 chrome
    29769 231172    30  10  262832  23704  11124 S  0.0  0.7   0:00.28 chrome
    29798 231172    30  10 20.338g  55520  27452 R  0.0  1.6  73:23.36 chrome
    29814 231172    30  10  303208  33656  16884 S  0.0  1.0   0:02.39 chrome
    29852 231172    30  10  386796  35936  20516 S  0.0  1.0  28:35.51 chrome
    29855 231172    30  10  262832  23044  10468 S  0.0  0.7   0:00.08 chrome
    29856 231172    30  10  262832  22720  10136 S  0.0  0.6   0:00.21 chrome
    29885 231172    30  10 20.338g  55364  28152 R  0.0  1.6  73:16.82 chrome
    29903 231172    30  10  303196  33548  16772 S  0.0  1.0   0:01.67 chrome
    


  • I can't determine what report is being requested because all report names are listed as 'anonymous'

    Are your templates persisted? I mean does your request looks like

    { "template": { "name": "xxx" } }
    

    If you open the profiler from the startup page, you should see what templates are currently running

    0_1663836344947_upload-e209f77e-1125-4d22-b5d4-7f137789f492

    Note the user you are logged in with needs to have permissions to the templates.

    It seems that the chrome render processes aren't quit after rendering. See this process list:

    By default, every worker keeps a single instance of chrome running, to avoid chrome startup time.
    https://jsreport.net/learn/chrome-pdf#chrome-process-allocation

    Do you apply some special configs?



  • According to the profiler nothing is still running, but the container is still using 100% CPU. There's no special config, apart from license key and admin users and such



  • I have done some load tests and there seems to be some problem in the chrome version used in the 3.7.1. We will release today jsreport 3.8.0 with the updated chrome which should eventually fix this.



  • The 3.8.0 is published, please try it out.



  • Number of processes seems much more normal right now. Also the reports are faster and cpu during processing is lower.
    Is this just an update chrome version?



  • The 3.8.0 contains all kinds of things...
    https://jsreport.net/blog/release-380
    https://github.com/jsreport/jsreport/releases

    The 3.8.0 default image includes updated chrome and the updated alpine base image.
    In our experience chrome in some versions isn't working as it should. Sometimes its performance degrades for example.


Log in to reply
 

Looks like your connection to jsreport forum was lost, please wait while we try to reconnect.