Handle Large no of request

gauravprwl14

Hey,
I am using Js Report and Heroku as deployment env. Heroku config is

When I am trying to call the JSreport Service multiple time in parallel, but in doing so it is crashing every time. I checked I am making 8 calls in parallel but in future I might call 20 calls in parallel. Can u guys please help me out
My JS report configuration

{
  "extensions": {
    "scripts": {
      "timeout": 150000,
      "strategy": "http-server"
    },
    "sample-template": {
      "createSamples": true
    },
    "assets": {
      "allowedFiles": "**/*.*"
    }
  },
  "httpPort": 5488,
  "httpsPort": null,
  "store": {
    "provider": "fs"
  },
  "blobStorage": {
    "provider": "fs"
  },
  "logger": {
    "console": {
      "transport": "console",
      "level": "debug"
    },
    "file": {
      "transport": "file",
      "level": "info",
      "filename": "logs/reporter.log"
    },
    "error": {
      "transport": "file",
      "level": "error",
      "filename": "logs/error.log"
    }
  },
  "allowLocalFilesAccess": true,
  "templatingEngines": {
    "timeout": 100000,
    "numberOfWorkers": 12,
    "strategy": "http-server",
    "templateCache": {
      "max": 100,
      "enabled": true
    }
  },
  "chrome": {
    "timeout": 400000,
    "strategy": "chrome-pool",
    "numberOfWorkers": 12,
    "launchOptions": {
      "args": [
        "--no-sandbox"
      ]
    }
  }
}

And I am getting below error


May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.387Z - debug: Base url not specified, skipping its injection.
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.403Z - debug: Replaced assets ["Helvetica.otf"]
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.405Z - debug: Rendering engine handlebars
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.414Z - debug: Replaced assets ["Helvetica.otf"]
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.417Z - debug: Rendering engine handlebars
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.429Z - debug: Replaced assets ["Helvetica.otf"]
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.430Z - debug: Rendering engine handlebars
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.565Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.570Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.699Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.708Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.711Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.724Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.794Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.796Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.797Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.798Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.798Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.800Z - debug: Compiled template not found in the cache, compiling
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.803Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.806Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.807Z - debug: Executing recipe chrome-pdf
May 25 04:01:10 airops-reporting-staging app/web.1 2020-05-25T11:01:10.809Z - debug: Executing recipe chrome-pdf
May 25 04:01:11 airops-reporting-staging app/web.1 2020-05-25T11:01:11.202Z - debug: Converting with chrome HeadlessChrome/79.0.3945.0 using chrome-pool strategy
May 25 04:01:11 airops-reporting-staging app/web.1 2020-05-25T11:01:11.430Z - debug: Converting with chrome HeadlessChrome/79.0.3945.0 using chrome-pool strategy
May 25 04:01:11 airops-reporting-staging app/web.1 2020-05-25T11:01:11.486Z - debug: Converting with chrome HeadlessChrome/79.0.3945.0 using chrome-pool strategy
May 25 04:01:14 airops-reporting-staging app/web.1 #1 0x55e7300d3ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:14 airops-reporting-staging app/web.1 #3 0x55e73014225f base::SimpleThread::Start()
May 25 04:01:14 airops-reporting-staging app/web.1 #4 0x55e72e910a72 content::SandboxHostLinux::Init()
May 25 04:01:14 airops-reporting-staging app/web.1 #8 0x55e72fd5a9dd headless::(anonymous namespace)::RunContentMain()
May 25 04:01:14 airops-reporting-staging app/web.1 #9 0x55e72fd5a73b headless::HeadlessShellMain()
May 25 04:01:14 airops-reporting-staging app/web.1 #12 0x55e72da1f0aa _start
May 25 04:01:14 airops-reporting-staging app/web.1 #0 0x55e73016fe29 base::debug::CollectStackTrace()
May 25 04:01:14 airops-reporting-staging app/web.1 #1 0x55e7300d3ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:14 airops-reporting-staging app/web.1 #2 0x55e73016f9b1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
May 25 04:01:14 airops-reporting-staging app/web.1 #3 0x7f1398c17890 <unknown>
May 25 04:01:14 airops-reporting-staging app/web.1 #6 0x55e73016e815 base::debug::BreakDebugger()
May 25 04:01:14 airops-reporting-staging app/web.1 #11 0x55e72fd5b76c service_manager::Main()
May 25 04:01:14 airops-reporting-staging app/web.1 #16 0x7f1392ba6b97 __libc_start_main
May 25 04:01:14 airops-reporting-staging app/web.1 #17 0x55e72da1f0aa _start
May 25 04:01:14 airops-reporting-staging app/web.1   r8: 0000000000000000  r9: 00007ffe148d3760 r10: 0000000000000008 r11: 0000000000000246
May 25 04:01:14 airops-reporting-staging app/web.1  r12: 00007ffe148d4a20 r13: 00007ffe148d3a08 r14: 00007ffe148d4a28 r15: 00007ffe148d3a20
May 25 04:01:14 airops-reporting-staging app/web.1  trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
May 25 04:01:14 airops-reporting-staging app/web.1 
May 25 04:01:14 airops-reporting-staging app/web.1 
May 25 04:01:14 airops-reporting-staging app/web.1 #1 0x55e7300d3ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:14 airops-reporting-staging app/web.1 #2 0x55e7300e86b3 logging::LogMessage::~LogMessage()
May 25 04:01:14 airops-reporting-staging app/web.1 #4 0x55e72e910a72 content::SandboxHostLinux::Init()
May 25 04:01:14 airops-reporting-staging app/web.1 #7 0x55e72fd0d541 content::ContentMain()
May 25 04:01:14 airops-reporting-staging app/web.1 #9 0x55e72fd5a73b headless::HeadlessShellMain()
May 25 04:01:14 airops-reporting-staging app/web.1 #10 0x55e72da1f218 ChromeMain
May 25 04:01:14 airops-reporting-staging app/web.1 #2 0x55e73016f9b1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
May 25 04:01:14 airops-reporting-staging app/web.1 #3 0x7f1398c17890 <unknown>
May 25 04:01:14 airops-reporting-staging app/web.1 #4 0x7f1392bc3e97 gsignal
May 25 04:01:14 airops-reporting-staging app/web.1 #5 0x7f1392bc5801 abort
May 25 04:01:14 airops-reporting-staging app/web.1 #6 0x55e73016e815 base::debug::BreakDebugger()
May 25 04:01:14 airops-reporting-staging app/web.1 #7 0x55e7300e8af9 logging::LogMessage::~LogMessage()
May 25 04:01:14 airops-reporting-staging app/web.1 #8 0x55e73014225f base::SimpleThread::Start()
May 25 04:01:14 airops-reporting-staging app/web.1 #9 0x55e72e910a72 content::SandboxHostLinux::Init()
May 25 04:01:14 airops-reporting-staging app/web.1 #10 0x55e72fd0eeda content::ContentMainRunnerImpl::Initialize()
May 25 04:01:14 airops-reporting-staging app/web.1   r8: 0000000000000000  r9: 00007ffe148d3760 r10: 0000000000000008 r11: 0000000000000246
May 25 04:01:14 airops-reporting-staging app/web.1  r12: 00007ffe148d4a20 r13: 00007ffe148d3a08 r14: 00007ffe148d4a28 r15: 00007ffe148d3a20
May 25 04:01:14 airops-reporting-staging app/web.1  trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
May 25 04:01:14 airops-reporting-staging app/web.1 [end of stack trace]
May 25 04:01:14 airops-reporting-staging app/web.1 Calling _exit(1). Core file will not be generated.
May 25 04:01:14 airops-reporting-staging app/web.1 
May 25 04:01:14 airops-reporting-staging app/web.1     at onClose (/app/node_modules/puppeteer/lib/Launcher.js:348:14)
May 25 04:01:14 airops-reporting-staging app/web.1     at Interface.close (readline.js:409:8)
May 25 04:01:16 airops-reporting-staging app/web.1 [0525/110111.920625:ERROR:platform_thread_posix.cc(129)] pthread_create: Resource temporarily unavailable (11)
May 25 04:01:16 airops-reporting-staging app/web.1 #1 0x55ec46576ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:16 airops-reporting-staging app/web.1 #3 0x55ec465d5dd4 base::internal::ThreadPoolImpl::Start()
May 25 04:01:16 airops-reporting-staging app/web.1 #4 0x55ec44e5770c content::StartBrowserThreadPool()
May 25 04:01:16 airops-reporting-staging app/web.1 #5 0x55ec461b232c content::ContentMainRunnerImpl::RunServiceManager()
May 25 04:01:16 airops-reporting-staging app/web.1 #6 0x55ec461b2236 content::ContentMainRunnerImpl::Run()
May 25 04:01:16 airops-reporting-staging app/web.1 #8 0x55ec461b0541 content::ContentMain()
May 25 04:01:16 airops-reporting-staging app/web.1 
May 25 04:01:16 airops-reporting-staging app/web.1 Received signal 6
May 25 04:01:16 airops-reporting-staging app/web.1 #0 0x55ec46612e29 base::debug::CollectStackTrace()
May 25 04:01:16 airops-reporting-staging app/web.1 #1 0x55ec46576ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:16 airops-reporting-staging app/web.1 #2 0x55ec466129b1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
May 25 04:01:16 airops-reporting-staging app/web.1 #3 0x7f2084cf5890 <unknown>
May 25 04:01:16 airops-reporting-staging app/web.1 #4 0x7f207eca1e97 gsignal
May 25 04:01:16 airops-reporting-staging app/web.1 #5 0x7f207eca3801 abort
May 25 04:01:16 airops-reporting-staging app/web.1 #6 0x55ec46611815 base::debug::BreakDebugger()
May 25 04:01:16 airops-reporting-staging app/web.1 #7 0x55ec4658baf9 logging::LogMessage::~LogMessage()
May 25 04:01:16 airops-reporting-staging app/web.1 #8 0x55ec465d5dd4 base::internal::ThreadPoolImpl::Start()
May 25 04:01:16 airops-reporting-staging app/web.1 #9 0x55ec44e5770c content::StartBrowserThreadPool()
May 25 04:01:16 airops-reporting-staging app/web.1 #10 0x55ec461b232c content::ContentMainRunnerImpl::RunServiceManager()
May 25 04:01:16 airops-reporting-staging app/web.1 #11 0x55ec461b2236 content::ContentMainRunnerImpl::Run()
May 25 04:01:16 airops-reporting-staging app/web.1 #12 0x55ec461feacd service_manager::Main()
May 25 04:01:16 airops-reporting-staging app/web.1 #13 0x55ec461b0541 content::ContentMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #14 0x55ec461fd9dd headless::(anonymous namespace)::RunContentMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #15 0x55ec461fd73b headless::HeadlessShellMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #16 0x55ec43ec2218 ChromeMain
May 25 04:01:16 airops-reporting-staging app/web.1 #17 0x7f207ec84b97 __libc_start_main
May 25 04:01:16 airops-reporting-staging app/web.1 #18 0x55ec43ec20aa _start
May 25 04:01:16 airops-reporting-staging app/web.1   r8: 0000000000000000  r9: 00007ffd29e712e0 r10: 0000000000000008 r11: 0000000000000246
May 25 04:01:16 airops-reporting-staging app/web.1  r12: 00007ffd29e72608 r13: 00007ffd29e71588 r14: 00007ffd29e72610 r15: 00007ffd29e715a0
May 25 04:01:16 airops-reporting-staging app/web.1   di: 0000000000000002  si: 00007ffd29e712e0  bp: 00007ffd29e71530  bx: 00007ffd29e71588
May 25 04:01:16 airops-reporting-staging app/web.1   dx: 0000000000000000  ax: 0000000000000000  cx: 00007f207eca1e97  sp: 00007ffd29e712e0
May 25 04:01:16 airops-reporting-staging app/web.1   ip: 00007f207eca1e97 efl: 0000000000000246 cgf: 0000000000000033 erf: 0000000000000000
May 25 04:01:16 airops-reporting-staging app/web.1  trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
May 25 04:01:16 airops-reporting-staging app/web.1 [end of stack trace]
May 25 04:01:16 airops-reporting-staging app/web.1 Calling _exit(1). Core file will not be generated.
May 25 04:01:16 airops-reporting-staging app/web.1 
May 25 04:01:16 airops-reporting-staging app/web.1 
May 25 04:01:16 airops-reporting-staging app/web.1 TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
May 25 04:01:16 airops-reporting-staging app/web.1  Error: Failed to launch chrome!
May 25 04:01:16 airops-reporting-staging app/web.1 [0525/110111.920625:ERROR:platform_thread_posix.cc(129)] pthread_create: Resource temporarily unavailable (11)
May 25 04:01:16 airops-reporting-staging app/web.1 [0525/110111.921357:FATAL:thread_pool_impl.cc(152)] Check failed: service_thread_->StartWithOptions(service_thread_options). 
May 25 04:01:16 airops-reporting-staging app/web.1 #0 0x55ec46612e29 base::debug::CollectStackTrace()
May 25 04:01:16 airops-reporting-staging app/web.1 #1 0x55ec46576ee3 base::debug::StackTrace::StackTrace()
May 25 04:01:16 airops-reporting-staging app/web.1 #2 0x55ec4658b6b3 logging::LogMessage::~LogMessage()
May 25 04:01:16 airops-reporting-staging app/web.1 #3 0x55ec465d5dd4 base::internal::ThreadPoolImpl::Start()
May 25 04:01:16 airops-reporting-staging app/web.1 #4 0x55ec44e5770c content::StartBrowserThreadPool()
May 25 04:01:16 airops-reporting-staging app/web.1 #5 0x55ec461b232c content::ContentMainRunnerImpl::RunServiceManager()
May 25 04:01:16 airops-reporting-staging app/web.1 #6 0x55ec461b2236 content::ContentMainRunnerImpl::Run()
May 25 04:01:16 airops-reporting-staging app/web.1 #7 0x55ec461feacd service_manager::Main()
May 25 04:01:16 airops-reporting-staging app/web.1 #8 0x55ec461b0541 content::ContentMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #9 0x55ec461fd9dd headless::(anonymous namespace)::RunContentMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #10 0x55ec461fd73b headless::HeadlessShellMain()
May 25 04:01:16 airops-reporting-staging app/web.1 #11 0x55ec43ec2218 ChromeMain
May 25 04:01:16 airops-reporting-staging app/web.1 #12 0x7f207ec84b97 __libc_start_main
May 25 04:01:16 airops-reporting-staging app/web.1 #13 0x55ec43ec20aa _start```

jan_blaha

Thank you for the question.

Not an expert on this, but it looks like the system used all the available threads, and chrome failed to launch, based on the error message: "pthread_create: Resource temporarily unavailable".

Could you revisit values templatingEngines.numberOfWorkers and chrome.numberOfWorkers in your config file?
I don't know your resource/hw specification like the number of available CPU cores, but these numbers look too big.
What if you use something like 3 for numberOfWorkers?

gauravprwl14

I am running Heroku 1.x dyno with max 512 MB of RAM

jan_blaha

That is very much a micro low-cost machine. I would recommend max of two workers.

gauravprwl14

@jan_blaha I changed the workers to two, but JSreport still crashes due when large no of request is made in parallel.
If I use PM2 service to run jsreport on multiple cores, will it help in handling the large request

jan_blaha

jsreport crashes or chrome crashes? With what error? 512MB doesn't need to be enough even for a single big report.