Scheduling component stressing out server storage

Ntinos Kefis

Hello,

so I've managed to implement jsreport's scheduling component and allow users to create their own scheduled reports using API calls. Everything works and schedules are created/deleted normally for some time now.

What I noticed is an increased usage of storage disk on server, upon searching I found out that the 'tasks' file is now quite large (~70MB) and I think jsreport is reading/writing on it on a short periodical time. That causes the disk on server to be constantly occupied at around 2MB/s which is quite a load. I believe this file works as something of an 'instance keeper' for the schedules instances on jsreport.

My question: Is there a safe way to somehow unload this file without compromising the correct function of jsreport's scheduling component?

jan_blaha

Hi,

thank you for the question.

Yes, the tasks file keeps a history of the schedules run and also eventually currently running schedules.

How many times the report was created from your schedules? 70MB is quite a large number, that could mean maybe 100 000 reports.
What do you do after the schedule runs? Are you interested in the history of schedules run?
Maybe we could add a configuration to automatically delete the old ones.

Something that could help you:

This config will change period in which the task file is compacted to 5minutes from the default 15s

{ "extensions": { "fs-store": { "compactionInterval":  300000 } } }

You can also add cleaning of the old tasks into the server.js this way

jsreport.initializeListeners.add('clean tasks', () => {
  setInterval(() => {
    console.log('removing')
    jsreport.documentStore.collection('tasks').remove({
      finishDate: {
        $lt: new Date(new Date().getTime() - 10000)
      }
    }).catch(console.error)
  }, 30000)
})

Ntinos Kefis

Hi,

thanks for your reply and scrpits, I will test and get back with results.

we've given users the ability to create their own scheduled reports and also used the scripting capability for dynamic data gathering and email send. So when a schedule triggers, dynamic data is gathered from external API source and the report is sent after render via email. So, 100 users having 1-2 scheduled reports with their data sent on a daily basis would quickly pile up those numbers.

we've checked schedule history from time to time using the studio for debugging i.e. when a schedule fails to execute for some reason keeping error state. Being able to keep track of a certain # of instances for each schedule (i.e. last 5) via configuration would be helpful I suppose

jan_blaha

Being able to keep track of a certain # of instances for each schedule (i.e. last 5) via configuration would be helpful I suppose

Good idea. We have it now in the backlog - https://github.com/jsreport/jsreport/issues/720

jan_blaha

This is now part of the jsreport 2.10.0