Files updated externally are not consistently detected

holocentric-bmsnext

Hello,

We are generating the templates, data and assets in an application external to the jsreport server. The files are generated directly into the data directory specified in

    "fs-store": {
      "dataDirectory": "data",
      "syncModifications": true,
      "sync": {
        "usePolling": true
      }
    }

When the files are generated, the files are not consistently updated and the template that we want to generate a pdf for is undetected by the jsreport server. When we restart the jsreport, the template is detected. Is there a way to force jsreport to rescan all files in the data-directory? We have tried setting usePolling to false and it has not improved anything.

The version we are using is jsreport 2.3.0

Thanks,

Thomas

jan_blaha

Hm, it works for me.

You have it like this in config, right?

 "extensions": {
    "fs-store": {
      "syncModifications": true
    }
}

What OS/file system you have? Can you replicate it manually? Like that you just copy one template there, does it get detected?

holocentric-bmsnext

Hi Jan,

I'm running it in dev so I think syncModification is true by default. It works fine with one template.

The issue is that files seem to get partially detected, so when we are create an entire directory of templates, assets and folders it seems to fail to detect the later files that are created. It's non-deterministic so at best all the files are detected and at worst only assets are detected (as they are the first to be created).

EDIT: This is currently being locally tested on my Windows 10 machine using nodejs

jan_blaha

Thank you for the details.
I was able to replicate the issue when cp many entities.

I am afraid the sync feature was designed for a different use case.
Which is editing templates in an external editor like vscode and parallel real-time preview in the studio.
I believe your use case is more about deployment.
I would probably recommend to stop jsreport the first. Then copy entities. Then start jsreport again.

I would love the entities synchronization works perfectly also for your case.
However, there is a technical problem that is not easy to solve.

We are using some techniques to guarantee consistent writes.
This means that if the server or hardware crashes during the write.
We should not corrupt our data but recover to the previous state.
If you copy many entities when the jsreport is running, it can corrupt the database.
Files monitoring is a bit tricky, currently, we are not sure if the edits to the fs come from you our from our writes. We solve it just with a silly treshold here. Which partially filters out your edits by accident.

I know that we could be able to solve these problems at least partially. However, I think it would be some afford and it wouldn't be perfect. Therefore I would rather propose to change your deployment.

What do you think? Is stopping and starting jsreport an issue for you?

holocentric-bmsnext

Hi Jan,

Thanks for the quick and detailed response. Unfortunately, we will be running multiple report generating jobs in parallel so restarting the JsReport server is not really an option. It also sounds like that would be quite an expensive option. Currently our work around is to:

Create an empty template in the folder
Rename the folder to trigger a refresh on the entities
Use the OData API to query for the empty template
If the empty template is found then the entities are all there

This is a bit hacky so it is undesirable. Ideally there would be some API method to trigger the entities refresh without having to rename the parent folder name and another API method to query the JsReport server status for when it is idle (not in the process of updating entities from an external change).

Thanks again

jan_blaha

I see. That is really a complicated workaround.

Could you please elaborate more on your use case, why do you actually need this?
How often do you deploy entities this way?
Do you have multiple servers/instances or are you trying to get at least close to 0 downtime with single server/instance?

holocentric-bmsnext

Hi Jan,

Our use case is that we are generating all the required entities for a pdf on the fly. To reduce the amount of chatter we are generating the entity files and folders ourselves directly into the data folder of jsreport rather than going through the odata API to create entities (assets can contain large images, etc.).

We will be generating these entity files per pdf that we want to generate and the frequency of generation will be client initiated so they can occur frequently and in parallel.

Ideally this will work with a single server and no downtime. We would perhaps have more instances/servers in the future but that would be to alleviate load rather than downtime.

Thanks!

holocentric-bmsnext

Hi Jan,
We are thinking to implement a refresh template method on the jsreport server, so that we can call that via the api. Are you okay with that?
Thank.

jan_blaha

We are thinking to implement a refresh template method on the jsreport server, so that we can call that via the api. Are you okay with that?

Sure, you can replace your server.js with this

const jsreport = require('jsreport')()

if (process.env.JSREPORT_CLI) {
  module.exports = jsreport
} else {
  jsreport.init().then(() => {
    jsreport.express.app.get('/api/reload', (req, res) => {
      jsreport.documentStore.provider._load()
        .then(() => res.send('done'))
        .catch((e) => res.send(e.message))
    })
  }).catch((e) => {
    console.trace(e)
    process.exit(1)
  })
}

Our use case is that we are generating all the required entities for a pdf on the fly. To reduce the amount of chatter we are generating the entity files and folders ourselves directly into the data folder of jsreport rather than going through the odata API to create entities (assets can contain large images, etc.).

You can also consider using import-export extension and upload compressed data through API call.
However if the previous works for you, then great.

holocentric-bmsnext

Hi Jan,

We ended up creating the data in the import zip format and posted it using the import api end point as you suggested. We considering adding the reload api as you mentioned, but we were worried there might be errors when generating reports simultaneously. Also didn't want to have to roll our own docker image.

Thanks again