How does the performance differ if you use html recipe? I guess the bottleneck will be the phantomjs html->pdf conversion.
How does chrome-pdf perform for you? The perf is the same?
Please check the perf notes for chrome, the same will apply for phantom
https://jsreport.net/learn/chrome-pdf#performance-optimization