I am using the latest orthannc docker with a postgres back end. The postgres files are stored on a local SSD, postgres itself is running in another docker and everything is running on an Amazon VM with 8 CPUS and 32 GB of memory, with no real other load. I’ve got about 100 studies loaded: the typical study is composed of 4 series, with each series containing roughly 4 to 5 hundred images. The problem I’m seeing is that calls to get metadata to an individual study seem quite slow. For example, I logged into the orthanc container and ran this query:
That seems odd to me and leads me to believe I have done something wrong. I’ve attached my orthanc.json file. Any thoughts/suggestions would be greatly appreciated.
Yes, this is happening because the metadata needs more information than what is in the orthanc index database. The dicomweb plugin has to fetch each of the dicom files in the study and read out all of the tags. Over a slow filesystem that gets really slow.
Thanks much for the reply. The postgres filesystem is on an SSD. Is there some faster way of going about this? I tried to configure postgres to load everything into memory, but it didn’t seem to help.
I’m experiencing something related to this issue. What I came understand is that the size of the index affects dicom-web’s performance. If we were to plot the total dicom-web request time as measured by the size of the associated index, it would look like a log function: a steep climb right at the start, followed by a plateau going into infinity.
Sure, there could be many more things affecting (I reckon there are plenty). One of them is what Chris pointed out.
In my case studies rarely change once they are added. I put a caching proxy in front of orthanc and it did the trick after the first slow access. There’s alot of metadata in a MRI study. I’ve seen 10’s of MB of JSON metadata in some of our studies, which seems impressive. I suppose even with an SSD, reading out all of those files to get that volume of metadata each time is still slow.
At some point in the future, I may need to investigate what metadata is really needed in my case and see if there is a way to convince Orthanc to store those in its index DB and also the dicomweb plugin to only serve those out. I’ve seen some interesting code comments in the dicomweb plugin in this area suggesting that there could be a configuration parameter some day to adjust the cost vs. spec compliance tradeoff of the output.
I second Chris’ answer: Retrieving DICOMweb metadata requires to access all the DICOM files of the study/series of interest on the filesystem, which has a high cost.
However, the DICOMweb calls to “/metadata” have recently been optimized in the mainline of the plugin, so you should give a try to the latest development version of the code (or wait for a new official release), which should already speed up performance: https://bitbucket.org/sjodogne/orthanc-dicomweb/src/default/
Furthermore, as Chris suggests, we are considering to implement two complementary strategies to improve the performance of calls to “…/metadata”:
Provide an option to only retrieve the so-called “main DICOM tags” that are stored within the Orthanc index: This will only use the SQL database (no DICOM file will be accessed), hereby greatly reducing the reading time on the filesystem. Unfortunately, the retrieved information will be basic, and will not be sufficient for some applications.
Implement a caching system where the DICOMweb JSON is computed only once (either on store or on first access), and stored as an attachment to the DICOM files. This avoids the cost of parsing large DICOM files multiple times, but the implementation is complex, and cache coherency between multiple versions of Orthanc might be problematic.
Thanks very much for the response. I’ve downloaded and built the dev version of the dicom-web plugin. I’m running orthanc as a docker, so I thought I could just add my version of the dicom-web plugin to the plugin list and map the plugin into the docker. But I am getting errors like this:
So I just build the database plugins, added them to my RENCI directory and got rid of the other plugin directories from orthanc.json, so now the docker is only using the plugins I build. It all works and it is faster, so that’s good news.
From reading your previous response, it seems that the metadata call is still reading metadata from files on the disk. I’ll consider moving our files to a faster storage solution. I can’t comment much about the 2 possible strategies you suggest for further speedups, other than to state the obvious, which is that if the “main DICOM tags” are sufficient for the downstream apps (in my case the OHIF viewer) then the first strategy could be a real benefit.
Thank you for considering optimizing the retrieve speed of the DicomWeb plugin. I was also looking at these speed issue with OHIF team, that does optimization on their side too.
To speed up the data retrieve OHIF has changed the called API, they were retrieving the full study metadata with /study/UID/metadata that could be very long to get an answer due to the huge number of instances in this single call.
They now splitted it into individual /series/UID/metadata call that are faster.
I noticed something strange, the first call they do is to list the series level with /study/UID/series but this call took 18 seconds to generate on my server (Orthanc + mySQL database in docker + HDD storage).
This 18 seconds seems to indicate that the study/UID/series triggers a DICOM file scan.
While when I look to the generated answer, I see only one item that seems to be not directly accessible in the “MainDicomTags” which I supposed are stored in Orthanc database, this tag is 0008,0005 Specific Character Set
If my supposition is correct I wonder if it would be possible to store this additional tag in the database to free the study/UID/series from individual file scanning ?
Just to give some additional “thought” as I realize my post might not be easy to understood.
The /Study/UID/Metadata is very long to generate beauce of the reading of all instances in a study as it can have thousands of instances (multiple MRI sequences, PET/CT acquisition etc…)
So metadata retrieval at the study level probably should be avoided, and preferentially loaded at the serie level. when the images of a series are loaded this is sufficiantly long operation to make the metadata retrieval without huge impact in the loading time (images are much longuer to retrieve than metadata).
On the other hand, when opening the study it is important to see what series are in it, to be able to select the series you want to pay the price to open.
So the trouble is that the Studies/UID/Series still trigger a scan of all DICOMs instance so this call is still having more or less the same timing cost than the full metadata retrieve at study level.
Maybe i’m missing another way to retrieve the serie’s details (just the main tags) from DicomWeb without triggering all the instances scan ?
Briefly speaking, those configuration options allow to choose how Orthanc will compute the “metadata” endpoints: either by reading the full DICOM instances from the storage area (“Full” mode), or by using only the main DICOM tags as stored in the index database (“MainDicomTags” mode).
Besides these configuration options, forthcoming 1.1 release of the DICOMweb also divides by two the number of files that are read from the storage area (the “DICOM-as-JSON” attachments are not used anymore by the “metadata” routes).
As far as DICOMweb viewers are concerned, it is recommended to:
Set option “StudiesMetadata” to value “MainDicomTags”.
Set option “SeriesMetadata” to value “Full”.
First make a request to “/studies/…/metadata” to have an insight about all the instances that are stored in the study. This should be fast, thanks to “StudiesMetadata” being to “MainDicomTags”.
Then make a request to “/studies/…/series/…/metadata” when it comes to rendering one given series from the study. This will give accurate results, actually the same as in releases <= 1.0 of the DICOMweb plugin.
If Extrapolate mode is used, the plugin will read up to 3 DICOM instances at random that belong to the study/series of interest. It will then test whether the majority of these instances share the same value for a predefined subset of DICOM tags. If so, this value is added to the metadata response; otherwise, the tag is not reported. In other words, this mode extrapolates the value of some predefined tags by assuming that these tags should be constant across all the instances of the study/series. This mode is a compromise between MainDicomTags (focus on speed) and Full (focus on accuracy).