DicomWeb performance seems slow on the metadata query

Hi All

I am using the latest orthannc docker with a postgres back end. The postgres files are stored on a local SSD, postgres itself is running in another docker and everything is running on an Amazon VM with 8 CPUS and 32 GB of memory, with no real other load. I’ve got about 100 studies loaded: the typical study is composed of 4 series, with each series containing roughly 4 to 5 hundred images. The problem I’m seeing is that calls to get metadata to an individual study seem quite slow. For example, I logged into the orthanc container and ran this query:

time curl http://orthanc:8042/dicom-web/studies/1.2.124.113532.129.255.5.220.20100630.75033.7448748/metadata

and the result was

real 0m54.636s
user 0m0.009s
sys 0m0.653s

That seems odd to me and leads me to believe I have done something wrong. I’ve attached my orthanc.json file. Any thoughts/suggestions would be greatly appreciated.

My docker compose looks like this:

version: ‘3.6’
services:

cloudtop:
image: heliumdatastage/renci-guacamole-vnc:use-all-defaults
container_name: cloudtop
volumes:

  • ${ORTHANC_DATA_DIR:-./data}:/usr/local/renci/data:ro
    entrypoint: /init
    ports:
  • 8080:8080

viewer:
image: ohif/viewer:v1.1.14.3708
container_name: viewer
volumes:

  • ${VIEWER_CONFIG:-./config/renci.js}:/usr/share/nginx/html/app-config.js
  • ${NGINX_CONFIG:-./config/nginx-with-proxy.conf}:/etc/nginx/conf.d/default.conf
    environment:
  • MONGO_URL=mongodb://mongo:27017/ohif
  • APP_CONFIG=/usr/share/nginx/html/app-config.js
    depends_on:
  • mongo
  • orthanc
    ports:
  • “3000:80”

orthanc:
image: jodogne/orthanc-plugins:latest
container_name: orthanc
depends_on:

  • postgres
    volumes:
  • ${ORTHANC_CONFIG:-./config/orthanc.json}:/etc/orthanc/orthanc.json:ro
    ports:
  • “8042:8042”
  • “4242:4242”

mongo:
image: mongo:latest
container_name: mongo
ports:

  • “27017:27017”
    volumes:
  • ${MONGO_DATA:-./mongo_data}:/data/db

postgres:
image: postgres:11
container_name: postgres

volumes:

  • ${POSTGRES_DATA_MNT:-/orthadb/postgresql/data/pgdata}:${PGDATA:-/var/lib/postgresql/data}
    environment:
  • POSTGRES_USER=${POSTGRES_USER:-NOTMYREALUSER}
  • POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-NOTMYREALPASSWORD}
  • PGDATA=${PGDATA:-/var/lib/postgresql/data}
  • POSTGRES_DB=orthanc
    ports:
  • “5432:5432”

Thanks
Howard

orthanc.json (20.8 KB)

Yes, this is happening because the metadata needs more information than what is in the orthanc index database. The dicomweb plugin has to fetch each of the dicom files in the study and read out all of the tags. Over a slow filesystem that gets really slow.

Chris

Hi Chris.

Thanks much for the reply. The postgres filesystem is on an SSD. Is there some faster way of going about this? I tried to configure postgres to load everything into memory, but it didn’t seem to help.

Howard

Hi, friends!

I’m experiencing something related to this issue. What I came understand is that the size of the index affects dicom-web’s performance. If we were to plot the total dicom-web request time as measured by the size of the associated index, it would look like a log function: a steep climb right at the start, followed by a plateau going into infinity.

Sure, there could be many more things affecting (I reckon there are plenty). One of them is what Chris pointed out.

Best,
Luiz

Hi Howard,

In my case studies rarely change once they are added. I put a caching proxy in front of orthanc and it did the trick after the first slow access. There’s alot of metadata in a MRI study. I’ve seen 10’s of MB of JSON metadata in some of our studies, which seems impressive. I suppose even with an SSD, reading out all of those files to get that volume of metadata each time is still slow.

At some point in the future, I may need to investigate what metadata is really needed in my case and see if there is a way to convince Orthanc to store those in its index DB and also the dicomweb plugin to only serve those out. I’ve seen some interesting code comments in the dicomweb plugin in this area suggesting that there could be a configuration parameter some day to adjust the cost vs. spec compliance tradeoff of the output.

Cheers,
Chris

Hello,

I second Chris’ answer: Retrieving DICOMweb metadata requires to access all the DICOM files of the study/series of interest on the filesystem, which has a high cost.

However, the DICOMweb calls to “/metadata” have recently been optimized in the mainline of the plugin, so you should give a try to the latest development version of the code (or wait for a new official release), which should already speed up performance:
https://bitbucket.org/sjodogne/orthanc-dicomweb/src/default/

Furthermore, as Chris suggests, we are considering to implement two complementary strategies to improve the performance of calls to “…/metadata”:

  • Provide an option to only retrieve the so-called “main DICOM tags” that are stored within the Orthanc index: This will only use the SQL database (no DICOM file will be accessed), hereby greatly reducing the reading time on the filesystem. Unfortunately, the retrieved information will be basic, and will not be sufficient for some applications.
  • Implement a caching system where the DICOMweb JSON is computed only once (either on store or on first access), and stored as an attachment to the DICOM files. This avoids the cost of parsing large DICOM files multiple times, but the implementation is complex, and cache coherency between multiple versions of Orthanc might be problematic.

At the time of writing, it is not clear whether either of these strategies (or both of them) will finally get implemented. You can follow this development in the function “WriteInstanceMetadata()” within the following file:
https://bitbucket.org/sjodogne/orthanc-dicomweb/src/default/Plugin/WadoRs.cpp

HTH,
Sébastien-

Hi Sébastien

Thanks very much for the response. I’ve downloaded and built the dev version of the dicom-web plugin. I’m running orthanc as a docker, so I thought I could just add my version of the dicom-web plugin to the plugin list and map the plugin into the docker. But I am getting errors like this:

W1007 17:02:17.332577 main.cpp:702] Loading plugin(s) from: /usr/share/orthanc/plugins/renci/libOrthancDicomWeb.so
E1007 17:02:17.344985 PluginsManager.cpp:264] Plugin ‘dicom-web’ already registered

followed eventually by

E1007 17:02:19.308291 main.cpp:1502] Uncaught exception, stopping now: [Error while using a shared library (plugin)] (code 25)

Do I need to rebuild the docker? Any other advice on how to get this to work?

For reference, here’s the relevant part of orthanc.json

“Plugins” : [
“/usr/share/orthanc/plugins”,“/usr/local/share/orthanc/plugins”,“/usr/share/orthanc/plugins/renci/libOrthancDicomWeb.so”
],

And the relevant portion of my docker compose file:

orthanc:
image: jodogne/orthanc-plugins:latest
container_name: orthanc
depends_on:

  • postgres
    volumes:
  • ${ORTHANC_CONFIG:-./config/orthanc.json}:/etc/orthanc/orthanc.json:ro
  • ${ORTHANC_CONFIG:-./localPlugins}:/usr/share/orthanc/plugins/renci:ro
    ports:
  • “8042:8042”
  • “4242:4242”

Thanks again
Howard

Hi all

So I just build the database plugins, added them to my RENCI directory and got rid of the other plugin directories from orthanc.json, so now the docker is only using the plugins I build. It all works and it is faster, so that’s good news.

From reading your previous response, it seems that the metadata call is still reading metadata from files on the disk. I’ll consider moving our files to a faster storage solution. I can’t comment much about the 2 possible strategies you suggest for further speedups, other than to state the obvious, which is that if the “main DICOM tags” are sufficient for the downstream apps (in my case the OHIF viewer) then the first strategy could be a real benefit.

Thanks
Howard

Dear Sebastien,

Thank you for considering optimizing the retrieve speed of the DicomWeb plugin. I was also looking at these speed issue with OHIF team, that does optimization on their side too.

To speed up the data retrieve OHIF has changed the called API, they were retrieving the full study metadata with /study/UID/metadata that could be very long to get an answer due to the huge number of instances in this single call.

They now splitted it into individual /series/UID/metadata call that are faster.

I noticed something strange, the first call they do is to list the series level with /study/UID/series but this call took 18 seconds to generate on my server (Orthanc + mySQL database in docker + HDD storage).
This 18 seconds seems to indicate that the study/UID/series triggers a DICOM file scan.
While when I look to the generated answer, I see only one item that seems to be not directly accessible in the “MainDicomTags” which I supposed are stored in Orthanc database, this tag is 0008,0005 Specific Character Set

If my supposition is correct I wonder if it would be possible to store this additional tag in the database to free the study/UID/series from individual file scanning ?

Best regards,

Salim

Just to give some additional “thought” as I realize my post might not be easy to understood.

The /Study/UID/Metadata is very long to generate beauce of the reading of all instances in a study as it can have thousands of instances (multiple MRI sequences, PET/CT acquisition etc…)

So metadata retrieval at the study level probably should be avoided, and preferentially loaded at the serie level. when the images of a series are loaded this is sufficiantly long operation to make the metadata retrieval without huge impact in the loading time (images are much longuer to retrieve than metadata).

On the other hand, when opening the study it is important to see what series are in it, to be able to select the series you want to pay the price to open.

So the trouble is that the Studies/UID/Series still trigger a scan of all DICOMs instance so this call is still having more or less the same timing cost than the full metadata retrieve at study level.

Maybe i’m missing another way to retrieve the serie’s details (just the main tags) from DicomWeb without triggering all the instances scan ?

Best regards,

Salim

Dear all,

This is an update to this issue. I have just committed the following changeset:
https://bitbucket.org/sjodogne/orthanc-dicomweb/commits/27001924c4564a517f1a4c6dacc3453a629aff04

This changeset introduces two new configuration option, “StudiesMetadata” and “SeriesMetadata” that are documented in the Orthanc Book:

https://book.orthanc-server.com/plugins/dicomweb.html#server-related-options

Briefly speaking, those configuration options allow to choose how Orthanc will compute the “metadata” endpoints: either by reading the full DICOM instances from the storage area (“Full” mode), or by using only the main DICOM tags as stored in the index database (“MainDicomTags” mode).

Besides these configuration options, forthcoming 1.1 release of the DICOMweb also divides by two the number of files that are read from the storage area (the “DICOM-as-JSON” attachments are not used anymore by the “metadata” routes).

As far as DICOMweb viewers are concerned, it is recommended to:

  • Set option “StudiesMetadata” to value “MainDicomTags”.

  • Set option “SeriesMetadata” to value “Full”.

  • First make a request to “/studies/…/metadata” to have an insight about all the instances that are stored in the study. This should be fast, thanks to “StudiesMetadata” being to “MainDicomTags”.

  • Then make a request to “/studies/…/series/…/metadata” when it comes to rendering one given series from the study. This will give accurate results, actually the same as in releases <= 1.0 of the DICOMweb plugin.

HTH,

Sébastien-

Amazing,

Thanks Sebastien !

Hello again,

FYI, I have just implemented a third mode of operation that is named “Extrapolate”, and that is now documented in the Orthanc Book:
https://book.orthanc-server.com/plugins/dicomweb.html#fine-tuning-server-for-wado-rs-retrieve-metadata

If Extrapolate mode is used, the plugin will read up to 3 DICOM instances at random that belong to the study/series of interest. It will then test whether the majority of these instances share the same value for a predefined subset of DICOM tags. If so, this value is added to the metadata response; otherwise, the tag is not reported. In other words, this mode extrapolates the value of some predefined tags by assuming that these tags should be constant across all the instances of the study/series. This mode is a compromise between MainDicomTags (focus on speed) and Full (focus on accuracy).

The related changeset is the following:
https://bitbucket.org/sjodogne/orthanc-dicomweb/commits/c467391b3585eae306fde6d1776a24e9c2173b59

This mode should be useful to people developing DICOMweb viewers, and will in particular be used in the future Stone Web viewer.

Again, this mode will be part of forthcoming 1.1 release of the DICOMweb plugin.

HTH,
Sébastien-