Slow upload speed for large series

Google Groups archive
1

Hi all,
I have discovered a slow down in file loading that is dependent on the size of the dicom series.

I have discovered that load speeds are dependent on series size and get crippling slow at >20000 images in a series.

I tested on Orthanc 1.3 (also observed on 1.1 and 1.2), using postgresql plugin (index=true, storage=false) running on Linux Mint 18.1 (built following Ubuntu build notes and using static build).

I loaded three local studies (via python script, but also observed during push from MRI scanner), each with ~53000 images and watched the storage folder grow - see plot 1 (note doing a simple file count on the storage folder so showing 2x number of images). Obvious rapid loading and a period of very slow loading for each study.
I then repeated with one subject with the large study comprising 21000 images removed (plot 2) almost constant, rapid loading.
I then looked at the number of files in each series and the rate of loading (files per second) (plot 3 - results shown only for series > 100 images) and there is a clear trend of decrease in speed with increase in the number of images per series.

A study with 53528 images (single image dicoms) took 212 minutes to load to a local Orthanc instance. 195 mins of that time (92%) was spent loading a single series of 21600 images (41% of images).

Is this known/understood?

Obviously I would like the high speeds irrelevant of the series size.

Anything that I can modify in my setup to achieve this?

Note - obviously things like file size are to be considered also - but all “large series” presented here (>5000 images) are smaller individual files than other, faster loading, series.

Thanks for your help.
Fraser

2

Hello,

First of all, please make sure you compiled Orthanc and its associated plugins with the “-DCMAKE_BUILD_TYPE=Release” option while invoking CMake:
http://book.orthanc-server.com/faq/troubleshooting.html#performance-issues

HTH,
Sébastien-

3

Thanks - yes have done this. Build commands tested were:

cmake -DALLOW_DOWNLOADS=ON -DUSE_GTEST_DEBIAN_SOURCE_PACKAGE=ON -DUSE_SYSTEM_MONGOOSE=OFF -DDCMTK_LIBRARIES=dcmjpls -DCMAKE_BUILD_TYPE=Release …/Orthanc-$ORTHANCVERS/

and

cmake -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE=Release …/Orthanc-$ORTHANCVERS/

Also:
cmake …/OrthancPostgreSQL-2.0/ -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE=Release

and in Config file:

“PostgreSQL” : {
“EnableIndex” : true,
“EnableStorage”: false,
“ConnectionUri” : “postgresql://localhost/orthanc?user=orthanc&password=orthanc”
},

Cheers,
Fraser

4

As a consequence, you are most probably observing the following issues:
https://bitbucket.org/sjodogne/orthanc/issues/47/index-improvements-for-pg-plugin

https://groups.google.com/d/msg/orthanc-users/iAC-eAW1FSU/zdAh73YRAgAJ

Regards,
Sébastien-

5

I see.
Thanks for the links.

6

Hi as a final update to this topic.
I have used a work around for this issue.
The large (20000+) file series is not needed in our server. It is for research only. So we deal with that separately and don’t import.

I appreciate the software, thanks for everyone’s efforts. I hope the MySQL implementation has not been completely shelved. Sorry we were not aware of it earlier.

Regards,
Fraser

7

Hi,

Thanks for the feedback.

Regarding DB optimizations and MySQL support, be reassured that this is clearly still on our roadmap.

Sébastien-