Series Metadata retrieval is very long even with configuration optimization

General
1

Hi all,

I’m using a docker Orthanc image (osimis/orthanc:23.3.5) and trying to solve performance issues. We are using OHIF Viewer v3 but these issues are not related to the viewer for now).

Here are the benchmarks I’ve made(there’s only 3 studies and 13 series stored in Orthanc) :

The query:
http://localhost:3000/dicom-web/studies/<study_uid>/series/<series_uid>/metadata

1- When using the default Orthanc configuration, it can take up to 7,5 seconds to retrieve a series metadata.

2- To avoid Orthanc to read all series instances to retrieve a series metadata, I’ve enabled the PostgreSQL plugin (v4) and enabled only the index and not the storage. I’ve then set the StudiesMetadata and SeriesMetadata option to MainDicomTags in the DicomWeb setup and add ExtraMainDicomTags for Study, Series and Instance (by the way HouseKeeper is enabled so it reconstruct the DB).

But even with this configuration, it takes up to 4 seconds (with VERBOSE_ENABLED set to true) to retrieve the series metadata which is very long.

As I’ve set VERBOSE_ENABLED to true, I can see in the log that OrthancPlugins accesses to all instances (see below). Did I miss something to improve the performance of Orthanc? Should I set SeriesMetadata to Extrapolate and use SeriesMetadataExtrapolatedTags to avoid Orthanc retrieving informations from instances?

Any help would be really appreciated :wink:
Eric

Logs:
osimis-postgresql-orthanc-1 | I0413 13:54:05.289703 HttpServer.cpp:1262] (http) GET /dicom-web/studies/1.2.826.0.1.3680043.2.406.1.2.3754642702.36140.1668072150.577/series/1.2.826.0.1.3680043.2.406.1.3.3754642702.36140.1668072171.310/metadata
osimis-postgresql-orthanc-1 | I0413 13:54:05.289823 OrthancPlugins.cpp:2443] (plugins) Delegating HTTP request to plugin for URI: /dicom-web/studies/1.2.826.0.1.3680043.2.406.1.2.3754642702.36140.1668072150.577/series/1.2.826.0.1.3680043.2.406.1.3.3754642702.36140.1668072171.310/metadata
osimis-postgresql-orthanc-1 | I0413 13:54:05.289941 OrthancPlugins.cpp:3981] (plugins) Plugin making REST POST call to URI /tools/find (after plugins)
osimis-postgresql-orthanc-1 | I0413 13:54:05.301449 ServerContext.cpp:1445] Number of candidate resources after fast DB filtering on main DICOM tags: 1
osimis-postgresql-orthanc-1 | I0413 13:54:05.308148 ServerContext.cpp:1593] Number of matching resources: 1
osimis-postgresql-orthanc-1 | I0413 13:54:05.308396 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /series/71e92b73-76291ad6-0c4ff86d-015544d2-5fc874ad (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.315534 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /instances/e97309ef-e8a756e1-f6410ef4-dc9eb924-6357906d?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.321383 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /series/71e92b73-76291ad6-0c4ff86d-015544d2-5fc874ad?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.327321 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /studies/20c4ab10-e5efe345-f1c7ee75-30f4c878-9a36b464?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.334622 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /instances/0a7ced74-2b799be8-dfa280cb-43d808a5-3e3d5a4e?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.339044 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /instances/09ebde41-c2bb253c-b218e620-127ba880-9c142412?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.346502 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /instances/39b5f065-cd8db184-7a895633-5909ba78-137d5a88?full (built-in API)
osimis-postgresql-orthanc-1 | I0413 13:54:05.352948 OrthancPlugins.cpp:3144] (plugins) Plugin making REST GET call on URI /instances/aef50cac-925fb71e-06e2452c-34325ef2-20e3f579?full (built-in API)

2

Hi Eric,

I’m afraid you’ve already performed all possible optimizations. Orthanc is not reading from the storage. What you see is the plugin calling the core Rest API to get the Main DICOM Tags from DB.

I have performed a quick analysis here and it seems there is room for improvements on our side but I probably won’t have time to look at it before a few weeks.

Stay tuned.

Alain.

2 Likes
3

Hi Alain,

Thanks for your answer and sorry for the delay.
I have the same conclusion as you: Orthanc does not hit the storage any more.

I have also enabled the log statement of PostgreSQL and I’ve seen that for each call of the Rest API, 5 SQL statements are executed excluding BEGIN, COMMIT and SET TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY (see below) :

SELECT resource.internalId, resource.resourceType, parent.publicId
FROM Resources AS resource
LEFT JOIN Resources parent ON parent.internalId=resource.parentId
WHERE resource.publicId='ffb20784-e1195bf3-59efeac5-04e52641-ed324cbf';

SELECT a.publicId
FROM Resources AS a, Resources AS b
WHERE a.parentId = b.internalId AND b.internalId = 3417;

SELECT type, value FROM Metadata WHERE id = 3417;

SELECT uuid, uncompressedSize, compressionType, compressedSize, uncompressedHash, compressedHash, revision
FROM AttachedFiles
WHERE id=3417 AND fileType=1;

SELECT * FROM MainDicomTags WHERE id=3417;

So for a series of 600 instances, 3000 SQL statements are executed!
It seems to be a kind of “N+1 SELECT” issue no? And even if SQL statement are very fast, it’s a huge number of SQL requests for just “only” one series metadata…

I’m not a DICOM expert but are instances metadata required to be returned when asking for a series metadata?

Also those SQL requests seem to be executed in one built-in API, and I don’t know the schema of the database but an improvement could be (if possible) to minimize the number of requests using JOIN?

Thanks again Alain,
If you need help to benchmark your PR which will improve the metadata retrieval performance, just let me know. Have a nice day.
Eric

1 Like
4

Hello,

This is a well-known issue, which has motivated the introduction in Orthanc 1.12.0 of Protocol Buffers as a more flexible way to exchange commands between the Orthanc core and the database engine. The new “labels” feature was the first addition to benefit from this approach.

I’ll investigate SQL optimizations in the coming weeks, as soon as my work as a university professor allows me to.

Kind Regards,
Sébastien-

1 Like