Scalability of Orthanc and Jobs

Google Groups archive
1

Hello,

We have a composed-orthanc stack like:

  • one master instance (handle all write requests and jobs)
  • 1 to 5 slave instances (handle all read requests, especially web viewers)

For now, all job requests are handled by our master instance.
As described here, since v1.9.2 multiple writes are accepted and we attempted to update our Orthanc stack following this guideline but Orthanc Jobs are not handled correctly.

We use a gateway that balances requests over two orthanc instances connected to the same POSTGRES database. If job creation request is handled by the first instance and get list of jobs request is handled by the second instance the result is not the same (return a 404 http status code) because the instances don’t share the same database jobs registry.

Is this a known issue that will be fixed in the next versions?

Our configuration details:

  • Orthanc launched without NO_JOBS parameter

  • SaveJobs parameter has TRUE value in both instances

  • “ConcurrentJobs” : 50, LimitJobs" : 100, SaveJobs" : true, JobsHistorySize" : 25 in both instances

  • Nothing strange reported in orthanc console

  • POSTGRES server version is 13.2-1.pgdg100+1

01 instance logs:

Startup command: Orthanc /tmp/orthanc.json
W0728 11:17:05.244491 main.cpp:1942] Orthanc version: 1.9.3
W0728 11:17:05.244707 OrthancConfiguration.cpp:66] Reading the configuration from: “/tmp/orthanc.json”
W0728 11:17:05.262497 FromDcmtkBridge.cpp:339] Loading external DICOM dictionary: “/usr/share/libdcmtk14/dicom.dic”
W0728 11:17:05.271166 FromDcmtkBridge.cpp:339] Loading external DICOM dictionary: “/usr/share/libdcmtk14/private.dic”
W0728 11:17:05.284875 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins/
W0728 11:17:05.285504 PluginsManager.cpp:269] Registering plugin ‘authorization’ (version 0.2.4)
W0728 11:17:05.285535 PluginsManager.cpp:168] Initializing the authorization plugin
W0728 11:17:05.295762 PluginsManager.cpp:269] Registering plugin ‘gdcm’ (version 1.2)
W0728 11:17:05.296134 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.90
W0728 11:17:05.296153 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.91
W0728 11:17:05.296159 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.92
W0728 11:17:05.296166 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.93
W0728 11:17:05.296177 PluginsManager.cpp:168] Throttling GDCM to 4 concurrent thread(s)
W0728 11:17:05.296217 PluginsManager.cpp:168] Version of GDCM: 3.0.8
W0728 11:17:05.298198 PluginsManager.cpp:269] Registering plugin ‘python’ (version 3.2)
W0728 11:17:05.298231 PluginsManager.cpp:168] Python plugin is initializing
W0728 11:17:05.298429 PluginsManager.cpp:168] Using Python script “kafka-events-handler.py” from directory: /etc/orthanc/python-scripts
W0728 11:17:05.298453 PluginsManager.cpp:168] Force global loading of Python shared library: /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0
W0728 11:17:05.298549 PluginsManager.cpp:168] Program name: /usr/local/bin/Orthanc
PYTHON-EVENT-HANDLER :: Python event handler script loaded successfully!
W0728 11:17:05.406410 PluginsManager.cpp:269] Registering plugin ‘postgresql-storage’ (version 4.0)
W0728 11:17:05.406886 PluginsManager.cpp:168] The PostgreSQL storage area is currently disabled, set “EnableStorage” to “true” in the “PostgreSQL” section of the configuration file of Orthanc
W0728 11:17:05.407641 PluginsManager.cpp:269] Registering plugin ‘postgresql-index’ (version 4.0)
W0728 11:17:05.408019 PluginsManager.cpp:168] The index plugin will use 5 connection(s) to the database, and will retry up to 5 time(s) in the case of a collision
W0728 11:17:05.408696 PluginsManager.cpp:269] Registering plugin ‘dicom-web’ (version 1.6)
W0728 11:17:05.411610 PluginsManager.cpp:168] URI to the DICOMweb REST API: /dicom-web/
W0728 11:17:05.411943 PluginsManager.cpp:168] URI to the WADO-URI API: /wado
W0728 11:17:05.411979 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins-disabled/libOsimisWebViewer.so
W0728 11:17:05.425213 PluginsManager.cpp:269] Registering plugin ‘osimis-web-viewer’ (version 1.4.2.0-9d9eff4)
W0728 11:17:05.425267 PluginsManager.cpp:168] Initializing the Web viewer
W0728 11:17:05.425733 PluginsManager.cpp:168] Using GDCM instead of the DICOM decoder that is built in Orthanc
W0728 11:17:05.425822 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins-disabled/libOrthancAzureBlobStorage.so
W0728 11:17:05.430538 PluginsManager.cpp:269] Registering plugin ‘Azure Blob Storage’ (version 1.1.0)
W0728 11:17:05.431112 PluginsManager.cpp:168] Azure Blob Storage plugin is initializing
W0728 11:17:05.457161 PluginsManager.cpp:168] Azure Blob Storage: client-side encryption is disabled
W0728 11:17:05.457185 OrthancPlugins.cpp:495] Performance warning: The storage area plugin doesn’t implement reading of file ranges
W0728 11:17:05.457205 main.cpp:1610] Using a custom database from plugins
W0728 11:17:05.457212 main.cpp:1621] Using a custom storage area from plugins
W0728 11:17:05.482579 HttpClient.cpp:1176] HTTPS will use the CA certificates from this file: /etc/ssl/certs/ca-certificates.crt
W0728 11:17:05.483029 LuaContext.cpp:93] Lua says: Lua toolbox installed
W0728 11:17:05.483356 LuaContext.cpp:93] Lua says: Lua toolbox installed
W0728 11:17:05.484099 ServerContext.cpp:478] Disk compression is disabled
W0728 11:17:05.484118 ServerIndex.cpp:391] No limit on the number of stored patients
W0728 11:17:05.484123 ServerIndex.cpp:411] No limit on the size of the storage area
W0728 11:17:05.485682 ServerContext.cpp:220] Reloading the jobs from the last execution of Orthanc
W0728 11:17:05.487183 JobsEngine.cpp:271] The jobs engine has started with 50 threads
W0728 11:17:05.487274 main.cpp:298] Security risk in DICOM SCP: C-FIND requests are always allowed, even from unknown modalities
W0728 11:17:05.487285 main.cpp:303] Security risk in DICOM SCP: C-GET requests are always allowed, even from unknown modalities
W0728 11:17:05.487653 main.cpp:1249] DICOM server listening with AET ORTHANC on port: 4242
W0728 11:17:05.487687 HttpServer.cpp:1992] HTTP compression is enabled
W0728 11:17:05.487697 main.cpp:1000] ====> Remote access is enabled while user authentication is explicitly disabled, your setup is POSSIBLY INSECURE <====
W0728 11:17:05.487707 main.cpp:1124] Remote LUA script execution is disabled
W0728 11:17:05.494258 HttpServer.cpp:1769] HTTP server listening on port: 8042 (HTTPS encryption is disabled, remote access is allowed)
W0728 11:17:05.494293 main.cpp:876] Orthanc has started

02 instance logs:

Startup command: Orthanc /tmp/orthanc.json
W0728 11:17:06.247574 main.cpp:1942] Orthanc version: 1.9.3
W0728 11:17:06.247784 OrthancConfiguration.cpp:66] Reading the configuration from: “/tmp/orthanc.json”
W0728 11:17:06.264819 FromDcmtkBridge.cpp:339] Loading external DICOM dictionary: “/usr/share/libdcmtk14/dicom.dic”
W0728 11:17:06.273027 FromDcmtkBridge.cpp:339] Loading external DICOM dictionary: “/usr/share/libdcmtk14/private.dic”
W0728 11:17:06.286291 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins/
W0728 11:17:06.286920 PluginsManager.cpp:269] Registering plugin ‘authorization’ (version 0.2.4)
W0728 11:17:06.286949 PluginsManager.cpp:168] Initializing the authorization plugin
W0728 11:17:06.297193 PluginsManager.cpp:269] Registering plugin ‘gdcm’ (version 1.2)
W0728 11:17:06.297491 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.90
W0728 11:17:06.297508 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.91
W0728 11:17:06.297513 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.92
W0728 11:17:06.297518 PluginsManager.cpp:168] Orthanc will use GDCM to decode transfer syntax: 1.2.840.10008.1.2.4.93
W0728 11:17:06.297536 PluginsManager.cpp:168] Throttling GDCM to 4 concurrent thread(s)
W0728 11:17:06.297571 PluginsManager.cpp:168] Version of GDCM: 3.0.8
W0728 11:17:06.298657 PluginsManager.cpp:269] Registering plugin ‘postgresql-storage’ (version 4.0)
W0728 11:17:06.298945 PluginsManager.cpp:168] The PostgreSQL storage area is currently disabled, set “EnableStorage” to “true” in the “PostgreSQL” section of the configuration file of Orthanc
W0728 11:17:06.299750 PluginsManager.cpp:269] Registering plugin ‘postgresql-index’ (version 4.0)
W0728 11:17:06.300059 PluginsManager.cpp:168] The index plugin will use 5 connection(s) to the database, and will retry up to 5 time(s) in the case of a collision
W0728 11:17:06.300698 PluginsManager.cpp:269] Registering plugin ‘dicom-web’ (version 1.6)
W0728 11:17:06.300980 PluginsManager.cpp:168] URI to the DICOMweb REST API: /dicom-web/
W0728 11:17:06.301301 PluginsManager.cpp:168] URI to the WADO-URI API: /wado
W0728 11:17:06.301338 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins-disabled/libOsimisWebViewer.so
W0728 11:17:06.313066 PluginsManager.cpp:269] Registering plugin ‘osimis-web-viewer’ (version 1.4.2.0-9d9eff4)
W0728 11:17:06.313113 PluginsManager.cpp:168] Initializing the Web viewer
W0728 11:17:06.313479 FromDcmtkBridge.cpp:409] Warning: You are registering a private tag (7331,1000), but no private creator was associated with it
W0728 11:17:06.314166 PluginsManager.cpp:168] Using GDCM instead of the DICOM decoder that is built in Orthanc
W0728 11:17:06.314274 main.cpp:864] Loading plugin(s) from: /usr/share/orthanc/plugins-disabled/libOrthancAzureBlobStorage.so
W0728 11:17:06.318669 PluginsManager.cpp:269] Registering plugin ‘Azure Blob Storage’ (version 1.1.0)
W0728 11:17:06.318883 PluginsManager.cpp:168] Azure Blob Storage plugin is initializing
W0728 11:17:06.336023 PluginsManager.cpp:168] Azure Blob Storage: client-side encryption is disabled
W0728 11:17:06.336050 OrthancPlugins.cpp:495] Performance warning: The storage area plugin doesn’t implement reading of file ranges
W0728 11:17:06.336068 main.cpp:1610] Using a custom database from plugins
W0728 11:17:06.336075 main.cpp:1621] Using a custom storage area from plugins
W0728 11:17:06.371146 HttpClient.cpp:1176] HTTPS will use the CA certificates from this file: /etc/ssl/certs/ca-certificates.crt
W0728 11:17:06.371758 LuaContext.cpp:93] Lua says: Lua toolbox installed
W0728 11:17:06.371982 LuaContext.cpp:93] Lua says: Lua toolbox installed
W0728 11:17:06.372168 ServerContext.cpp:478] Disk compression is disabled
W0728 11:17:06.372481 ServerIndex.cpp:391] No limit on the number of stored patients
W0728 11:17:06.372494 ServerIndex.cpp:411] No limit on the size of the storage area
W0728 11:17:06.372510 main.cpp:1314] Orthanc Explorer UI is disabled
W0728 11:17:06.374780 ServerContext.cpp:220] Reloading the jobs from the last execution of Orthanc
W0728 11:17:06.377712 JobsEngine.cpp:271] The jobs engine has started with 50 threads
W0728 11:17:06.378161 main.cpp:1249] DICOM server listening with AET ORTHANC on port: 4242
W0728 11:17:06.378201 HttpServer.cpp:1992] HTTP compression is enabled
W0728 11:17:06.378213 main.cpp:1000] ====> Remote access is enabled while user authentication is explicitly disabled, your setup is POSSIBLY INSECURE <====
W0728 11:17:06.378222 main.cpp:1124] Remote LUA script execution is disabled
W0728 11:17:06.386603 HttpServer.cpp:1769] HTTP server listening on port: 8042 (HTTPS encryption is disabled, remote access is allowed)
W0728 11:17:06.386644 main.cpp:876] Orthanc has started

Thanks!

2

Hi Alexandru,

This is indeed a limitation of the current architecture. Each Orthanc instance saves its jobs in the DB with a custom identifier. When you try to read the jobs from an Orthanc, it returns only its own jobs.

Also note that the jobs status are mainly maintained in memory and saved to DB every 10 seconds. Therefore, modifying Orthanc and the DB plugins to return all jobs from all Orthanc is not an “easy task” and would require a bunch of changes (first serialize the jobs to DB after every state change + add a route + plugin primitive to return all jobs). If you’re interested by such a development, please get in touch with Osimis or another developer to implement this feature (since it is useful mainly for commercial companies scaling Orthanc, we consider that this feature shall be funded by the industry).

However, there are a few workarounds:

  • configure “sticky load balancing” such that a given HTTP client will always talk to the same Orthanc slave. It might be difficult to implement !
  • add your own route “/all-orthanc-jobs” that would query all Orthanc and concatenate the responses into a single one.

HTH

Alain

3

Hello Alain,

Thanks for your answer.
I think that specifying this limitation on the scalability page can be nice for new users.
For now we use a gateway to route all /jobs/* requests to a single WRITE orthanc instance but I would like to take advantage of orthanc scalability and make the distribution of the requests in round-robin fashion for example.

Regarding the implementation (as a summary), I think that a global jobs registry can be kept in the database and each orthanc instance can implement a “pooling” mechanism to fetch pending/update running jobs (using table locking/transaction serialization in order to avoid collisions between multiple instance transactions).
I would have been interested in development but to be honest I’m not a C lover :frowning: sorry …

Thanks for your time, have a nice day !

4

I’ve updated the doc accordingly: https://hg.orthanc-server.com/orthanc-book/rev/32be8356e55a