Orthanc - high memory usage on Windows

Google Groups archive
1

I am using Orthanc 20.1.0 with MySQL to manage a PACS migration. I currently have gathered approximately 4TB of dicom images (compressed) equating to ~9400 studies and have ~7500 more studies to go.
Most of the time, things are going fairly smooth, but I get failures on the source PACS where Orthanc refuses the dicoms or something doesn’t happen right between the two systems.
Most of the time, because I’m compressing the images, I sit around 20-30% CPU usage and around 2GB of memory.
There are times when Orthanc doesn’t want to handle the incoming dicom and the memory usage will spike to 15-16GB. Sometimes in fact, halting server operation and I have to find someone to physically reboot the server (I am off-site).

The source PACS is sending up to 5 studies at a time, but if Orthanc wants to refuse the dicom, it will even spike to 15-16GB memory use even when I manually send a single dicom file from the source PACS upon finding the discrepancy.

Sometimes I can go into the study at the source and re-send the individual dicoms, and yet other times, Orthanc just won’t process certain dicoms.

All transfer syntaxes are set to true as well as unknown SOP class.

// The transfer syntaxes that are accepted by Orthanc C-Store SCP
“DeflatedTransferSyntaxAccepted” : true,
“JpegTransferSyntaxAccepted” : true,
“Jpeg2000TransferSyntaxAccepted” : true,
“JpegLosslessTransferSyntaxAccepted” : true,
“JpipTransferSyntaxAccepted” : true,
“Mpeg2TransferSyntaxAccepted” : true,
“RleTransferSyntaxAccepted” : true,

// Whether Orthanc accepts to act as C-Store SCP for unknown storage
// SOP classes (aka. “promiscuous mode”)
“UnknownSopClassAccepted” : true,

// Set the timeout (in seconds) after which the DICOM associations
// are closed by the Orthanc SCP (server) if no further DIMSE
// command is received from the SCU (client).
“DicomScpTimeout” : 60,

I have to compress the data otherwise I wouldn’t have room for it on this server.

Do I need to add more transfer syntaxes such as littleendian/bigendian in the config?

2

E0310 14:09:59.599914 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU
E0310 14:10:45.510794 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU
E0310 14:11:01.001622 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU
E0310 14:14:26.235582 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU
E0310 14:14:26.625583 StoreScp.cpp:304] Store SCP Failed: Peer aborted Association (or never connected)
E0310 14:15:45.343321 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU
E0310 14:16:23.064187 StoreScp.cpp:304] Store SCP Failed: DIMSE Failed to send message
0006:031d TCP I/O Error (An established connection was aborted by the software in your host machine.) occurred in routine: writeDataPDU

3

Hi Steve,

Any chance you could isolate one of these “problematic” DICOM files or are these “random” files ?

From the error logs, I would say Orthanc tries to send an “acknowledge” for the C-Store but the PACS is not listening anymore (https://stackoverflow.com/a/14305484). Orthanc probably tries to send the ACK after it has compressed the file which might be considered as “too long” by the PACS.

Any chance you could increase that timeout on the PACS ? In Orthanc, we have such a timeout:

  // The timeout (in seconds) after which the DICOM associations are
  // considered as closed by the Orthanc SCU (client) if the remote
  // DICOM SCP (server) does not answer.
  "DicomScuTimeout" : 10,

Concerning the memory usage exploding, we would need to find a way to reproduce the issue first … any idea ?

4

I was thinking of trying to increase the timeout on SCU.
I’ve already increased the timeout on SCP from 30 to 60 seconds as noted in my original post :slight_smile:

I may be able to isolate the problem files…however, I don’t have direct access to them on the source PACS. This is sort of a convoluted project.

But I am finding that when I look at the source PACS next to Orthanc and see the discrepancy, I can send the individual dicom instances without issue…USUALLY. Sometimes there’s a dicom that just won’t copy and it just eats the memory on the Orthanc server. Once I find one that won’t go through, I can reproduce this behavior every time I attempt the to copy the study/image.

5

FYI, I’m able to reproduce the network error and high memory usage with this command:

storescu -aec ORTHANC localhost 4246 -ts 1 IM_0001.dcm

where the dcm file is 150 MB which consumes around 500MB in RAM while being compressed.

The storescu commands exits after 1 second while Orthanc has already received the file and started compressing it (consuming 500MB).
If I issue a storescu command directly after, Orthanc is actually compressing 2 files at the same time → 1GB of RAM usage.
If I pile up more commands like these, memory continues to increase until I let Orthanc finish compressing the files …

So I would say you would really need to increase the timeout on the PACS or “slow it down” in case of errors.

6

I’m increasing the SCU timeout to 60 seconds from 10.
Sometimes these dicom files are larger files…maybe 1.2-1.4GB for a single dicom instance. I’m not sure why they would eat all 14GB of RAM the server has available, but we will see if increasing the SCU timeout helps.

If the dicom fails to be received into Orthanc and subsequently generates a DIMSE error in one of these situations, it usually requires a complete reboot of the server…Orthanc will hang at 14GB memory usage and I cannot stop the service to restart it. We will see what increasing the SCU does.

I may be able to get in touch with the source PACS vendor and have them decrease the PDU to better accomodate these large files being received. You may be correct in that it’s the compression that’s causing the issue. It takes extra time on the Orthanc side to process due to the fact that I’m compressing the files as they come in. Makes sense.

7

I attempted to send the “problem” instances one at a time, with no other send jobs in the source PACS queue. The memory usage of Orthanc jumped to about 6-8GB just with that single dicom instance. It processed fine.

I am not sure if the setting the SCU/SCP timeouts within Orthanc helped with this, or if it was just because the source PACS is not sending any other dicom instances at the same time and Orthanc only has to process/compress one dicom instance.

Either way, I believe you are correct, Alain. I will see if I the vendor of the source PACS will work with me on reducing the PDU so Orthanc can accept some of these larger instances that seem to be causing the problem.

8

Hello guys, I have similar issues like yours, but instead of one single instance of 150MB, I have 20k+ instances in one single study that would take up about 700MB file storage. Both downloading and transferring this large study would consume huge memory and long time to complete. And I guess it is because orthanc is trying to compress and uncompress during the process.

NOTE: I’ve turned off StorageCompression in config file and set {“Compression”: “none”} in transferring body.

在2020年3月12日星期四 UTC+8 00:57:45alain...@osimis.io 写道：

9

Qaler, your question is unrelated to the present thread (that is about storescu, not about the transfers accelerator plugin).

I’m now locking this thread, as you have also started another one to discuss your specific problem:

https://groups.google.com/g/orthanc-users/c/_EgkiqmjoDk/m/i8_vjEC-DgAJ