Orthanc 1.5.6 eating /tmp disk continuously

Google Groups archive
1

Dear group!

We’ve been running our production system on orthanc 1.5.6 on docker for roughly 6 months and we see a behaviour which needs managing. The system continuously eats /tmp space inside the docker container. You can see the content of the docker container /tmp folder prior to today’s cleaning activity. I’ve also attached an image from our disk monitoring of the system over the last 6 months. on that image you see the blue line representing the DICOM storage growth (on a seperate volume) and the yellow line the system volume growth including the docker container.

The system uses an RDS postgres DB for indexing and a seperate EBS volume for DICOM storage. So the growth is purely related to the inner workings of orthanc.

We find that by restarting the container, some of these tmp files go away, but not all.
by removing the container and rebuilding it, the /tmp folder of course reset to zero.

a couple of questions:

  1. is there a way to make orthanc manage it’s temp files so that they don’t grow indefinitely? maybe a system config parameter or a cron job which can be scheduled.
  2. is the behaviour normal, or does it indicate some misconfiguration?

thank you all for a great forum!
Pär
www.cmrad.com

root@d67874e5e343:/tmp# ls -l

total 2695360

-rw-r–r-- 1 root root 114623382 Aug 26 20:50 Orthanc-040209a4-0be5-4be2-be20-9f2ea8a86bbc

-rw-r–r-- 1 root root 521502922 Jul 24 09:21 Orthanc-063c0737-3a0b-4486-872e-18a4550eea47

-rw-r–r-- 1 root root 5926051 Jul 24 09:02 Orthanc-19c38036-b95e-428f-a0c7-be7d58a3d45d

-rw-r–r-- 1 root root 51177778 Jul 29 11:15 Orthanc-310ac083-e68e-4cac-adf7-6c62dcd64f6f

-rw-r–r-- 1 root root 114623382 Aug 26 19:10 Orthanc-389e26e0-abdb-4597-af19-73ba794c134f

-rw-r–r-- 1 root root 45888007 May 8 18:46 Orthanc-53ea677e-cca8-4ef1-94c8-b55751707b14

-rw-r–r-- 1 root root 114623382 Aug 26 19:09 Orthanc-5bca9c66-fc26-47da-9ee1-10a1c52c36d9

-rw-r–r-- 1 root root 9032716 Aug 26 20:53 Orthanc-5ccc1d00-4273-4821-a612-b427a12c3a6d

-rw-r–r-- 1 root root 58735156 Jul 29 11:17 Orthanc-6238ad88-c7fe-41a9-82f7-b18eca0dfea1

-rw-r–r-- 1 root root 114623382 Aug 26 19:10 Orthanc-6acc86e2-9670-41b5-b978-9c5c705231e2

-rw-r–r-- 1 root root 60428428 Aug 5 16:47 Orthanc-80ecd5ae-a238-439e-b3ee-47a694c593f0

-rw-r–r-- 1 root root 114623382 Aug 26 19:09 Orthanc-87d4da87-df24-447b-ad48-b7354bb2281b

-rw-r–r-- 1 root root 114623382 Aug 26 19:10 Orthanc-93c7556f-12c0-4726-9ca6-be58b69faaeb

-rw-r–r-- 1 root root 112004615 Jul 24 08:35 Orthanc-97e9ab87-5eb8-459c-a993-a6db8bee4668

-rw-r–r-- 1 root root 725912 Aug 19 12:40 Orthanc-9a64f11c-0e71-470f-8500-07c9789679de

-rw-r–r-- 1 root root 143512751 Aug 26 19:08 Orthanc-a0c30653-ba3e-477e-b0ef-6e66354fbf42

-rw-r–r-- 1 root root 69276993 Aug 5 14:24 Orthanc-a70ddd72-e368-4ec2-a7b8-5ddf6754745e

-rw-r–r-- 1 root root 990 Jul 20 18:05 Orthanc-b5d0adc8-c58a-4b87-aef0-74e291667fc8

-rw-r–r-- 1 root root 60428428 Aug 5 10:22 Orthanc-c449e87f-7e8f-48af-8dc7-78c54a14f5af

-rw-r–r-- 1 root root 48585850 May 8 18:48 Orthanc-c8005ffe-2264-4c73-a20a-f3d71245104f

-rw-r–r-- 1 root root 114623382 Aug 26 20:49 Orthanc-d7bf597d-55fa-467e-9d17-e7ebe19c8a39

-rw-r–r-- 1 root root 53122752 Aug 5 14:27 Orthanc-dd2a26df-dc7a-408e-bbb9-282bb5efeab7

-rw-r–r-- 1 root root 31771305 Aug 26 19:03 Orthanc-e4957bd6-5eab-42cf-9f30-c1a7ba4a7c24

-rw-r–r-- 1 root root 523741287 May 8 18:23 Orthanc-e8881bf6-d365-4333-8ebe-d354a89d7cf7

-rw-r–r-- 1 root root 114623382 Aug 26 19:04 Orthanc-f5882bfb-bb22-40fa-b33c-056c4e2f204e

-rw-r–r-- 1 root root 47119557 Jul 29 11:23 Orthanc-fe399ce9-80e5-4342-b08b-e5bb40b82dc2

orthanc disk growth 6m.png

2

Hello,

Orthanc typically creates temporary files when it creates ZIP archives.

The fact that temporary files are left on the disk indicate that Orthanc has exited in a non-clean way, without being able to conclude its finalization (either because of a crash, because of a SIGKILL signal, or because of two SIGTERM signals sent in a row). You should check that your systemd/init services that start/stop your Docker containers are not too aggressive when stopping Orthanc.

This might also result from a bug. If so, please post a minimal working example showing how Orthanc can create a temporary files while not removing it, so that we can work on a fix:
https://en.wikipedia.org/wiki/Minimal_working_example

I personally have no clue about the best practice to handle the “/tmp” folder in conjunction with Docker, maybe other users in this forum can give a hint.

HTH,
Sébastien-

3

Hi Pär!
Hi Sébastien!

If I may suggest something, I’d suggest checking the file type for the leftover files. The ID after the “Orthanc-” prefix suggest it’s, as Sébastien said, a byproduct of compression. This should happen prior to Orthanc sending archives as responses to “…/archive” URL’s.

Unless you mapped your /tmp as a docker volume, your logs suggest this instance ran for a very long time because a couple of files date back to May 8th. Also, something odd happened starting at 19:00 on August 26th; there are many many files from two days ago. Maybe this is a backup routine pulling files through the “…/archive” URL (or any client, for that matter) and it’s timing out; it’s tough to say much now.

What you can do however to mitigate the problem is actually mount your /tmp and have a cron job (or Scheduled Task in Windows lingo) do a “find -delete” on old /tmp files, like so:

4

Hi, Sébastien!

I also have no clue on docker’ised /tmp directories best practices. I just suggested a way to try and buy him some time while a proper solution is worked on.

HTH

5

Hello,

Here are 3 additions wrt. to my message of yesterday:

1- Are you aware of the “TemporaryDirectory” and “MediaArchiveSize” configuration options?
https://bitbucket.org/sjodogne/orthanc/src/default/Resources/Configuration.json

2- You might remove the temporary files with pattern “/tmp/Orthanc-*” as part of your startup script, which would clean up any orphan file before restarting Orthanc.

3- If the second solution is not applicable (e.g. because you run more than one instance of Orthanc on your server), please note that I’ve just submitted a modification to include the process ID of Orthanc in the temporary files it generates:
https://bitbucket.org/sjodogne/orthanc/commits/70524b4acc726adc6a63ecb5540b0586ed429165

This information is useful to detect orphan files that do not belong to any running instance of Orthanc, hereby opening the path to the design of a cron job that would remove such orphan files. This feature will be part of forthcoming 1.5.8 release.

HTH,
Sébastien-

6

Thank you both for the great suggestions. I will decide on an update which stabilises the situation through a housekeeping script to make sure we stop the growth. Your suggestions for that script are appreciated.

A few more details. The only archive compression which takes place on this server are:

  1. The odd download of a zipped study by our sysadmins to troubleshoot incidents. This is a rare activity.
  2. The reception of gzipped transfers from the orthanc-proxies installed at hospitals. This is a very common event and in my head is more likely to be the source. The Transfer plug-in is used.

If I’m able to replicate the behaviour in our non-production environment, I’ll submit a minimal working example. However the replication may not be straight forward and it may depend on the heavy usage of production.

thanks again,
Pär

7

Fine, please keep us updated

Sébastien-