Error with the Orthanc Logging ( Log Rotation )

General
1

Hey Everyone,
I have configured my Orthanc using my custom docker image with both postgres (metadata only ) and S3 bucket (for storing the actual media files i.e. the .dcm files ) .
I have also mounted the logs generated by orthanc to my host system for easy accessibility and proper maintainance.
Now, the problem occurred when i deleted some recently created logs (because of some storage issue with my server ) say around 50 of them, then onwards , each time the logging is facing some issue , each new logging is getting appended to the most recent file and it’s size is getting increased rapidly.
At start it’s size is around 200 kb max than it goes upto gb’s (max i’ve seen yet is 5gb) .
So far I’ve done following things :

  1. I’ve tried deleting the last file , but still i guess the data (logs) were getting added to that deleted file (as per gpt) , or it wasn’t getting added anywhere.
  2. I’ve tried recreating the containers several times by keeping the mounts intact with the logs (removing the last problem file) , it still had the same issue , a new file was created and then gets appended again.
  3. I’ve tried deleting the logs completely and then restart/rebuild the containers , still the last file is getting all the logs appended.
    This wasn’t happening before.
    If anyone is suggesting to use logrotate , than let me say that this was working fine before , the logging took max space of some Kb’s and then it automatically internally got rotated as per it’s internal configuration , which is happening right now with the rest of the logs prior/previous to the last log file.
    I am also sharing the size of the last three files and the remaining log files are having almost same size (in some kb’s).
    username@ip-x-x-x-x:/orthanc-logs$ ls -lh Orthanc.log.20241214-100111.1
    -rw-r–r-- 1 root root 329M Dec 16 16:18 Orthanc.log.20241214-100111.1
    username@ip-x-x-x-x:/orthanc-logs$ ls -lh Orthanc.log.20241216-104906.1
    -rw-r–r-- 1 root root 2.1G Dec 19 11:37 Orthanc.log.20241216-104906.1
    username@ip-x-x-x-x:/orthanc-logs$ ls -lh Orthanc.log.20241214-093333.1
    -rw-r–r-- 1 root root 180K Dec 14 15:31 Orthanc.log.20241214-093333.1
    username@ip-x-x-x-x:/orthanc-logs$ ls -lh Orthanc.log.20241214-091458.1
    -rw-r–r-- 1 root root 74K Dec 14 14:46 Orthanc.log.20241214-091458.1
    username@ip-x-x-x-x:/orthanc-logs$

I appreciate your help in this concern of mine.
Thanks in advance.

Himanshu.

2

Hi Himanshu

Maybe Docker has somehow cached the handle to the file somewhere.

When I have trouble with bind mounts, usually, running docker-compose down --volumes fixes the problem. Beware: this will delete all Docker volumes from this particular Compose setup.

You might want to give it a try.

(If you’re not using docker compose, the equivalent docker command would be docker rm -v, it seems)

If these don’t work, restarting the Docker daemon might help (either through the Docker Desktop GUI or with sudo systemctl restart docker)

Hope this helps.

3

Hey Benjamin,
Thanks for addressing out these possible options to resolve my problem.
However , i do not have any volumes attached in my system, I am having simply bind mounts.
Although, I have tried restarting the docker daemon, completely deleting the logs again , and then now instead of new log file creation , only one file is created as of now , and it is increasing rapidly.
Here is the size comparison :
Just after the container creation :
this was the size just immediately when i restarted the container :

aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls -lh Orthanc.log.20250103-105817.1

-rw-r–r-- 1 root root 67K Jan 3 16:31 Orthanc.log.20250103-105817.1

After 4 minutes, this is the log size when i tried to access some of the files present in it :

aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls

Orthanc.log Orthanc.log.20250103-105817.1

aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls -lh Orthanc.log.20250103-105817.1

-rw-r–r-- 1 root root 29M Jan 3 16:35 Orthanc.log.20250103-105817.1

aman.gupta@ip-10-12-1-83:/orthanc-logs$

and this is now when i accessed for further more time :

aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls -lh Orthanc.log.20250103-105817.1

-rw-r–r-- 1 root root 31M Jan 3 16:38 Orthanc.log.20250103-105817.1

aman.gupta@ip-10-12-1-83:/orthanc-logs$

and now this has grown up to this size :
aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls -lh Orthanc.log.20250103-105817.1
-rw-r–r-- 1 root root 98M Jan 3 16:55 Orthanc.log.20250103-105817.1
aman.gupta@ip-10-12-1-83:/orthanc-logs$ docker volume ls
DRIVER VOLUME NAME
aman.gupta@ip-10-12-1-83:/orthanc-logs$ ls -lh Orthanc.log.20250103-105817.1
-rw-r–r-- 1 root root 108M Jan 3 17:34 Orthanc.log.20250103-105817.1
aman.gupta@ip-10-12-1-83:/orthanc-logs$

So , this shouldn’t happen.
Also, let me give some brief too , i am having my dicom images available on the orthanc server , having db (postgres) on a different server and s3 configuration for media files.
Now, i am using the orthanc rest api to fetch the data from orthanc and use it further accordingly.

If you need any further details, please mention it.
And in case you need the compose file and configuration file details, you can access it from here :

In this linked conversation you have also helped me in the configuration part too.

So , if you can assist me further in this issue, that will be really appreciated.
Even i can arrange a short meeting (if you are available) to explain you the entire scenario.

Thanks and Regards,
Himanshu

4

Hi Himanshu,

Actually I don’t really understand how your system is supposed to work: since Orthanc does not deal with log file rotation, it’s pretty normal that you have a single file that always grows, is that right? Is your problem related to the fact that it grows too quickly? If so, the obvious question: what does it contain?

If your problem is that the file does not get recreated when Orthanc restarts, then indeed that is very weird, but I am not sure this is what you’re saying (I guess Orthanc will simply name the file with the current datetime when creating it, so it’s highly unlikely that a file gets recycled, unless there are details I am not aware of in the log mgmt)

But, more importantly, I have never tried to use log files with containerized processes in this fashion, and I do not think it’s really a common scenario nowadays.

Usually, you simply let the process send the log output to stdout and you use the container orchestration tools to grab the logs (docker logs, docker-compose logs, etc…)

If I understand correctly, you are trying to rotate the log files manually while Orthanc is running, is that right? Working with log files in this fashion is unusual. I am not saying it should not work, but depending on how the container process and your own actions interfere, I guess that nasty things can happen.

Can’t you instead deal with the logs outside of the containers? If I were you I would try to stop using log files and get back to regular docker logging (standard output), and check how things are going: for instance, is the log volume normal when working like that?

Then you might want to compare this regular log output with what you’re getting in your “shared logs folder” system: where do these huge logs come from? What do they contain?

Maybe someone else has experience with this kind of log management, but I don’t.

A few pointers on log mgmt: Container Logging: Best Practices for Docker and Kubernetes | by Anshuman Tripathi | Medium

UPDATE: there’s an example of a Docker logging driver configuration that enables rotation here: How to setup log rotation for a Docker container | by Ying Kit Yuen | We’ve moved to freeCodeCamp.org/news | Medium

HTH