I recently upgraded my main server to 1.12.5 and my Orthanc has been restart randomly, sometimes its every hour, other times it could be every 3-5mins. I checked Orthanc logs and it doesnt show anything related to crashing. On my system logs, it shows a segfault in libc.so.6
Feb 04 22:09:25 irv-dcm-dash kernel: Orthanc[90311]: segfault at c000000 ip 00007c08cd8add91 sp 00007c07317fab50 error 4 in libc.so.6[7c08cd828000+188000] likely on CPU 5 (core 1, socket 1)
Feb 04 20:23:22 irv-dcm-dash kernel: Orthanc[60785]: segfault at 0 ip 00007d7b4c2add91 sp 00007d79af7fab50 error 4 in libc.so.6[7d7b4c228000+188000] likely on CPU 7 (core 3, socket 1)
Feb 04 20:10:53 irv-dcm-dash kernel: Orthanc[55988]: segfault at 28000000 ip 0000728a5e0add91 sp 00007288c1dfe050 error 4 in libc.so.6[728a5e028000+188000] likely on CPU 2 (core 2, socket 0)
Feb 04 19:30:51 irv-dcm-dash kernel: Orthanc[47073]: segfault at 30000000 ip 000071e4e68add91 sp 000071e34abfab50 error 4 in libc.so.6[71e4e6828000+188000] likely on CPU 7 (core 3, socket 1)
Feb 04 19:22:05 irv-dcm-dash kernel: Orthanc[43195]: segfault at 40000000 ip 00007285e3aadd91 sp 00007284477fab50 error 4 in libc.so.6[7285e3a28000+188000] likely on CPU 6 (core 2, socket 1)
-- Boot 73bc3168d7e54c81b36694abaea0d406 --
Feb 04 14:29:26 irv-dcm-dash kernel: Orthanc[4869]: segfault at 1c000000 ip 00007a0fee8add91 sp 00007a0e523ff6e0 error 4 in libc.so.6[7a0fee828000+188000] likely on CPU 7 (core 3, socket 1)
-- Boot 7c07f58733864eaebd4918cfb3d9b6ef --
-- Boot 2b359d0b4ee84bd68cf6cbde484ef3b6 --
Feb 04 12:18:20 irv-dcm-dash kernel: Orthanc[27247]: segfault at 14000000 ip 000076f62e8a5449 sp 000076f4927faaf0 error 4 in libc.so.6 (deleted)[76f62e828000+195000] likely on CPU 0 (core 0, socket 0)
Feb 04 11:29:59 irv-dcm-dash kernel: Orthanc[25952]: segfault at 8000000 ip 00007c96f62a5449 sp 00007c95599ff680 error 4 in libc.so.6[7c96f6228000+195000] likely on CPU 3 (core 3, socket 0)
Feb 04 09:27:08 irv-dcm-dash kernel: Orthanc[1342]: segfault at 18000000 ip 0000714ef44a5449 sp 0000714d585faaf0 error 4 in libc.so.6[714ef4428000+195000] likely on CPU 5 (core 1, socket 1)
-- Boot 0878a5e2ff024ce2afdfe53b45bf442c --
I have another server running the same version with no issue. They are basically setup the same, however this server is used the most. Lots of DICOM in and out.
Is it worth trying to upgrade to 1.12.6 and see if that fixes it. Any input would be helpful.
Edit: Updated to mainline, segfault within the first 10 mins.
I created a new VM, installed Ubuntu 22.04.5 with HWE Kernel. Installed 1.12.6, with the Postgresql 7.1 plugin, and now it seg fault on the postgresql plugin. Am I doing something wrong, primary with my way of upgrading Orthanc and it’s plugins?
Feb 05 18:55:34 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=11/SEGV
Feb 05 18:55:34 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 05 18:55:34 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 8.
Feb 05 18:55:34 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 05 18:55:34 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 05 18:55:37 irv-dash2 kernel: Orthanc[34694]: segfault at 6bd8f9a ip 0000725b0600d3fa sp 00007ffeca0eaff0 error 4 in libOrthancPostgreSQLIndex.so[725b05e9b000+3f3000] likely on CPU 1 (core 1, socket 0)
Feb 05 18:55:37 irv-dash2 kernel: Code: c8 e9 27 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 54 31 c0 49 89 d4 55 48 89 f5 53 89 fb e8 1d de e8 ff 48 98 48 85 c0 74 07 <0f> b6 08 84 c9 75 1f 83 fb 73 77 24 48 8d 05 a7 3a 2a 00 89 da 48
Feb 05 18:55:38 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=11/SEGV
Feb 05 18:55:38 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 05 18:55:38 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 9.
Feb 05 18:55:38 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 05 18:55:38 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
I went ahead and compiled 1.12.6 Orthanc and OrthancPostgresql 7.1 on Ubuntu 24.04, copied it over to my main server and so far no issues. I will see how it goes for the next few hours.
That’s exactly how I upgraded. But in a manual fashion using SCP and setting the permission.
Also, it still segfault after a few hours. Im pretty stump so far. I’m trying everything fresh on a VM with Ubuntu 22.04. Will finish my testing tomorrow.
Worst case, I may have to revert back to using 1.12.2 and revert my postgresql back to the original schema.
Currently running 1.12.6 with Postgresql 7.1 on Ubuntu 22.04. Not getting seg fault now, however I am getting malloc errors causing Orthanc to restart randomly.
Feb 06 21:19:05 irv-dash2 Orthanc[211746]: realloc(): invalid next size
Feb 06 21:19:06 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=6/ABRT
Feb 06 21:19:06 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 06 21:19:06 irv-dash2 systemd[1]: orthanc.service: Consumed 4min 20.980s CPU time.
Feb 06 21:19:06 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 10.
Feb 06 21:19:06 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 21:19:06 irv-dash2 systemd[1]: orthanc.service: Consumed 4min 20.980s CPU time.
Feb 06 21:19:06 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 21:28:39 irv-dash2 snapd[654]: storehelpers.go:954: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Feb 06 21:39:01 irv-dash2 CRON[217042]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 06 21:39:01 irv-dash2 CRON[217043]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 06 21:39:01 irv-dash2 CRON[217042]: pam_unix(cron:session): session closed for user root
Feb 06 21:39:20 irv-dash2 systemd[1]: Starting Clean php session files...
Feb 06 21:39:21 irv-dash2 systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb 06 21:39:21 irv-dash2 systemd[1]: Finished Clean php session files.
Feb 06 21:45:04 irv-dash2 Orthanc[213427]: realloc(): invalid next size
Feb 06 21:45:04 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=6/ABRT
Feb 06 21:45:04 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 06 21:45:04 irv-dash2 systemd[1]: orthanc.service: Consumed 12min 21.372s CPU time.
Feb 06 21:45:05 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 11.
Feb 06 21:45:05 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 21:45:05 irv-dash2 systemd[1]: orthanc.service: Consumed 12min 21.372s CPU time.
Feb 06 21:45:05 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 22:01:30 irv-dash2 Orthanc[218009]: malloc(): unaligned fastbin chunk detected
Feb 06 22:01:30 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=6/ABRT
Feb 06 22:01:30 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 06 22:01:30 irv-dash2 systemd[1]: orthanc.service: Consumed 9min 54.801s CPU time.
Feb 06 22:01:31 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 12.
Feb 06 22:01:31 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 22:01:31 irv-dash2 systemd[1]: orthanc.service: Consumed 9min 54.801s CPU time.
Feb 06 22:01:31 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 22:08:27 irv-dash2 systemd[1]: Starting Update APT News...
Feb 06 22:08:27 irv-dash2 systemd[1]: Starting Update the local ESM caches...
Feb 06 22:08:27 irv-dash2 systemd[1]: apt-news.service: Deactivated successfully.
Feb 06 22:08:27 irv-dash2 systemd[1]: Finished Update APT News.
Feb 06 22:08:28 irv-dash2 systemd[1]: esm-cache.service: Deactivated successfully.
Feb 06 22:08:28 irv-dash2 systemd[1]: Finished Update the local ESM caches.
Feb 06 22:09:01 irv-dash2 CRON[222617]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 06 22:09:01 irv-dash2 CRON[222618]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 06 22:09:01 irv-dash2 CRON[222617]: pam_unix(cron:session): session closed for user root
Feb 06 22:09:02 irv-dash2 systemd[1]: Starting Clean php session files...
Feb 06 22:09:03 irv-dash2 systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb 06 22:09:03 irv-dash2 systemd[1]: Finished Clean php session files.
Feb 06 22:17:01 irv-dash2 CRON[223841]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 06 22:17:01 irv-dash2 CRON[223842]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 06 22:17:01 irv-dash2 CRON[223841]: pam_unix(cron:session): session closed for user root
Feb 06 22:19:19 irv-dash2 Orthanc[220746]: malloc(): unaligned tcache chunk detected
Feb 06 22:19:19 irv-dash2 systemd[1]: orthanc.service: Main process exited, code=dumped, status=6/ABRT
Feb 06 22:19:19 irv-dash2 systemd[1]: orthanc.service: Failed with result 'core-dump'.
Feb 06 22:19:19 irv-dash2 systemd[1]: orthanc.service: Consumed 16min 27.231s CPU time.
Feb 06 22:19:20 irv-dash2 systemd[1]: orthanc.service: Scheduled restart job, restart counter is at 13.
Feb 06 22:19:20 irv-dash2 systemd[1]: Stopped Lightweight, RESTful DICOM server for healthcare and medical research.
Feb 06 22:19:20 irv-dash2 systemd[1]: orthanc.service: Consumed 16min 27.231s CPU time.
Feb 06 22:19:20 irv-dash2 systemd[1]: Started Lightweight, RESTful DICOM server for healthcare and medical research.
Could you give more details about what you are doing with Orthanc when it crashes by looking at the verbose logs and correlate the time-stamps from the system logs showing the Orthanc restart.
What plugins are enabled ? Any custom lua/python code ?
I spent today looking through everything, logs in verbose mode. I saw it would crash when it triggers my lua script to HTTP POST to my server the study info.
Look at the size of data it’s sending, its rather large (my fault for making it submit both series data and study data combined.), I revised the lua script to transmit key fields I need, plus the combined total image count of the exam.