Exception while invoking plugin service 3010: Error in the plugin implementing a custom storage area

Google Groups archive
1

Compiled the AWS S3 plugin (Osimis version) for OSX and it seemed to compile and load OK:

W1121 20:18:56.189536 main.cpp:802] Loading plugin(s) from: libOrthancAwsS3Storage.mainline.dylib
W1121 20:18:56.216348 PluginsManager.cpp:269] Registering plugin ‘AWS S3 Storage’ (version mainline)
W1121 20:18:56.217111 PluginsManager.cpp:168] AWS S3 Storage plugin is initializing
W1121 20:18:58.237113 PluginsManager.cpp:168] AWS S3 Storage: client-side encryption is disabled
W1121 20:18:58.237224 main.cpp:1405] Using a custom database from plugins
W1121 20:18:58.237244 main.cpp:1416] Using a custom storage area from plugins

I sent a 1 image study to to my local instance of Orthanc via a local HOROS, and it sent to Orthanc, but nothing in the log with that. When I look at my Orthanc Explorer, the study info shows up.

When I try to view using the Osimis Viewer I get this in the log:

E1121 20:29:26.048836 PluginsManager.cpp:164] AWS S3 Storage: error while reading object 1ef9c380-4feb-40f3-a0cd-4165f1e44e6a, size of file is too small: 0 bytes
E1121 20:29:26.049068 PluginsManager.cpp:197] Exception while invoking plugin service 3010: Error in the plugin implementing a custom storage area
E1121 20:29:26.049207 PluginsManager.cpp:164] (SeriesController) Orthanc::OrthancException 7/404 Accessing an inexistent item

In the browser console I see a 404 error for:

http://localhost:8042/osimis-viewer/series/8bccacc7-d470b154-446c649d-2f08a482-b18cd25d

In my AWS Console I see a list of a .json file and .dcm file in my bucket, and that matches when is stored in the attachedfiles table in my Postgres database.

The .json shows on AWS shows:

AccessDenied Access Denied 1S7MDWBTCV8XAM7R omVL0SZv9YtceuCr8siZBUlQodvv2kUkD7gmeeEMblXtyBgpR6eWzw6r37+Q5ZXh3oI1SykUvb8=

My storage config is like:

“AwsS3Storage” : {
“BucketName”: “xxxx”,
“Region” : “us-east-2”,
“AccessKey” : “xxxx”,
“SecretKey” : “xxxx”,
“Endpoint”: “”, // custom endpoint
“ConnectionTimeout”: 30, // connection timeout in seconds
“RequestTimeout”: 1200, // request timeout in seconds (max time to upload/download a file)
“RootPath”: “”, // see below
“MigrationFromFileSystemEnabled”: false, // see below
“StorageStructure”: “flat”, // see below
“VirtualAddressing”: true // see the section related to MinIO
},

“PostgreSQL” : {
“EnableIndex” : true,
“EnableStorage” : false,
“Host” : “localhost”,
“Port” : 5432,
“Database” : “Orthanc”,
“Username” : “xxx”,
“Password” : “xxx”,
“EnableSsl” : false // New in version 3.0
},

I presume that that has something to do with the AWS config and nothing to do with the plugin or Orthanc ??

BTW, the Google interface changed again, and some issues with sending messages. Is there a way to send messages using your own e-mail client if you use the same e-mail as the google account ?

Stephen D. Scotti

2

Hi Stephen,

I’m afraid we won’t be able to provide support. We’ve never planned to run this plugin on OSX.
This plugin is supposed to run in a VM in the same datacenter as the S3 storage (which is obviously not possible with OSX)

Please test it first with a Linux setup, then, once you’ve validated your configuration with the Linux setup, move to the OSX setup.

Best regards,

Alain.

BR

Alain

3

Alain,

I did go ahead and compiled everything from scratch for UBUNTU on my VPS. Instructions below for both OS X and LINUX: Seems to be working now with my Trial of AWS, although I probably need to understand the “MigrationFromFileSystemEnabled” option a bit better. Also need to check out the performance. I’m really looking to use that as a backup and not as real-time access since it’ll less expensive that way if we use Glacier storage and do not make repeated requests.

BTW, just wondering if there are S3 services that provide similar service for a lesser price than Amazon ?

Also, I am running Orthanc on a VPS with a hosting service and I have the AWS trial S3 bucket. Not sure if that is how it is suppose to be used, but I’m just experimenting around a bit now.

I’m actually looking for a way to just store in the cloud for long-term backup and disaster recovery while have most studies still on our local server until we get somewhat over capacity, so wondering if I can leverage the MigrationFromFileSystemEnabled somehow for that or if I’d have to have a local server running also and use a Lua script or something store studies there also.

“Orthanc will store all new files into the object storage but will try to read/delete files from both the file system and the object storage.” I’d be interested in storing in both places. Perhaps there would also be a way just not use the S3 plugin but still use S3 storage as a backup using the AWS SDK ?

I have an inquiry with AWS about why it was not working with the OS X plug-in. It is strange because the files with the uuid’s were created in the S3 bucket, but the file size was 0B and they were empty. My home system is in an apartment building with layers of routers, so wonder if it might have something to do with that and there is nothing wrong with the plug-in. Using OS X is a matter of convenience mostly when it comes to development.

Compilations Instructions for AWS SDK and for Osimis AWS S3 Plugin, OS X and LINUX

4

Apologies for resurrecting an old thread, but I wanted to add that under debian buster I get a segmentation fault when starting Orthanc with the Aws plugin built in accordance with the instructions on https://book.orthanc-server.com/plugins/object-storage.html#id10.

The backtrace is as follows (using the orthanc 1.7.3 debug binary):
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007fb2c71fd535 in __GI_abort () at abort.c:79
#2 0x00007fb2c7254508 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fb2c735f28d “%s\n”) at …/sysdeps/posix/libc_fatal.c:181
#3 0x00007fb2c725ac1a in malloc_printerr (str=str@entry=0x7fb2c735d43b “free(): invalid pointer”) at malloc.c:5341
#4 0x00007fb2c725c42c in _int_free (av=, p=, have_lock=) at malloc.c:4165
#5 0x00007fb2c5d8f904 in Json::Reader::parse(std::_cxx11::basic_string<char, std::char_traits, std::allocator > const&, Json::Value&, bool) () from /usr/lib/x86_64-linux-gnu/libjsoncpp.so.1
#6 0x00007fb2c60395e0 in OrthancPlugins::OrthancString::ToJson (this=0x7ffe8de2d998, target=…) at /root/build/aws/orthanc/OrthancServer/Plugins/Samples/Common/OrthancPluginCppWrapper.cpp:394
#7 0x00007fb2c6039dde in OrthancPlugins::OrthancConfiguration::LoadConfiguration (this=0x7ffe8de2deb0) at /root/build/aws/orthanc/OrthancServer/Plugins/Samples/Common/OrthancPluginCppWrapper.cpp:522
#8 0x00007fb2c6039f44 in OrthancPlugins::OrthancConfiguration::OrthancConfiguration (this=0x7ffe8de2deb0) at /root/build/aws/orthanc/OrthancServer/Plugins/Samples/Common/OrthancPluginCppWrapper.cpp:534
#9 0x00007fb2c602206a in OrthancPluginInitialize (context=0x3b35980) at /root/orthanc-object-storage/Common/StoragePlugin.cpp:287
#10 0x00000000009569e6 in Orthanc::CallInitialize (plugin=…, context=…) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Plugins/Engine/PluginsManager.cpp:98
#11 0x0000000000957be2 in Orthanc::PluginsManager::RegisterPlugin (this=0x3b76470, path=“/usr/local/share/orthanc/plugins/libOrthancAwsS3Storage.so”) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Plugins/Engine/PluginsManager.cpp:272
#12 0x0000000000958015 in Orthanc::PluginsManager::ScanFolderForPlugins (this=0x3b76470, folder=“/usr/local/share/orthanc/plugins”, isRecursive=false) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Plugins/Engine/PluginsManager.cpp:316
#13 0x000000000095777d in Orthanc::PluginsManager::RegisterPlugin (this=0x3b76470, path=“/usr/local/share/orthanc/plugins”) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Plugins/Engine/PluginsManager.cpp:248
#14 0x000000000083bdf9 in LoadPlugins (plugins=…) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Sources/main.cpp:803
#15 0x000000000083f4e4 in ConfigurePlugins (argc=2, argv=0x7ffe8de2eec8, upgradeDatabase=false, loadJobsFromDatabase=true) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Sources/main.cpp:1400
#16 0x000000000083f741 in StartOrthanc (argc=2, argv=0x7ffe8de2eec8, upgradeDatabase=false, loadJobsFromDatabase=true) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-LSB_Debug/build/OrthancServer/Sources/main.cpp:1453
#17 0x000000000083ffb4 in main (argc=2, argv=0x7ffe8de2eec8) at /home/jodogne/BuildBotWorker/Orthanc_1_7_3-_LSB_Debug/build/OrthancServer/Sources/main.cpp:1667

When I ask cmake to build a static library (-DSTATIC_BUILD=ON) for the plugin however, the seg fault is not raised anymore.

Looking at the cmake output differences between both cases, I notice libjson is newer in the static case (1.8.4 vs 1.7.3, which is packaged in debian buster). Additionally I also noticed that cmake also appears to re-download and build the aws sdk in the dynamic lib case.

Should some special care be taken when trying to build the AWS plugin as a dynamic library in regards to its dependencies?

Sincerely,
Floris

5

Floris,

I actually left the group for a bit because it was suggested that I not make too many posts, but I was still checking the forum and saw this one, so I am responding.

I did build successfully on OS X, although I never tried doing so on UBUNTU. I did eventually get it to work with AWS as far as storing the data files as well, and then tried it in a dev environment. The latency was a bit too long for production use when not hosing Orthanc on AWS, (We are running our own local server for that) and the cost for a regular S3 bucket would be to costly. I have not abandoned the idea of using S3 Glacier for archival / backup purposes, but I think I’ll look for another solution for backup and recovery. I might try again with using both plug-in and UBUNTU.

I think there is another S3 plug-in for Orthanc on GitHub, but not sure if that is. maintained. radpointhq / orthanc-s3-storage

6

Hello,

This is a general rule: The LSB (Linux Standard Base) binaries of Orthanc will almost always crash (segmentation fault) if loading a plugin (shared library) that is not statically linked.

Roughly speaking, this is caused by the fact that a dynamically-linked plugin will try in the first place to “reuse” the symbols that are stored in the LSB binaries. But, because the libraries used in the LSB binaries have a different version than those of the plugin, the memory layout of some of these symbols and their associated data structures are different (ABI incompatibility), which inevitably leads to a crash.

The solution is either to compile the main Orthanc binaries natively on your system (no use of the LSB binaries anymore), so that Orthanc and the plugins will all use the same version of the system-wide libraries, or to statically link the plugins (by compiling them either using the LSB SDK, or using the “-DSTATIC_BUILD=ON” CMake option).

Note however that the “-DSTATIC_BUILD=ON” option is not currently available in some plugins, notably those related to cloud storage, because of the huge number of dependencies of their C++ client libraries.

If you don’t want to compile the plugins by yourself, Osimis provides professional Docker images containing precompiled versions of these plugins.

HTH,
Sébsatien-

7

Thanks you for your pointers Sébsatien,

I’ve noticed that even when getting orthanc to load the AWS plugin and in cases where orthanc doesn’t segfault when processing a DCM file (this appears to be only the case for -DSTATIC_BUILD=ON when compiling the plugin), the AWS plugin still does not succeed in uploading to S3. Orthanc logs the following error when processing a dcm instance upload:

orthanc_1 | E1223 08:01:36.828817 PluginsManager.cpp:164] AWS S3 Storage: error while creating object 99f22db3-cda2-48f9-afde-b4f145d5f8ba: error while writing file 99f22db3-cda2-48f9-afde-b4f145d5f8ba.dcm: curlCode: 60, SSL peer certificate or SSH remote key was not OK

This appears to be related to the AWS CPP SDK, which the plugin uses for its s3 functionality. However I have been unsuccessful in debugging this further. Note libcurl4-openssl-dev is used under Buster for the AWS SDK’s curl dependency.
Maybe it is due to the plugin not actually supporting static linking (eventhough it does execute). Maybe the problem lies elsewhere. I’d say my config section looks ok though:

“AwsS3Storage” : {
“BucketName”: “orthanc”,
“Region” : “eu-west-1”,
“AccessKey” : “AKxxx”,
“SecretKey” : “Kyy”,
“ConnectionTimeout”: 30, // connection timeout in seconds
“RequestTimeout”: 1200, // request timeout in seconds (max time to upload/download a file)
“RootPath”: “”, // see below
“MigrationFromFileSystemEnabled”: false, // see below
“StorageStructure”: “flat”, // see below
“VirtualAddressing”: true // see the section related to MinIO
},

Is there any documentation available about compiling both orthanc and the plugins on the same system, not using the LSB? Following https://book.orthanc-server.com/faq/compiling.html#compiling and the instructions for every plugin should get me there?

Sincerely,
Floris

8

orthanc_1 | E1223 08:01:36.828817 PluginsManager.cpp:164] AWS S3 Storage: error while creating object 99f22db3-cda2-48f9-afde-b4f145d5f8ba: error while writing file 99f22db3-cda2-48f9-afde-b4f145d5f8ba.dcm: curlCode: 60, SSL peer certificate or SSH remote key was not OK

This error indicates that the statically-linked libcurl library, that is used by the AWS C++ client library for its REST calls, has no access to the HTTPS certificates.

On Debian systems, this file is stored at “/etc/ssl/certs/ca-certificates.crt”. You might try to set the “HttpsCACertificates” configuration option as for the Google Cloud Platform plugin, but I’m not the author of the S3 plugin, so I cannot tell whether this would work:
https://book.orthanc-server.com/plugins/google-cloud-platform.html#common-parameters

You would have better chance in using dynamic compilation, or in using the Docker images by Osimis that are already properly configured (cf. my previous answer).

Sébastien-

9

Just to provide another update: I’ve managed to get the statically linked plugin working by patching the S3 client initialization in orthanc-object-storage to pass along a CA certificate file to the AWS SDK (similar to what you described with HttpsCACertificates). Passing the root CA file ‘/etc/ssl/certs/ca-certificates.crt’ fixed curl failing to verify its peers for me.

I’ve added the patch as an attachment. Whether you’d want to use the resulting 17 MB libOrthancAwsS3Storage.so file is another matter. A better option would indeed be to recompile the entire project and use dynamic linking or to subscribe to the osimis images.

I also edited the orthanc-python Dockerfile for building the AWS plugin and creating a new image with the static plugin embedded. It’s available here https://github.com/hyperfusion-ai/OrthancDocker/tree/1.7.3-aws/orthanc-aws The image size delta is considerable at 115 MB over the orthanc-python image.

Thank you for your work on packaging orthanc in docker btw, Sebastian. It really helps a lot to get going with orthanc (even though in this case it would be better to compile orthanc from scratch).

Sincerely,

Floris

orthanc-object-storage_AwsS3StoragePlugin-ca.patch (963 Bytes)

10

Thank you so much, Floris! Your patch was exactly what I needed. I now have the S3 plugin working.

11

Hi there, i´ve found the same error, my solution was =>

nano /etc/orthanc/orthanc.json => edit

“HttpsVerifyPeers” : false,
“HttpsCACertificates” : “/etc/ssl/certs/ca-certificates.crt”,