Cannot get libOrthancAwsS3Storage.so working on latest Orthanc on Ubuntu24

My config is as follows:

{
    "AwsS3Storage": {
        "BucketName": "dicom",
        "Region": "us-east-1",
        "AccessKey": "********************", // optional: if not specified, the plugin will use the default credentials manager (available from version 1.3.0)
        "SecretKey": "********************", // optional: if not specified, the plugin will use the default credentials manager (available>
        "Endpoint": "", // optional: custom endpoint
        "ConnectionTimeout": 30, // optional: connection timeout in seconds
        "RequestTimeout": 1200, // optional: request timeout in seconds (max time to upload/download a file)
        "RootPath": "", // optional: see below
        "MigrationFromFileSystemEnabled": false, // optional: see below
        "StorageStructure": "flat", // optional: see below
        "EnableLegacyUnknownFiles": true, // optional: see below
        "VirtualAddressing": true, // optional: see the section related to MinIO
        "StorageEncryption": {}, // optional: see the section related to encryption
        "EnableAwsSdkLogs": true
    }
}

However, in the logs I always get the following. I never had this issue with the old version. It worked just fine.

Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0 response headers:
W0910 20:38:11.029000     7f36b7e006c0 PluginsManager.cpp:157] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
W0910 20:38:11.029011     7f36b7e006c0 PluginsManager.cpp:157] AWSClientRequest failed, now waiting 6400 ms before attempting again.

Hi Cali,
From the look of the logs it looks like a SSL error. I would try manually checking if you can use curl to access the S3 url directly. Ie: curl https://s3.us-east-2.amazonaws.com -v

In my case the response is

* Host s3.us-east-2.amazonaws.com:443 was resolved.
* IPv6: (none)
* IPv4: 52.219.232.1, 52.219.178.97, 52.219.228.217, 52.219.178.33, 52.219.80.91, 3.5.132.187, 3.5.132.223, 52.219.94.89
*   Trying 52.219.232.1:443...
* Connected to s3.us-east-2.amazonaws.com (52.219.232.1) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=*.s3.us-east-2.amazonaws.com
*  start date: Feb 29 00:00:00 2024 GMT
*  expire date: Feb 12 23:59:59 2025 GMT
*  subjectAltName: host "s3.us-east-2.amazonaws.com" matched cert's "s3.us-east-2.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: s3.us-east-2.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 307 Temporary Redirect
< x-amz-id-2: y95ngiVgg3qKEi49OjXEN9+akHhFwLa78aF2SBIaWBjuY7SZQaWvuVH8Yx9gVGZk5heCHoDtqhg=
< x-amz-request-id: TZ0NEY695Y6YKEAS
< Date: Wed, 11 Sep 2024 01:17:40 GMT
< Location: https://aws.amazon.com/s3/
< Server: AmazonS3
< Content-Length: 0
< 
* Connection #0 to host s3.us-east-2.amazonaws.com left intact

But if you get a security warning along the lines of

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It is likely to be an issue validating the certificate on your particular system. This could be because of incorrect time, lack of ca-certificates installed or many other reasons.

Hope this helps.

James

I’ve verified that all of my certs are valid and that my timing on my system is valid through Amazon standards. I’ve manually tested the cURL connections myself even. Something is going wrong… Whenever I try to store an instance I get this:

E0911 07:16:21.461980          DICOM-0 PluginsManager.cpp:153] AWS S3 Storage: error while creating object 36cb8de2-104d-4948-8168-05e40c03f9c9: error while writing file 36cb8de2-104d-4948-8168-05e40c03f9c9.dcm: response code = -1  curlCode: 60, SSL peer certificate or SSH remote key was not OK
E0911 07:16:21.462276          DICOM-0 StoreScp.cpp:198] Exception while storing DICOM: Error in the plugin implementing a custom storage area

@James Additionally, I had this exact same config prior to upgrading the kernel from Ubuntu22 to Ubuntu24 and it worked just fine.

I’m not familiar with running Orthanc directly on Linux - we exclusively use Orthanc in docker and definitely don’t have this issue with S3.

My suspicion is that orthanc can’t find the list of trusted certificates. Can you turn on verbose logs and see where Orthanc is loading CA Certs from to double check that path is correct?

The issue seems to extend beyond Ubuntu. I just provisioned a fresh Ubuntu22, Ubuntu24 and Debian machine. Of which I only compiled and installed the latest version Orthanc along with libOrthancAwsS3Storage.so. All of them resulted in the same error. There’s something wrong with that library on the latest version and I cannot get the old library to compile. @alainmazy I really need your help! :sob:

Hi Cali,

Same on our side, we always use the docker image and have never encountered the issue. Possibly check this message if not already done.

Instead of rebuilding, could you try with the LSB binaries (S3 and Orthanc) ?

There is also an option to enable the AWS SDK Logs that might help (EnableAwsSdkLogs).

If none of this helps, feel free to share a complete procedure for us to reproduce the issue, including the exact build instructions that you use and configuration file …

I’m also sharing this part of the Orthanc book because you probably understand that this is the kind of investigations that can take us many hours:

Note about proprietary cloud storage

These plugins are used to interface Orthanc with commercial and proprietary cloud services that you accept to pay. As a consequence, the Orthanc project usually doesn’t freely update them or fix them unless the requester purchases a support contract, e.g., at Orthanc Team.

Although you are obviously free to compile these plugins by yourself (instructions are given below), purchasing such support contracts makes the Orthanc project sustainable in the long term, to the benefit of the worldwide community of medical imaging.

Best regards,

Alain.

Hey @alainmazy I had checked all of those messages on this board. Additionally, I had EnableAwsSdkLogs enabled. I had tried both LSB binaries in addition to compiling multiple different versions of the sources. I tried self-updating some of the old object storage sources. I spent two sleepless days trying to resolve this issue. Our primary production machine had an issue and so I was in a desperate panic to provision a replacement. I eventually had to roll back to one of my snapshots and get it all current and connected back properly. Unfortunately using Docker is not an option for us currently due to our complex infrastructure and lean expenses. Hopefully someday soon I can convert everything over to Docker.

I’ll share the details with you as requested once I’ve had a chance to regain some of my sanity. :sob: