Aws S3 plugin Error

General
1

Hi, I have set up an orthanc server on an EC2 instance on a private subnet (which is behind an nginx reverse proxy on a public subnet) and I have attached a role to access an S3 bucket. I am able to access the S3 bucket from the EC2 instance but when using the storage plugin, I get this error from the logs when I try to upload a file -
error while writing file: response code = -1 curlCode: 60, SSL peer certificate or SSH remote key was not OK

There’s nothing special in my bucket’s config, I haven’t enabled HTTPS anywhere. I’ve set HTTPSVerifyPeers to false as well. I’m assuming Orthanc is loading the plugin correctly as I get this in the logs -
AWS S3 Storage plugin is initializing
AWS S3 Storage: HybridMode is disabled: writing to object-storage and reading only from object-storage
AWS S3 Storage: client-side encryption is disabled
Using a custom storage area for plugins

Any idea as to what is going wrong?

2

Additionally, if I set HttpsVerifyPeers to true and provide /etc/ssl/certs/ca-certificates.crt as the path for HttpsCACertificates, Orthanc just crashes and when I try to restart it, it says - Failed with result ‘core-dump’.

3

Hello

There is a configuration option in the S3 plugin named EnableAwsSdkLogs (this settings has been added in version 2.4.0 of the plugin, that is available in the orthancteam/orthanc image 24.6.3 and higher) the You might want to enable it and also run Orthanc as verbose and perhaps paste the logs here?

4

This is after restarting Orthanc -

W0708 18:26:38.904247 MAIN PluginsManager.cpp:261] Registering plugin ‘AWS S3 Storage’ (version mainline)
W0708 18:26:38.905194 MAIN PluginsManager.cpp:157] AWS S3 Storage plugin is initializing
W0708 18:26:38.905411 MAIN PluginsManager.cpp:157] AWS S3 Storage: HybridMode is disabled: writing to object-storage and reading only from object-storage
E0708 18:26:38.917088 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderInitializing config loader against fileName /var/lib/orthanc/.aws/credentials and using profilePrefix = 0
E0708 18:26:38.917112 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderInitializing config loader against fileName /var/lib/orthanc/.aws/config and using profilePrefix = 1
E0708 18:26:38.917139 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderUnable to open config file /var/lib/orthanc/.aws/credentials for reading.
E0708 18:26:38.917147 MAIN PluginsManager.cpp:153] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
E0708 18:26:38.917161 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderUnable to open config file /var/lib/orthanc/.aws/config for reading.
E0708 18:26:38.917166 MAIN PluginsManager.cpp:153] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
E0708 18:26:38.917298 MAIN PluginsManager.cpp:153] EC2MetadataClientUsing IMDS endpoint: http://REDACTED
E0708 18:26:38.918247 MAIN PluginsManager.cpp:153] ClientConfigurationRetry Strategy will use the default max attempts.
E0708 18:26:38.918296 MAIN PluginsManager.cpp:153] EC2MetadataClientCreating AWSHttpResourceClient with max connections 2 and scheme http
E0708 18:26:38.919160 MAIN PluginsManager.cpp:153] CurlHandleContainerInitializing CurlHandleContainer with size 2
E0708 18:26:38.919191 MAIN PluginsManager.cpp:153] ClientConfigurationRetry Strategy will use the default max attempts.
E0708 18:26:38.921231 MAIN PluginsManager.cpp:153] CurlHandleContainerPool grown by 2
E0708 18:26:38.928143 MAIN PluginsManager.cpp:153] EC2MetadataClientDetected current region as ap-south-1
E0708 18:26:38.929180 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderInitializing config loader against fileName /var/lib/orthanc/.aws/credentials and using profilePrefix = 0
E0708 18:26:38.929207 MAIN PluginsManager.cpp:153] ProfileConfigFileAWSCredentialsProviderSetting provider to read credentials from /var/lib/orthanc/.aws/credentials for credentials file and /var/lib/orthanc/.aws/config for the config file , for use with profile default
E0708 18:26:38.929215 MAIN PluginsManager.cpp:153] ProcessCredentialsProviderSetting process credentials provider to read config from default
W0708 18:26:38.929222 MAIN PluginsManager.cpp:157] STSAssumeRoleWithWebIdentityCredentialsProviderToken file must be specified to use STS AssumeRole web identity creds provider.
E0708 18:26:38.929909 MAIN PluginsManager.cpp:153] SSOBearerTokenProviderSetting sso bearerToken provider to read config from default
E0708 18:26:38.929922 MAIN PluginsManager.cpp:153] SSOCredentialsProviderSetting sso credentials provider to read config from default
E0708 18:26:38.929934 MAIN PluginsManager.cpp:153] InstanceProfileCredentialsProviderCreating Instance with default EC2MetadataClient and refresh rate 300000
E0708 18:26:38.929939 MAIN PluginsManager.cpp:153] DefaultAWSCredentialsProviderChainAdded EC2 metadata service credentials provider to the provider chain.
E0708 18:26:38.931776 MAIN PluginsManager.cpp:153] Aws::Config::AWSConfigFileProfileConfigLoaderUnable to open config file /var/lib/orthanc/.aws/credentials for reading.
E0708 18:26:38.931793 MAIN PluginsManager.cpp:153] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
E0708 18:26:38.931801 MAIN PluginsManager.cpp:153] ProcessCredentialsProviderFailed to find credential process’s profile: default
E0708 18:26:38.931843 MAIN PluginsManager.cpp:153] SSOCredentialsProviderUnable to open token file on path: /var/lib/orthanc/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
E0708 18:26:38.931854 MAIN PluginsManager.cpp:153] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
E0708 18:26:38.935374 MAIN PluginsManager.cpp:153] Aws::Config::EC2InstanceProfileConfigLoaderSuccessfully pulled credentials from metadata service with access key ASIAVRUVPXURCNZP2EDS
E0708 18:26:38.935396 MAIN PluginsManager.cpp:153] Aws::Config::AWSProfileConfigLoaderBaseSuccessfully reloaded configuration.
E0708 18:26:38.935535 MAIN PluginsManager.cpp:153] CurlHandleContainerInitializing CurlHandleContainer with size 25
W0708 18:26:38.939622 MAIN PluginsManager.cpp:157] AWS S3 Storage: client-side encryption is disabled

This is after I try to upload a file -

Receiving a DICOM file of 89.11KB through HTTP
I0708 18:45:21.147042 HTTP-28 OrthancRestApi.cpp:163] (http) Receiving a DICOM file of 89.11KB through HTTP
I0708 18:45:21.154144 HTTP-28 PluginsManager.cpp:161] (plugins) AWS S3 Storage: creating attachment fa25cb98-187e-4718-bf38-175168403761 of type 1
I0708 18:45:21.154248 HTTP-27 PluginsManager.cpp:161] (plugins) AWS S3 Storage: creating attachment 01dbb2d3-859d-4116-8433-03345e6e0817 of type 1
I0708 18:45:21.159474 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Region = ap-south-1
I0708 18:45:21.159496 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseFIPS = 0
I0708 18:45:21.159514 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseDualStack = 0
I0708 18:45:21.159520 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseArnRegion = 0
I0708 18:45:21.159525 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
I0708 18:45:21.159531 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Bucket = REDACTED
I0708 18:45:21.159592 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules engine evaluated the endpoint: https://REDACTED
I0708 18:45:21.159607 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules evaluated props: {“authSchemes”:[{“disableDoubleEncoding”:true,“name”:“sigv4”,“signingName”:“s3”,“signingRegion”:“ap-south-1”}]}
I0708 18:45:21.159722 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Region = ap-south-1
I0708 18:45:21.159733 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseFIPS = 0
I0708 18:45:21.159739 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseDualStack = 0
I0708 18:45:21.159744 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseArnRegion = 0
I0708 18:45:21.159749 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
I0708 18:45:21.159754 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Bucket = REDACTED
I0708 18:45:21.159789 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules engine evaluated the endpoint: https://REDACTED
I0708 18:45:21.159801 HTTP-27 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules evaluated props: {“authSchemes”:[{“disableDoubleEncoding”:true,“name”:“sigv4”,“signingName”:“s3”,“signingRegion”:“ap-south-1”}]}
I0708 18:45:21.160780 HTTP-28 PluginsManager.cpp:161] (plugins) AWSClientFound body, but content-length has not been set, attempting to compute content-length
I0708 18:45:21.161002 HTTP-28 PluginsManager.cpp:161] (plugins) InstanceProfileCredentialsProviderChecking if latest credential pull has expired.
E0708 18:45:21.161014 HTTP-28 PluginsManager.cpp:153] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
I0708 18:45:21.161031 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientCalling EC2MetadataService to get token
I0708 18:45:21.161042 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientRetrieving credentials from http://169.254.169.254/latest/api/token
I0708 18:45:21.161057 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to http://169.254.169.254/latest/api/token
I0708 18:45:21.161064 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I0708 18:45:21.161069 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: 169.254.169.254
I0708 18:45:21.161075 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/6.8.0-1010-aws md/arch#x86_64 lang/c++#C++14 md/GCC#9.3.1 cfg/retry-mode#default
I0708 18:45:21.161080 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-aws-ec2-metadata-token-ttl-seconds: 21600
I0708 18:45:21.161093 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I0708 18:45:21.161098 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I0708 18:45:21.161104 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x5efae36615c0
I0708 18:45:21.161114 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x5efae36615c0
I0708 18:45:21.161282 HTTP-27 PluginsManager.cpp:161] (plugins) AWSClientFound body, but content-length has not been set, attempting to compute content-length
I0708 18:45:21.161536 HTTP-27 PluginsManager.cpp:161] (plugins) InstanceProfileCredentialsProviderChecking if latest credential pull has expired.
I0708 18:45:21.162698 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientHTTP/1.0 200 OK

I0708 18:45:21.162717 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Length: 56

I0708 18:45:21.162727 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Type: text/plain

I0708 18:45:21.162734 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientDate: Mon, 08 Jul 2024 18:45:21 GMT

I0708 18:45:21.162742 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientX-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600

I0708 18:45:21.162750 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientConnection: close

I0708 18:45:21.162757 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientServer: EC2ws

I0708 18:45:21.162764 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient

I0708 18:45:21.162773 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient56 bytes written to response.
I0708 18:45:21.162817 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned http response code 200
I0708 18:45:21.162829 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned content type text/plain
I0708 18:45:21.162834 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse content-length header: 56
I0708 18:45:21.162839 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse body length: 56
I0708 18:45:21.162844 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReleasing curl handle 0x5efae36615c0
I0708 18:45:21.162863 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReleasing curl handle 0x5efae36615c0
I0708 18:45:21.162869 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerNotified waiting threads.
I0708 18:45:21.162896 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientRetrieving credentials from http://169.254.169.254/latest/meta-data/iam/security-credentials
I0708 18:45:21.162908 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to http://169.254.169.254/latest/meta-data/iam/security-credentials
I0708 18:45:21.162914 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I0708 18:45:21.162919 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: 169.254.169.254
I0708 18:45:21.162925 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/6.8.0-1010-aws md/arch#x86_64 lang/c++#C++14 md/GCC#9.3.1 cfg/retry-mode#default
I0708 18:45:21.162930 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-aws-ec2-metadata-token: AQAAACyJ7stViTlvx1O-4Swd7Oej0gyKF0qOocE9-xZNzXsib0ATSQ==
I0708 18:45:21.162961 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I0708 18:45:21.162966 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I0708 18:45:21.162970 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x5efae36615c0
I0708 18:45:21.162991 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x5efae36615c0
I0708 18:45:21.164141 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientHTTP/1.0 200 OK

I0708 18:45:21.164155 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientAccept-Ranges: bytes

I0708 18:45:21.164164 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Length: 12

I0708 18:45:21.164171 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Type: text/plain

I0708 18:45:21.164179 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientDate: Mon, 08 Jul 2024 18:45:21 GMT

I0708 18:45:21.164186 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientLast-Modified: Mon, 08 Jul 2024 18:24:12 GMT

I0708 18:45:21.164194 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientX-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600

I0708 18:45:21.164202 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientConnection: close

I0708 18:45:21.164209 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientServer: EC2ws

I0708 18:45:21.164216 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient

I0708 18:45:21.164224 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient12 bytes written to response.
I0708 18:45:21.164278 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned http response code 200
I0708 18:45:21.164293 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned content type text/plain
I0708 18:45:21.164298 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse content-length header: 12
I0708 18:45:21.164303 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse body length: 12
I0708 18:45:21.164308 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReleasing curl handle 0x5efae36615c0
I0708 18:45:21.164323 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReleasing curl handle 0x5efae36615c0
I0708 18:45:21.164328 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerNotified waiting threads.
I0708 18:45:21.164344 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientCalling EC2MetadataService resource, /latest/meta-data/iam/security-credentials with token returned profile string u4radStorage
I0708 18:45:21.164358 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientCalling EC2MetadataService resource http://169.254.169.254/latest/meta-data/iam/security-credentials/u4radStorage with token.
I0708 18:45:21.164369 HTTP-28 PluginsManager.cpp:161] (plugins) EC2MetadataClientRetrieving credentials from http://169.254.169.254/latest/meta-data/iam/security-credentials/u4radStorage
I0708 18:45:21.164380 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to http://169.254.169.254/latest/meta-data/iam/security-credentials/u4radStorage
I0708 18:45:21.164386 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I0708 18:45:21.164391 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: 169.254.169.254
I0708 18:45:21.164397 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/6.8.0-1010-aws md/arch#x86_64 lang/c++#C++14 md/GCC#9.3.1 cfg/retry-mode#default
I0708 18:45:21.164401 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-aws-ec2-metadata-token: AQAAACyJ7stViTlvx1O-4Swd7Oej0gyKF0qOocE9-xZNzXsib0ATSQ==
I0708 18:45:21.164408 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I0708 18:45:21.164433 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I0708 18:45:21.164438 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x5efae36615c0
I0708 18:45:21.164443 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x5efae36615c0
I0708 18:45:21.165819 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientHTTP/1.0 200 OK

I0708 18:45:21.165838 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientAccept-Ranges: bytes

I0708 18:45:21.165847 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Length: 1574

I0708 18:45:21.165855 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientContent-Type: text/plain

I0708 18:45:21.165862 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientDate: Mon, 08 Jul 2024 18:45:21 GMT

I0708 18:45:21.165869 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientLast-Modified: Mon, 08 Jul 2024 18:24:12 GMT

I0708 18:45:21.165877 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientX-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600

I0708 18:45:21.165884 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientConnection: close

I0708 18:45:21.165891 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientServer: EC2ws

I0708 18:45:21.165898 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient

I0708 18:45:21.165908 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClient1574 bytes written to response.
I0708 18:45:21.165949 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned http response code 200
I0708 18:45:21.165959 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReturned content type text/plain
I0708 18:45:21.165963 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse content-length header: 1574
I0708 18:45:21.165968 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientResponse body length: 1574
I0708 18:45:21.165973 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientReleasing curl handle 0x5efae36615c0
I0708 18:45:21.165989 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReleasing curl handle 0x5efae36615c0
I0708 18:45:21.165995 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerNotified waiting threads.
E0708 18:45:21.166039 HTTP-28 PluginsManager.cpp:153] Aws::Config::EC2InstanceProfileConfigLoaderSuccessfully pulled credentials from metadata service with access key ASIAVRUVPXURCNZP2EDS
E0708 18:45:21.166050 HTTP-28 PluginsManager.cpp:153] Aws::Config::AWSProfileConfigLoaderBaseSuccessfully reloaded configuration.
I0708 18:45:21.166058 HTTP-28 PluginsManager.cpp:161] (plugins) Aws::Config::AWSProfileConfigLoaderBasereloaded config at 2024-07-08T18:45:21Z
I0708 18:45:21.166104 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerNote: Http payloads are not being signed. signPayloads=0 http scheme=https
I0708 18:45:21.166151 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Header String: amz-sdk-invocation-id:ABF071D1-3AA4-4392-BB8C-8876828E0439
amz-sdk-request:attempt=1
content-length:91250
content-md5:fZe7qJL6MY43u9KqgLXslg==
content-type:binary/octet-stream
host:u4rad.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240708T184521Z
x-amz-security-token:

I0708 18:45:21.166165 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigned Headers value:amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
I0708 18:45:21.166179 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Request String: PUT
/01dbb2d3-859d-4116-8433-03345e6e0817.dcm

amz-sdk-invocation-id:ABF071D1-3AA4-4392-BB8C-8876828E0439
amz-sdk-request:attempt=1
content-length:91250
content-md5:fZe7qJL6MY43u9KqgLXslg==
content-type:binary/octet-stream
host:u4rad.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240708T184521Z
x-amz-security-token:
amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
UNSIGNED-PAYLOAD
I0708 18:45:21.166233 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal String to sign: AWS4-HMAC-SHA256
20240708T184521Z
20240708/ap-south-1/s3/aws4_request
141cb8a3a4423649b3a589e89e8e1a8e922902ee1465d76f810479f56d3ead4c
I0708 18:45:21.166247 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal computed signing hash: 4729ecad122b2912cde728d6a352c14e27bee8803e5d4fceb410e6908e4c6e6f
I0708 18:45:21.166255 HTTP-27 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigning request with: AWS4-HMAC-SHA256 Credential=ASIAVRUVPXURCNZP2EDS/20240708/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=4729ecad122b2912cde728d6a352c14e27bee8803e5d4fceb410e6908e4c6e6f
I0708 18:45:21.166290 HTTP-27 PluginsManager.cpp:161] (plugins) AWSClientRequest Successfully signed
I0708 18:45:21.166308 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to https://u4rad.s3.ap-south-1.amazonaws.com/01dbb2d3-859d-4116-8433-03345e6e0817.dcm
I0708 18:45:21.166317 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I0708 18:45:21.166323 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-invocation-id: ABF071D1-3AA4-4392-BB8C-8876828E0439
I0708 18:45:21.166328 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-request: attempt=1
I0708 18:45:21.166339 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientauthorization: AWS4-HMAC-SHA256 Credential=ASIAVRUVPXURCNZP2EDS/20240708/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=4729ecad122b2912cde728d6a352c14e27bee8803e5d4fceb410e6908e4c6e6f
I0708 18:45:21.166344 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-length: 91250
I0708 18:45:21.166349 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-md5: fZe7qJL6MY43u9KqgLXslg==
I0708 18:45:21.166354 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-type: binary/octet-stream
I0708 18:45:21.166359 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: u4rad.s3.ap-south-1.amazonaws.com
I0708 18:45:21.166364 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/6.8.0-1010-aws md/arch#x86_64 lang/c++#C++14 md/GCC#9.3.1 cfg/retry-mode#default api/S3
I0708 18:45:21.166369 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-content-sha256: UNSIGNED-PAYLOAD
I0708 18:45:21.166374 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-date: 20240708T184521Z
I0708 18:45:21.166382 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-security-token:
I0708 18:45:21.166390 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I0708 18:45:21.166395 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerNo current connections available in pool. Attempting to create new connections.
I0708 18:45:21.166401 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerattempting to grow pool size by 2
E0708 18:45:21.166409 HTTP-27 PluginsManager.cpp:153] CurlHandleContainerPool grown by 2
I0708 18:45:21.166415 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I0708 18:45:21.166420 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x7eaf7c097520
I0708 18:45:21.166424 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x7eaf7c097520
I0708 18:45:21.166665 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerNote: Http payloads are not being signed. signPayloads=0 http scheme=https
I0708 18:45:21.166719 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Header String: amz-sdk-invocation-id:36A1D8BA-8029-4D12-B807-72E845BAFA8E
amz-sdk-request:attempt=1
content-length:91250
content-md5:fZe7qJL6MY43u9KqgLXslg==
content-type:binary/octet-stream
host:u4rad.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240708T184521Z
x-amz-security-token:

I0708 18:45:21.166733 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigned Headers value:amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
I0708 18:45:21.166756 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Request String: PUT
/fa25cb98-187e-4718-bf38-175168403761.dcm

amz-sdk-invocation-id:36A1D8BA-8029-4D12-B807-72E845BAFA8E
amz-sdk-request:attempt=1
content-length:91250
content-md5:fZe7qJL6MY43u9KqgLXslg==
content-type:binary/octet-stream
host:u4rad.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240708T184521Z
x-amz-security-token:
amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
UNSIGNED-PAYLOAD
I0708 18:45:21.166802 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal String to sign: AWS4-HMAC-SHA256
20240708T184521Z
20240708/ap-south-1/s3/aws4_request
1292ff57229f159c361a828102a2934bdb55b1a48af3f8e0b949399ef4ff3de5
I0708 18:45:21.166815 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal computed signing hash: baca9e47f2a39218145e54f9333c4b9995732aefe2b1a90384678dbed649b22c
I0708 18:45:21.166827 HTTP-28 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigning request with: AWS4-HMAC-SHA256 Credential=ASIAVRUVPXURCNZP2EDS/20240708/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=baca9e47f2a39218145e54f9333c4b9995732aefe2b1a90384678dbed649b22c
I0708 18:45:21.166839 HTTP-28 PluginsManager.cpp:161] (plugins) AWSClientRequest Successfully signed
I0708 18:45:21.166850 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to https://u4rad.s3.ap-south-1.amazonaws.com/fa25cb98-187e-4718-bf38-175168403761.dcm
I0708 18:45:21.166858 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I0708 18:45:21.166864 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-invocation-id: 36A1D8BA-8029-4D12-B807-72E845BAFA8E
I0708 18:45:21.166869 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-request: attempt=1
I0708 18:45:21.166874 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientauthorization: AWS4-HMAC-SHA256 Credential=ASIAVRUVPXURCNZP2EDS/20240708/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=baca9e47f2a39218145e54f9333c4b9995732aefe2b1a90384678dbed649b22c
I0708 18:45:21.166880 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-length: 91250
I0708 18:45:21.166885 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-md5: fZe7qJL6MY43u9KqgLXslg==
I0708 18:45:21.166889 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-type: binary/octet-stream
I0708 18:45:21.166894 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: u4rad.s3.ap-south-1.amazonaws.com
I0708 18:45:21.166899 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/6.8.0-1010-aws md/arch#x86_64 lang/c++#C++14 md/GCC#9.3.1 cfg/retry-mode#default api/S3
I0708 18:45:21.166904 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-content-sha256: UNSIGNED-PAYLOAD
I0708 18:45:21.166909 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-date: 20240708T184521Z
I0708 18:45:21.166916 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-security-token:
HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I0708 18:45:21.166932 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I0708 18:45:21.166938 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x7eaf7c096170
I0708 18:45:21.166943 HTTP-28 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x7eaf7c096170
E0708 18:45:21.179007 HTTP-27 PluginsManager.cpp:153] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
I0708 18:45:21.179055 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerDestroy curl handle: 0x7eaf7c097520
I0708 18:45:21.179078 HTTP-27 PluginsManager.cpp:161] (plugins) CurlHandleContainerCreated replacement handle and released to pool: 0x7eaf7c0d3450
I0708 18:45:21.179093 HTTP-27 PluginsManager.cpp:161] (plugins) AWSClientRequest returned error. Attempting to generate appropriate error codes from response
E0708 18:45:21.179101 HTTP-27 PluginsManager.cpp:153] AWSXmlClientHTTP response code: -1

It seems like the endpoint to the bucket that gets resolved includes https so maybe that’s where the SSL cert error is coming from.

5

I have the same problem

Here is my detailed log after restart

0|Orthanc  | E0712 05:22:15.270178          DICOM-2 PluginsManager.cpp:154] ProcessCredentialsProviderFailed to find credential process's profile: default
0|Orthanc  | E0712 05:22:15.270351          DICOM-2 PluginsManager.cpp:154] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
0|Orthanc  | E0712 05:22:15.270376          DICOM-2 PluginsManager.cpp:154] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
0|Orthanc  | E0712 05:22:15.275935          DICOM-2 PluginsManager.cpp:154] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
0|Orthanc  | E0712 05:22:15.276407          DICOM-2 PluginsManager.cpp:154] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
0|Orthanc  | W0712 05:22:15.276435          DICOM-2 PluginsManager.cpp:158] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
0|Orthanc  | E0712 05:22:15.276454          DICOM-2 PluginsManager.cpp:154] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
0|Orthanc  | E0712 05:22:15.276574          DICOM-2 PluginsManager.cpp:154] CurlHandleContainerPool grown by 2
0|Orthanc  | E0712 05:22:15.458953          DICOM-2 PluginsManager.cpp:154] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:22:15.459084          DICOM-2 PluginsManager.cpp:154] AWSXmlClientHTTP response code: -1
0|Orthanc  | Resolved remote host IP address: 
0|Orthanc  | Request ID: 
0|Orthanc  | Exception name: 
0|Orthanc  | Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | 0 response headers:
0|Orthanc  | W0712 05:22:15.459126          DICOM-2 PluginsManager.cpp:158] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
0|Orthanc  | W0712 05:22:15.459142          DICOM-2 PluginsManager.cpp:158] AWSClientRequest failed, now waiting 0 ms before attempting again.
0|Orthanc  | E0712 05:22:15.466516          DICOM-2 PluginsManager.cpp:154] ProcessCredentialsProviderFailed to find credential process's profile: default
0|Orthanc  | E0712 05:22:15.466617          DICOM-2 PluginsManager.cpp:154] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
0|Orthanc  | E0712 05:22:15.466638          DICOM-2 PluginsManager.cpp:154] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
0|Orthanc  | E0712 05:22:15.474738          DICOM-2 PluginsManager.cpp:154] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
0|Orthanc  | E0712 05:22:15.474790          DICOM-2 PluginsManager.cpp:154] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
0|Orthanc  | W0712 05:22:15.474816          DICOM-2 PluginsManager.cpp:158] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
0|Orthanc  | E0712 05:22:15.474835          DICOM-2 PluginsManager.cpp:154] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
0|Orthanc  | E0712 05:22:15.655157          DICOM-2 PluginsManager.cpp:154] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:22:15.655283          DICOM-2 PluginsManager.cpp:154] AWSXmlClientHTTP response code: -1
0|Orthanc  | Resolved remote host IP address: 
0|Orthanc  | Request ID: 
0|Orthanc  | Exception name: 
0|Orthanc  | Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | 0 response headers:
0|Orthanc  | W0712 05:22:15.655326          DICOM-2 PluginsManager.cpp:158] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
0|Orthanc  | W0712 05:22:15.655342          DICOM-2 PluginsManager.cpp:158] AWSClientRequest failed, now waiting 50 ms before attempting again.

And when trying to send an image

0|Orthanc  | E0712 05:33:15.368401          DICOM-3 PluginsManager.cpp:154] ProcessCredentialsProviderFailed to find credential process's profile: default
0|Orthanc  | E0712 05:33:15.368511          DICOM-3 PluginsManager.cpp:154] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
0|Orthanc  | E0712 05:33:15.368531          DICOM-3 PluginsManager.cpp:154] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
0|Orthanc  | E0712 05:33:15.379016          DICOM-3 PluginsManager.cpp:154] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
0|Orthanc  | E0712 05:33:15.379075          DICOM-3 PluginsManager.cpp:154] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
0|Orthanc  | W0712 05:33:15.379099          DICOM-3 PluginsManager.cpp:158] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
0|Orthanc  | E0712 05:33:15.379313          DICOM-3 PluginsManager.cpp:154] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
0|Orthanc  | E0712 05:33:15.559569          DICOM-3 PluginsManager.cpp:154] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:33:15.559703          DICOM-3 PluginsManager.cpp:154] AWSXmlClientHTTP response code: -1
0|Orthanc  | Resolved remote host IP address: 
0|Orthanc  | Request ID: 
0|Orthanc  | Exception name: 
0|Orthanc  | Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | 0 response headers:
0|Orthanc  | W0712 05:33:15.559746          DICOM-3 PluginsManager.cpp:158] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
0|Orthanc  | E0712 05:33:15.559806          DICOM-3 PluginsManager.cpp:154] AWS S3 Storage (Primary: object-storage): error while creating object fdb2371d-7b56-4097-9fee-e239dd1ce5ec: error while writing file fdb2371d-7b56-4097-9fee-e239dd1ce5ec.dcm: response code = -1  curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:33:15.560009          DICOM-3 StoreScp.cpp:199] Exception while storing DICOM: Error in the plugin implementing a custom storage area

aws s3 ls command works, so bucket has permission and aws cli is correct
Double checked orthanc.json config and keys are ok

When reviewing IAM keys, the last use comes from the aws cli command, orthanc is not using the keys at any moment

My plugins are:

libModalityWorklists.so (1.12.4)  libOrthancAWSS3.so (2.4.0) libOrthancPython.so (4.0)  libServeFolders.so (1.12.4)

and orthanc version is 1.12.4 with S3 Plugin config:

"AwsS3Storage" : {
    "BucketName" : "test-pacs",
    "Region" : "us-east-2",
    "AccessKeyId" : "",
    "SecretAccessKey" : "",
    "MigrationFromFileSystemEnabled": true,
    "StorageStructure": "flat",
    "Path" : "orthanc-storage/",
    "SslVerifyPeer" : false,
    "EnableAwsSdkLogs": true
  },
6

Updated to mainline version and logs are:

0|Orthanc  | E0712 05:44:42.747006          DICOM-0 PluginsManager.cpp:154] ProcessCredentialsProviderFailed to find credential process's profile: default
0|Orthanc  | E0712 05:44:42.747111          DICOM-0 PluginsManager.cpp:154] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
0|Orthanc  | E0712 05:44:42.747131          DICOM-0 PluginsManager.cpp:154] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
0|Orthanc  | E0712 05:44:42.750202          DICOM-0 PluginsManager.cpp:154] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
0|Orthanc  | E0712 05:44:42.750255          DICOM-0 PluginsManager.cpp:154] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
0|Orthanc  | W0712 05:44:42.750276          DICOM-0 PluginsManager.cpp:158] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
0|Orthanc  | E0712 05:44:42.750295          DICOM-0 PluginsManager.cpp:154] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
0|Orthanc  | E0712 05:44:42.930645          DICOM-0 PluginsManager.cpp:154] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:44:42.930770          DICOM-0 PluginsManager.cpp:154] AWSXmlClientHTTP response code: -1
0|Orthanc  | Resolved remote host IP address: 
0|Orthanc  | Request ID: 
0|Orthanc  | Exception name: 
0|Orthanc  | Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | 0 response headers:
0|Orthanc  | W0712 05:44:42.930810          DICOM-0 PluginsManager.cpp:158] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
0|Orthanc  | W0712 05:44:42.930826          DICOM-0 PluginsManager.cpp:158] AWSClientRequest failed, now waiting 12800 ms before attempting again.
0|Orthanc  | E0712 05:44:55.739910          DICOM-0 PluginsManager.cpp:154] ProcessCredentialsProviderFailed to find credential process's profile: default
0|Orthanc  | E0712 05:44:55.740009          DICOM-0 PluginsManager.cpp:154] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
0|Orthanc  | E0712 05:44:55.740030          DICOM-0 PluginsManager.cpp:154] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
0|Orthanc  | E0712 05:44:55.742210          DICOM-0 PluginsManager.cpp:154] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
0|Orthanc  | E0712 05:44:55.742258          DICOM-0 PluginsManager.cpp:154] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
0|Orthanc  | W0712 05:44:55.742280          DICOM-0 PluginsManager.cpp:158] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
0|Orthanc  | E0712 05:44:55.742299          DICOM-0 PluginsManager.cpp:154] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
0|Orthanc  | E0712 05:44:55.914507          DICOM-0 PluginsManager.cpp:154] CurlHttpClientCurl returned error code 60 - SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:44:55.914635          DICOM-0 PluginsManager.cpp:154] AWSXmlClientHTTP response code: -1
0|Orthanc  | Resolved remote host IP address: 
0|Orthanc  | Request ID: 
0|Orthanc  | Exception name: 
0|Orthanc  | Error message: curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | 0 response headers:
0|Orthanc  | W0712 05:44:55.914678          DICOM-0 PluginsManager.cpp:158] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
0|Orthanc  | E0712 05:44:55.914788          DICOM-0 PluginsManager.cpp:154] AWS S3 Storage (Primary: object-storage): error while creating object b96dff65-edee-4593-97b1-c2bb956be95f: error while writing file b96dff65-edee-4593-97b1-c2bb956be95f.dcm: response code = -1  curlCode: 60, SSL peer certificate or SSH remote key was not OK
0|Orthanc  | E0712 05:44:55.915092          DICOM-0 StoreScp.cpp:199] Exception while storing DICOM: Error in the plugin implementing a custom storage area
7

Hello,

Your system probably uses an outdated set of certificates (cf. for instance this related issue found on Internet). Make sure that your chain of certificates is up-to-date.

HTH,
Sébastien-

8

Thank you, Sébastien!

I made some other tests today:

Installed a fresh Ubuntu instance
Downloaded the latest Orthanc
Set the config file as follows:

{
  "Plugins" : [
    "/usr/share/orthanc/plugins/libOrthancAwsS3Storage.so"
  ],
  "AwsS3Storage" : {
    "BucketName" : "test-pacs",
    "Region" : "us-east-2",
    "AccessKeyId" : "AKIA4MTWJCTPVF44EWJG",
    "SecretAccessKey" : "secretKey"  }
}

Updated and verified my certificates:

Certificate chain
 0 s:CN = s3.amazonaws.com
   i:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 00:00:00 2024 GMT; NotAfter: May  2 23:59:59 2025 GMT
 1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   i:C = US, O = Amazon, CN = Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=CN = s3.amazonaws.com
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6113 bytes and written 382 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

and had the same SSL error code

Then I downloaded orthancteam/orthanc docker image and set the config as follows:

{
  "RegisteredUsers" : {
    "orthanc1" : "orthanc"
  },
  "Plugins" : [
    "/usr/share/orthanc/plugins/libOrthancAwsS3Storage.so"
  ],
  "AwsS3Storage" : {
    "BucketName" : "test-pacs",
    "Region" : "us-east-2",
    "AccessKeyId" : "AKIA4MTWJCTPVF44EWJG",
    "SecretKey" : "secretKey",
    "Verbose" : true,
    "EnableAwsSdkLogs": true
  },
  "HttpVerbose" : true,
  "RestApiWriteToFileSystemEnabled": true,
  "LogLevel" : "DEBUG"
}

And got the following as a result:

E0713 01:50:14.098069          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] Aws::Config::AWSConfigFileProfileConfigLoaderUnable to open config file /root/.aws/credentials for reading.
E0713 01:50:14.098860          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
E0713 01:50:14.099260          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] ProcessCredentialsProviderFailed to find credential process's profile: default
E0713 01:50:14.099713          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] SSOCredentialsProviderUnable to open token file on path: /root/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
E0713 01:50:14.100119          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] InstanceProfileCredentialsProviderCredentials have expired attempting to re-pull from EC2 Metadata Service.
E0713 01:50:14.105360          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] EC2MetadataClientHttp request to retrieve credentials failed with error code 404
E0713 01:50:14.105848          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] EC2MetadataClientCan not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
W0713 01:50:14.106270          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:460] EC2MetadataClientCalling EC2Metadataservice to get profiles failed
E0713 01:50:14.106707          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] Aws::Config::AWSProfileConfigLoaderBaseFailed to reload configuration.
W0713 01:50:14.166624          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:460] AWSErrorMarshallerEncountered AWSError 'AccessDenied': Access Denied
E0713 01:50:14.167248          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:464] AWSXmlClientHTTP response code: 403
Resolved remote host IP address: 52.219.101.186
Request ID: R1XC60E2M0M94C1V
Exception name: AccessDenied
Error message: Access Denied
7 response headers:
connection : close
content-type : application/xml
date : Sat, 13 Jul 2024 01:50:14 GMT
server : AmazonS3
transfer-encoding : chunked
x-amz-id-2 : hErX9DFxRnwoN1vPq+YvJ+WUnUKvxjkQJQWDbizBtlSBF27Pt0zMrK8n7mCrDkyJwKoLw0CaX/E=
x-amz-request-id : R1XC60E2M0M94C1V
W0713 01:50:14.169282          DICOM-0 AWS S3 Storage:/AwsS3StoragePlugin.cpp:460] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
E0713 01:50:14.169777          DICOM-0 AWS S3 Storage:/StoragePlugin.cpp:139] AWS S3 Storage: error while creating object 193eb390-ca54-4416-9f12-25e587009676: error while writing file 193eb390-ca54-4416-9f12-25e587009676.dcm: response code = 403 AccessDenied Access Denied
E0713 01:50:14.170452          DICOM-0 StoreScp.cpp:199] Exception while storing DICOM: Error in the plugin implementing a custom storage area

From what I see, the plugin is looking for an SSO token in my system. I did set the AWS CLI as well, and did list and upload objects to my bucket using the same keys I put in the config file.

I hope these logs are useful in finding out what’s going on
Otherwise, if you have any steps to follow to get Orthanc running with S3 plugin, I don’t have any problem making a new Orthanc server instance and building on top of it

I appreciate your time and help!

Emanuel

9

So, if I understand correctly, you are to read and write to the bucket with AWS tools when running inside the container?

What you can do, if you’re using SSO, to grab a temporary access key from the AWS portal and use them in the plugin configuration.

Another option you might want to try is to not set anything in the configuration file and only set these values as environment variables?

AWS_ACCESS_KEY_ID=xxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxx
AWS_SESSION_TOKEN=xxxxxxx

(edit: don’t forget to set the region, too!)

Just a few random ideas to try.

I had some trouble when working with containers running in AWS until we found out that we had to share the .aws/ folder between the host and the container for this to work (because, in our case, we could not used access keys)

10

Hi Klavierma,
If the 403 indicates insufficient permission, it is quite possible that the process running inside of the docker container isn’t able to grab the role credential. On EC2 instance, the instance metadata service (IMDS) is the provider of role credential to:
a. process running with AWS SDK on the operating system of the EC2 instance
b. process running in container using AWS SDK; with the container runtime (Docker daemon) running on the OS of the EC2 instance

In your case, a. has never been a problem, so the role does have permission; b. is very likely the problem.

To verify if b is the problem, try to manually grab role credential, by getting into the container shell and run some commands to grab the role credential yourself. You can use the command from this blog post (starting with TOKEN) to check.

To make things complicated. There are two versions of IMDS, and since 2020 the recommendation is to go with IMDSv2. IMDSv2 is more restrictive in terms of network hops per request. If your docker network is in bridge mode then it does take an additional hop to get instance metadata.

So I suspect:

  1. your docker network in bridge mode
  2. your IMDS is on v2
  3. you didn’t set hop limit correctly
    and that’s why you only have 403 when making call from container.

I recommend taking a look at the orthweb project with a reference implementation of orthanc in a docker on EC2. Relevant configurations are made here.

1 Like
11

Invaluable information, @digihunch , thank you very much!

1 Like
12

Thank you both, @benjamin.golinvaux and @digihunch!

As @digihunch outlined, following the blog entry he posted and after some digging, I sorted it out.

The solution was to assign the EC2 instance an IAM role (it had none), as the SDK looks to ignore the CLI tokens. That role should have access to modify the S3 bucket storing Orthanc information.

Appreciate the help.

(edit: I removed the access key id and secret key from orthanc config and it still works, so there is no need to add them)

Cheers,
Emanuel

2 Likes