AWS S3 connecting issue from running Orthanc on EC2

General
1

Hello there,

I’ve already set other EC2 servers which run dockerized Orthanc service with S3 plugin.
I created a template which is used for new instances setup, and I have never had problems with this configuration.

Now, I’m in trouble with a new instance (set like any other).

My docker-compose is like:

orthanc:
        image: osimis/orthanc:22.12.2 (tested also osimis/orthanc:23.11.1) 
        container_name: name
        restart: always
        ports:
            - 4242:4242
            - 8042:8042
        volumes:
            - /storage:/storage
        environment:
            - VERBOSE_STARTUP=true
            - AUTHORIZATION_PLUGIN_ENABLED=true
            - OSIMIS_WEB_VIEWER1_PLUGIN_ENABLED=true
            - POSTGRESQL_PLUGIN_ENABLED=true
        secrets:
            - orthanc.secret.json
        depends_on:
            - orthanc-db
        links:
            - orthanc-db
        networks:
            - orthanc_net

with also psql orthanc-db service.

My orthanc.json contains as usual the following:

"AwsS3Storage" : {
        "BucketName": "name",
        "Region": "eu-central-1",
        "AccessKey": "***",
        "SecretKey": "***",
        "HybridMode": "WriteToObjectStorage",
        "StorageStructure": "legacy",
       "RootPath": "root"
      },

Using AWS CLI with this IAM user, I can work on this bucket and root folder.
But in orthanc seems I cannot write to that path. Error is:

E1129 14:17:16.230035 PluginsManager.cpp:153] AWS S3 Storage (Primary: object-storage): error while creating object *****: error while writing file root/**/**/*****: response code = -1 curlCode: 7, Couldn't connect to server

I can’t understand why, especially because I have never had this issue in other replicas and aws cli works. Obviously I’m looking for a solution also at cloud architecture side.

Hope someone can help me

Thank you so much

Jacopo

2

Hi Jacopo,

FYI, if you use the latest (unreleased version) of the S3 plugin available in osimis/orthanc:mainline-2023.11.29, there is a new configuration option "EnableAwsSdkLogs" that might show you more information from the S3 client that is used inside Orthanc.

HTH,

Alain.

1 Like
3

Hi Alain,

Thank you for reply.

So, I’m checking logs. I think authentication step is going well, but then HttpClient cannot connect to server. I’m still investigating.

Here piece of log (attempt 1 of 11, all of those failed).

I1130 10:19:51.015208 HttpServer.cpp:1262] (http) POST /instances
I1130 10:19:53.098863 OrthancRestApi.cpp:163] (http) Receiving a DICOM file of 2.85MB through HTTP
I1130 10:19:53.109165 PluginsManager.cpp:161] (plugins) AWS S3 Storage (Primary: object-storage): creating attachment 4b66e793-6c02-40f2-89ea-9548fe09aeb6 of type 1
I1130 10:19:53.117968 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Region = eu-central-1
I1130 10:19:53.118057 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseFIPS = 0
I1130 10:19:53.118066 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseDualStack = 0
I1130 10:19:53.118074 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: UseArnRegion = 0
I1130 10:19:53.118082 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
I1130 10:19:53.118087 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint str eval parameter: Bucket = ***
I1130 10:19:53.118160 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules engine evaluated the endpoint: https://***.s3.eu-central-1.amazonaws.com
I1130 10:19:53.118173 PluginsManager.cpp:161] (plugins) Aws::Endpoint::DefaultEndpointProviderEndpoint rules evaluated props: {"authSchemes":[{"disableDoubleEncoding":true,"name":"sigv4","signingName":"s3","signingRegion":"eu-central-1"}]}
I1130 10:19:53.118228 PluginsManager.cpp:161] (plugins) AWSClientFound body, but content-length has not been set, attempting to compute content-length
I1130 10:19:53.124172 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerNote: Http payloads are not being signed. signPayloads=0 http scheme=https
I1130 10:19:53.124284 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Header String: amz-sdk-invocation-id:1E319989-9C31-4BA1-8EE7-****
amz-sdk-request:attempt=1
content-length:2987544
content-md5:/BUVOmCnXNHLDUkjJVb1+A==
content-type:binary/octet-stream
host:***.s3.eu-central-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20231130T101953Z

I1130 10:19:53.124298 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigned Headers value:amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date
I1130 10:19:53.124317 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerCanonical Request String: PUT
/***/**/**/4b66e793-6c02-40f2-89ea-9548fe09aeb6

amz-sdk-invocation-id:1E319989-9C31-4BA1-8EE7-****
amz-sdk-request:attempt=1
content-length:2987544
content-md5:/BUVOmCnXNHLDUkjJVb1+A==
content-type:binary/octet-stream
host:***.s3.eu-central-1.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20231130T101953Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
I1130 10:19:53.124354 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal String to sign: AWS4-HMAC-SHA256
20231130T101953Z
20231130/eu-central-1/s3/aws4_request
42e50e4a67b249bb101b6119b920ca77b1f8e2cff1d8eb428314833c7375d5e9
I1130 10:19:53.124367 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerFinal computed signing hash: 00a9b1503db08c712edb267b1a00745c9cbc3d1e91624bf7871d6c83416315bf
I1130 10:19:53.124377 PluginsManager.cpp:161] (plugins) AWSAuthV4SignerSigning request with: AWS4-HMAC-SHA256 Credential=***/20231130/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=00a9b1503db08c712edb267b1a00745c9cbc3d1e91624bf7871d6c83416315bf
I1130 10:19:53.124389 PluginsManager.cpp:161] (plugins) AWSClientRequest Successfully signed
I1130 10:19:53.124409 PluginsManager.cpp:161] (plugins) CurlHttpClientMaking request to https://***.s3.eu-central-1.amazonaws.com/***/**/**/4b66e793-6c02-40f2-89ea-9548fe09aeb6
I1130 10:19:53.124421 PluginsManager.cpp:161] (plugins) CurlHttpClientIncluding headers:
I1130 10:19:53.124430 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-invocation-id: 1E319989-9C31-4BA1-8EE7-***
I1130 10:19:53.124438 PluginsManager.cpp:161] (plugins) CurlHttpClientamz-sdk-request: attempt=1
I1130 10:19:53.124446 PluginsManager.cpp:161] (plugins) CurlHttpClientauthorization: AWS4-HMAC-SHA256 Credential=***/20231130/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=00a9b1503db08c712edb267b1a00745c9cbc3d1e91624bf7871d6c83416315bf
I1130 10:19:53.124454 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-length: 2987544
I1130 10:19:53.124462 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-md5: /BUVOmCnXNHLDUkjJVb1+A==
I1130 10:19:53.124470 PluginsManager.cpp:161] (plugins) CurlHttpClientcontent-type: binary/octet-stream
I1130 10:19:53.124478 PluginsManager.cpp:161] (plugins) CurlHttpClienthost: ***.s3.eu-central-1.amazonaws.com
I1130 10:19:53.124486 PluginsManager.cpp:161] (plugins) CurlHttpClientuser-agent: aws-sdk-cpp/1.11.178 ua/2.0 md/aws-crt# os/Linux/5.15.0-1045-aws md/arch#x86_64 lang/c++#C++14 md/GCC#10.2.1 cfg/retry-mode#default api/S3
I1130 10:19:53.124495 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-content-sha256: UNSIGNED-PAYLOAD
I1130 10:19:53.124503 PluginsManager.cpp:161] (plugins) CurlHttpClientx-amz-date: 20231130T101953Z
I1130 10:19:53.124514 PluginsManager.cpp:161] (plugins) CurlHandleContainerAttempting to acquire curl connection.
I1130 10:19:53.124523 PluginsManager.cpp:161] (plugins) CurlHandleContainerConnection has been released. Continuing.
I1130 10:19:53.124531 PluginsManager.cpp:161] (plugins) CurlHandleContainerReturning connection handle 0x7f9e54327dd0
I1130 10:19:53.124540 PluginsManager.cpp:161] (plugins) CurlHttpClientObtained connection handle 0x7f9e54327dd0
E1130 10:19:53.130502 PluginsManager.cpp:153] CurlHttpClientCurl returned error code 7 - Couldn't connect to server
I1130 10:19:53.130619 PluginsManager.cpp:161] (plugins) CurlHandleContainerDestroy curl handle: 0x7f9e54327dd0
I1130 10:19:53.130711 PluginsManager.cpp:161] (plugins) CurlHandleContainerCreated replacement handle and released to pool: 0x7f9e5c6007b0
I1130 10:19:53.130973 PluginsManager.cpp:161] (plugins) AWSClientRequest returned error. Attempting to generate appropriate error codes from response
E1130 10:19:53.131072 PluginsManager.cpp:153] AWSXmlClientHTTP response code: -1
Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: curlCode: 7, Couldn't connect to server
0 response headers:
W1130 10:19:53.131329 PluginsManager.cpp:157] AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
I1130 10:19:53.131507 PluginsManager.cpp:161] (plugins) AWSClientDate header was not found in the response, can't attempt to detect clock skew
W1130 10:19:53.131530 PluginsManager.cpp:157] AWSClientRequest failed, now waiting 0 ms before attempting again.

[... other attempts]

E1130 10:20:18.800877 PluginsManager.cpp:153] AWS S3 Storage (Primary: object-storage): error while creating object 4b66e793-6c02-40f2-89ea-9548fe09aeb6: error while writing file ***/**/**/4b66e793-6c02-40f2-89ea-9548fe09aeb6: response code = -1  curlCode: 7, Couldn't connect to server

I can share also user policy.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AllowGroupToSeeBucketListInTheConsole",
			"Action": [
				"s3:ListAllMyBuckets",
				"s3:GetBucketLocation"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::*"
			]
		},
		{
			"Sid": "AllowRootAndHomeListingOfCompanyBucket",
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::bucket-name"
			]
		},
		{
			"Sid": "AllowListingOfUserFolder",
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::bucket-name",
				"arn:aws:s3:::bucket-name/root/*"
			]
		},
		{
			"Sid": "AllowAllS3ActionsInUserFolder",
			"Action": [
				"s3:PutObject",
				"s3:GetObject",
				"s3:DeleteObject"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::bucketname/root/*"
			]
		}
	]
}

J.

4

Hello there,

Unfortunately I haven’t found the cause yet.
Does anyone have any suggestions?

Thank you

J.

5

Do you still have a problem? It must be on the AWS side with permissions.
Logs should show the exact issues better.

6

Thank you so much for reply.

Also in my opinion it is AWS side problem.
In last days I’m looking for possible mismatchings of configuration, but I found no problems in users, roles, permissions and networking.
Then, I tested S3 communication from instance using AWS CLI with same user and all worked well.

The only detail I noted is an IAM role associated directly to EC2 instance, so I promptly removed it and rebooted the instance, but problem has persisted: “Could not connect to server. AWSClientIf the signature check failed. This could be because of a time skew. Attempting to adjust the signer”.

My last attempt has been to reproduce same instance (without IAMRole, usign snaphot and template) and Orthanc-S3 communication works now.

Definitely, I think that IAM role associated to instance was troubled, but removing it didn’t work for some reasons I don’t know.

J.

7

Hello there,

I figured out the problem.

I was looking for Orthanc, S3 plugin, EC2 or S3 problem directly, bu the point was with HTTPS communication: “HttpClient cannot connect to server” means that S3 rejects the study sending I was attempting because of unsecure connection.

Putting a reverse-proxy over Orthanc resolved the problem.

I’ve not found some accurate documentations about this yet, but I’m pretty sure this was the issue.

Hope it can help someone

Regards

Jacopo