AWS S3 Storage: error while decrypting object

Google Groups archive
1

Hello all!

Upon setting up the client side encryption in the orthanc config file for a AwsS3Storage (MinIO), I’m faced with an issue which I have’nt found a solution to. On the first time running docker-compose up, everything is fine. However, when we shut down the containers and then restart them, a weird issue occurs: we are unable to decrypt the dicom files stored in MinIO. We are faced with the following error:

E0817 23:10:09.643131 PluginsManager.cpp:153] AWS S3 Storage: error while decrypting object 025d4a82-cf7f-4692-943d-8bcdb5264bb3: HashVerificationFilter: message hash or MAC not valid

From googling the issue, it seems that the hash is different between docker-compose restart which disable file reading.

Any suggestion on how to fix this issue?

Thanks, Denis

PS: The relevant files for our project are uploaded here

docker-compose.yml (1.33 KB)

orthanc.json (1.46 KB)

2

Hi Denis,

The docker-compose is incomplete and we are missing test files. Could you zip a full setup such that we don’t have to spend too much time rebuilding the missing parts ?

Alain

3

Here is a full working example where we experiment the issue. The issue occurs on CentOS Linux release 7.9.2009 (Core) kernel vesion 3.10.0-1160.62.1.el7.x86_64. To reproduce:

  1. Launch the docker containers with the make command at the root of the folder.
  2. Go to localhost:8040 and query a study from the remote modality.
  3. Terminate the docker containers by issuing the make clean command at the root of the folder.
  4. Relaunch the docker containers with the make command.

There is now 2 way to observe the bug:
5.1) Go back to localhost:8040 and select the previously queried instance and try to access it. The ‘docker-compose logs -f orthanc’ will tell you the following: AWS S3 Storage: error while decrypting object 025d4a82-cf7f-4692-943d-8bcdb5264bb3: HashVerificationFilter: message hash or MAC not valid
5.2) You can also observe the bug by making a get request (ie using postman):
GET http://localhost:8040/instances/2be137cb-f89054c3-a7e8d270-3dc2f38d-937491f/file
You will then be faced with a 500 error with an error message telling you that the error commes from the plugin implementation. The returned orthanc status is 32.

I have zipped the whole project file.

PS: The issue is intermitent. ie, the issue appears on the second restart but might go away on the third or fourth restart for unknown reasons. It can also only start occuring after the third, fourht or even fifth restart.

Thanks, Denis

OrthancTest.zip (2.7 KB)

4

Hi Denis,

I confirm I’ve been able to reproduce the issue with your docker setup running on Ubuntu 20.4. This seems to occur at every run.

Right now, I have not been able to reproduce it with Orthanc running natively which means I can not debug it easily.

I’ll continue to investigate but I’m off next week so I can not commit on any date for a fix.

Best regards,

Alain.

5

Hi again Denis,

Some feedback on my findings. I’ve rebuilt multiple Docker images adding more and more logs. It appears that the MasterKey that is read is changing between the first start and the second one. This is very weird since I don’t see any difference when dumping the content of the file before starting Orthanc:

  • BEFORE_ORTHANC_STARTUP_SCRIPT=cat /var/lib/orthanc/encryption/test.key

At this stage, I’m highly suspecting the LineFeed character at the end of your MasterKey to trigger the issue. This character is taken into account and considered as being part of the key. At some point, it seems that some docker mapping adds/removes/transforms this character but I have no idea how this could happen.

I’ve made a few tests with your key without this LF at the end it seems to work so I would advise you to consider this as a workaround.

Please tell me if that works for you …

Best regards,

Alain.

6

Hi Alain,

Sadly the issue is still occurring even with this workaround. I wish you a good time off for next week and I hope we can get that resolved when you get back!

The error for reference for future readers:
{
“HttpError” : “Internal Server Error”,
“HttpStatus” : 500,
“Message” : “Error in the plugin implementing a custom storage area”,
“Method” : “GET”,
“OrthancError” : “Error in the plugin implementing a custom storage area”,
“OrthancStatus” : 32,
“Uri” : “/instances/caf52782-28ca0888-75e6fb46-5b707591-5092aaa7/file”
}

Cheers, Denis

7

Hi Denis,

I have produced 2 test images for you:

image: osimis/orthanc:s3test-b64-decoder

and

image: osimis/orthanc:s3test-hex-decoder

The ‘hex’ one is identical to the current one with more logs. The ‘b64’ has a fix and logs.

Both these images display the master key at startup:
orthanc_1 | I0830 08:05:49.722095 PluginsManager.cpp:161] (plugins) AWS S3 storage initialized
orthanc_1 | ReadKey CFACC2B6A12FDFA086799352C4C5EFBF0529C315F82AFC84BB6C769CE777E966

With the ‘hex’ version, the master key may have 2 values while the “b64” version always display the same value (at least on my machine)

Only change in the code is here:

void EncryptionHelpers::ReadKey(CryptoPP::SecByteBlock& key, const std::string& path)
{
try
{
FileSource fs(path.c_str(), true,
// new HexDecoder(
new Base64Decoder(
new ArraySink(key.begin(), key.size())
)
);

std::cout << "ReadKey " << ToHexString(key) << std::endl;

The documentation of the s3 plugin states that you should feed it with a master key encoded in base64 so, the current code with the HexDecoder is clearly wrong. According to the Crypto++ doc, the HexDecoder should ignore any character that is not Hex → that is bad wrt the key entropy of course but that should make it deterministic (which does not seem to happen).

I have also converted the key from base64 to hex with this command:
cat test.key | base64 -d -i | hexdump -v -e '/1 “%02x” ’ >> test.hex.key

And it seems that the output is deterministic even with the “hex” image (which is expected of course !)

Could you give it a try on your side ?

Then, we’ll decide if we upgrade the documentation or the code but, anyway, that will trigger a breaking change. I don’t think that there are many users of this client-side encryption feature and I would favor b64 keys (generating hex keys is a bit tricky)

Best regards,

Alain.

8

I will definitely take a look at that today! Many thanks for your help!

Will get back to you today, thanks again!

Denis

9

Wow what a catch! It is in fact non deterministic with the s3test-hex-decoder image (Which is expected from your findings)!

The s3test-b64-decoder image also gives me the same key value on each run, which is amazing.

Finally, converting the key to hex and using the unfixed image also works on my end.

I confirm all of your findings on my system!

Until an official fix is released, we will be using the image with the fixed b64 decoder.

Many thanks again, Denis

10

Hi,

I’ve just released version 2.0.0 of the object-storage plugins with this fix; https://hg.orthanc-server.com/orthanc-object-storage/rev/ac596874d997

However, I’m waiting for Orthanc 1.11.2 to be ready in order to make a new Docker image. Should come soon.

Best regards,

Alain