Transfer Accelerator Issues

Good Day All,
I am trying to test out Transfer Accelerator but running into issues. This is most commonly with CT/MG studies where they will fail to transfer using the accelerator. (Continual retries it will eventually work). Below are the verbose logs, if there is any input. Interestingly it gives an error, does not complete, then says the job has completed with succes though checking job status that is now the case:

Job Status

{
“CompletionTime”: “20190513T155610.789644”,
“Content”: {
“CompletedHttpQueries”: 4,
“Compression”: “gzip”,
“NetworkSpeedKBs”: 1105,
“Peer”: “TRANSFER”,
“TotalInstances”: 404,
“TotalSizeMB”: 203,
“UploadedSizeMB”: 7
},
“CreationTime”: “20190513T155130.240532”,
“EffectiveRuntime”: 154.74199999999999,
“ErrorCode”: 1,
“ErrorDescription”: “Error encountered within the plugin engine”,
“ID”: “6d6aa73f-00d0-48fd-92cd-6dca51da252e”,
“Priority”: 1,
“Progress”: 9,
“State”: “Failure”,
“Timestamp”: “20190513T155634.603025”,
“Type”: “PushTransfer”
}

Verbose Logs:

I0513 11:51:57.405093 JobsEngine.cpp:144] Executing job with priority 1 in worker thread 1: 6d6aa73f-00d0-48fd-92cd-6dca51da252e
I0513 11:51:57.406093 OrthancPlugins.cpp:1603] Plugin making REST GET call on URI /studies/5fc27dca-d90d91a9-20193bc0-30ca3d2f-cda7e86f/instances (built-in API)
I0513 11:51:57.919144 HttpClient.cpp:671] HTTP status code 200 after POST request on: https://transfer.servername.com:443//transfers/push
I0513 11:52:03.486442 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/0
I0513 11:52:03.815461 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/3
I0513 11:52:04.200483 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/2
I0513 11:52:04.438497 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/1
I0513 11:52:09.235794 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/6
I0513 11:52:09.855808 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/7
I0513 11:52:10.093820 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/4
I0513 11:52:10.340835 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/5
I0513 11:52:14.453073 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/8
I0513 11:52:15.234128 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/10
I0513 11:52:15.578137 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/11
I0513 11:52:16.181185 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/9
I0513 11:52:19.358366 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/12
I0513 11:52:20.288421 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/13
I0513 11:52:20.677428 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/14
I0513 11:52:23.580608 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/15
I0513 11:52:25.199700 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/18
I0513 11:52:25.785722 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/16
I0513 11:52:26.252749 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/17
I0513 11:52:28.344881 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/21
I0513 11:52:29.729961 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/22
I0513 11:52:31.789078 HttpClient.cpp:671] HTTP status code 404 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/19
E0513 11:52:31.790067 HttpClient.cpp:700] Error in HTTP request, received HTTP status 404 (Not Found)
I0513 11:52:31.790067 PluginsManager.cpp:172] Reached the maximum number of retries for a HTTP query
I0513 11:52:33.580182 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/20
I0513 11:52:33.890190 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/23
I0513 11:52:34.770239 HttpClient.cpp:671] HTTP status code 200 after PUT request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7/24
I0513 11:52:35.157274 HttpClient.cpp:671] HTTP status code 200 after DELETE request on: https://transfer.servername.com:443//transfers/push/fc36433f-5ed2-4b1e-ac67-684c215fa6c7
I0513 11:52:35.157274 JobsRegistry.cpp:486] Job has completed with success: 6d6aa73f-00d0-48fd-92cd-6dca51da252e

Some additional info. Again showing Success then failure:

  • Excess found in a non pipelined read: excess = 2 url = //transfers/push/61c76084-ba59-4413-a7d3-df0018767412 (zero-length body)
  • Connection #0 to host transfer.realtimeradiology.com left intact
    I0513 15:17:59.295997 HttpClient.cpp:671] HTTP status code 200 after DELETE request on: https://transfer.server.com:443//transfers/push/61c76084-ba59-4413-a7d3-df0018767412
    I0513 15:17:59.295997 HttpClient.cpp:676] cURL status code: 0
    I0513 15:17:59.296997 JobsRegistry.cpp:486] Job has completed with success: db35aefb-1078-4fcf-9767-73ac155daf89
    W0513 15:17:59.296997 LuaContext.cpp:103] Lua says: Job failure. Submitting job to Retry Service db35aefb-1078-4fcf-9767-73ac155daf89

Hello,

It would be interesting to look at the verbose logs of the receiving Orthanc server.

The following line could indicate that another transfer in push mode has been initiated, and that because of an insufficient value of option “MaxPushTransactions”, the push transfer “fc36433f-5ed2-4b1e-ac67-684c215fa6c7” in receiving Orthanc was discarded:

E0513 11:52:31.790067 HttpClient.cpp:700] Error in HTTP request, received HTTP status 404 (Not Found)

Regarding the misleading message “Job has completed with success”, this was a typo for which a fix has just been submitted in the Orthanc mainline:
https://bitbucket.org/sjodogne/orthanc/commits/fe73717105b6788409254974a37f0c52fb035034

HTH,
Sébastien-

Thanks, I have increased the MaxPushTransactions on the receiver to 50 (It is a 24 core, 32Gb Ram server) and still have the error. I have an alert when a job fails, to retry the job and it will go through, but is very inconsistent. Peer works with no issues and no other server is sending via Transfer Accelerator (Many via Peer)
I will see if I can enable logging on the receiving Orthanc server.

Checking the receiving log is really a must.

On another hand, does the issue also occur if using the “pull mode” instead of the “push mode”? This could indicate that some middleman router dislikes HTTP PUT requests, as used by the plugin.
http://book.orthanc-server.com/plugins/transfers.html#sending-in-pull-vs-push-mode

You could also try and run both the Orthanc sender and the Orthanc receiver on the same computer, in order to check whether the issue disappears in the absence of network transmissions.

In either case, for us to provide further help, we would need a robust way to reproduce your issue independently of your setup. Would it be possible to provide us with sample DICOM data, and a reproducible scenario?

Thanks. I have tested direct and it looks as though it may be an issue with the Apache Proxy in the middle. With HTTP verbose on I am getting
HTTP_BAD_GATEWAY.html.var
I will take a peek and see what I can understand what may be the issue.

Great! Please keep us updated: This information might be very useful to other users, e.g.:
https://groups.google.com/d/msg/orthanc-users/Hz0jtgPbgss/KPW1b1wgAgAJ