Orthanc 1.2.0 on sqlite to postgres

Hello,
My orthanc server with 500 GB of data (~600 000 instances) recently became quite slow with 1 GB sqlite (only when accepting new files - CPU load was 100%)
I decided to migrate to postgres database with without postgres storage.
What's the recommended way?
I decided to setup new empty orthanc with postgres use ImportDicomFile.py on old database folder, but it is surprisingly slow. (I expect it to finish in 24 hours on my hardware).
Maybe it would be faster to use Replicate.py? Or maybe there is a way to profit from --upgrade parameter?
Although I do not except to do it once again I might still profit somehow ; )

Hello,
My orthanc server with 500 GB of data (~600 000 instances) recently became
quite slow with 1 GB sqlite (only when accepting new files - CPU load was
100%)
I decided to migrate to postgres database with without postgres storage.
What's the recommended way?

Running the Replicate.py script is the recommended way.

I decided to setup new empty orthanc with postgres use ImportDicomFile.py
on old database folder, but it is surprisingly slow. (I expect it to finish
in 24 hours on my hardware).

Are your files transferred over the network or are your running your script
on localhost ?
If you're on Windows, make sure to use 127.0.0.1 for the Orthanc address
instead of localhost (there are sometimes some DNS issues on windows where
"localhost" takes one second to resolve)

Can you eventually paste part of your logs (Orthanc and scripts) ?

Maybe it would be faster to use Replicate.py? Or maybe there is a way to

profit from --upgrade parameter?

Replicate.py is probably a bit more appropriate since ImportDicomFile.py
will probably also update all json files that are in the old database
folder while Replicate.py will use the REST API and transmit only relevant
files. However, that will not make a big speed difference.

I’ve already finished the migration using ImportDicomFile.py with 127.0.0.1 (on Debian). It took around 30 hours to make full copy. Moreover, read and write was on the same hard disk location (some kind of RAID).

I don’t actually know where the bottleneck was and what I could do to make it faster - certainly it was not the cpu.

Will the database differ in any way using replicate.py? (Maybe time of import will be preserved as replicate does not parse json files?)

You would like to see config files of which orthanc instance, the new, postgres one?

I've already finished the migration using ImportDicomFile.py with
127.0.0.1 (on Debian). It took around 30 hours to make full copy. Moreover,
read and write was on the same hard disk location (some kind of RAID).
I don't actually know where the bottleneck was and what I could do to make
it faster - certainly it was not the cpu.

Will the database differ in any way using replicate.py? (Maybe time of
import will be preserved as replicate does not parse json files?)

No, actually, the time of import will be lost in both replicate.py and
importDicomFile.py.

You would like to see config files of which orthanc instance, the new,
postgres one?

It's more the logs (output from the script and the new orthanc during the
migration) that would have been nice to see to get an idea of timings,
check for errors ...

Thank You.

In fact, I haven’t even saved ImportDicomFile.py output.

/var/log/orthanc contains some repetitive errors concerning UTF8, and no other relevant data from the import:

E0630 19:42:28.443552 ServerIndex.cpp:833] EXCEPTION [The plugin implementing a custom database back-end does not fulfill the proper interface]
E0630 19:42:28.443872 ServerContext.cpp:282] Store failure
E0630 19:43:57.860308 PluginsManager.cpp:163] Exception in database back-end: Error in PostgreSQL: BŁĄD: niepoprawna sekwencja bajtów dla kodowania “UTF8”: 0xe2 0x20 0x20
(in random intervals, more or less every 3 minutes).

Few days later I had some warnings:
W0705 15:54:40.557208 FromDcmtkBridge.cpp:369] Value of Specific Character Set (0008,0005) is not supported: 0, fallback to ASCII (remove all special characters)
W0705 15:54:40.772506 FromDcmtkBridge.cpp:369] Value of Specific Character Set (0008,0005) is not supported: 0, fallback to ASCII (remove all special characters)

Currently my Orthanc accepts around 40-50 MBit/s of data without any clear server-side bottlenecks (CPU usage ~15%, split more or less equally between postgres and Orthanc binaries). I’ve got 684 GB of data and 3,2 GB for postgres database.

There are some issues currently - but in my opinion they are minor. I should probably report them on bitbucket. or open a new thread here, i’ll just mention them:

  • running several find requests in row from radiant from one host makes the server stalled for ~10 minutes,
  • very large (~500MB) single dicom instances does not upload from Osirix
  • patient list json file sometimes timeouts in web explorer - it’s around 1,1 MB now