Hi friends!
We’ve got a couple of Orthanc instances on production! And more are expected to go live this year, yay! It’s awesome to see Orthanc run in in real world scenarios with real world data. It’s enlightening.
But I have observed over the last couple of weeks what I believe to be a memory leak, thus this e-mail is written.
It is not “strongly” related to Orthanc itself as far as I know. It’s related to the PostgreSQLIndex plugin. Let me explain:
Our setup for this stage of the project is pretty simple: our custom Orthanc storing files on disk and indexes in memory. Every five minutes the current “de facto” PACS server sends us new instances to be stored through DICOM.
What has been observed is that the memory consumption for one PostgreSQL process seemingly only grows. The production host has some 62.5GB of RAM and I’ve seen a particular postgres process reach 50GB+ at one time. The one associated with the one connection.
Has anyone experienced this? How did you deal with it?
Going back to the lab I was able to reproduce the problem with the postgresql-easy setup from Osimis’ samples. To reproduce it, fire the the compose file and start a non-stop stream of instances. You can use storescu with a large directory of DCM’s and the +sd and +r parameters of storescu.
Here’s a couple of screenshots illustrating this apparent leak:
Image #1: This was taken right before I started the sending of instances (15M of RES memory)
Image #2: This was taken at the end of the process (now it’s some 57MB)
Image #3: Orthanc’s memory consumption after the process was done. This image also show (in the second terminal) that roughly 6.4GB of instances were sent.
I’ve looked a little at both Orthanc and PostgreSQLIndex plugin source codes. What I’ve learned so far is it seems every time GetDatabase is called, PostgreSQLDatabase::open is as well but this connection is never closed. There are only two points in which PQfinish is called, both on orthanc-databases/Framework/PostgreSQL/PostgreSQLDatabase.cpp:
- Line 62, during PostgreSQLDatabase::Close: There’s a message that greping our INFO-enabled logs fail to find.
- Line 102, during PostgreSQLDatabase::Open: As part of a error condition, the PQfinish is called and a message is logged. Again, grep fails to find it.
I’m thinking that maybe our PostgreSQL setup need tuning but I’m not sure. Also, I could have the plugin close the connection but again I’m not sure. I could have a pgPool-II docker image sit between Orthanc and PostgreSQL as well. What I’m most likely to do is:
- Have a pgPool-II docker image sit between Orthanc and PostgreSQL.
- Have pgPool-II timeout idle connections.
- If needed, rewrite the PostgreSQLDatabase::Open like the attachment. But I may experience issues should the connection be recreated during a DICOM burst.
Any ideas are mostly welcome!
Thanks in advance!
proposed-postgresqldatabase-open.cpp (1.22 KB)
