Hello,
I’ve been writing a script to add an API that generates an overview of a 3D volume into a single image recently but i’ve encountered a bug that I can’t seem to resolve so I thought I could ask you guys for help.
So the problem I have is that when I use multithreading (here) it gives me this error when calling the API but I don’t have the error when running the script manually, so I thought it was linked to the fact that multithreading didn’t work because the package requires that the main module.
If any one has a idea, I would be glad to listen to it.
Best regards,
Théophilus
Hi Théophilus,
The error message you mention seems incomplete (at least I don’t get useful info from it). I see that you have a full docker setup but you should also provide instruction on how to reproduce your issue (like a curl command and sample data to upload).
BTW, python multiprocessing should work within Orthanc: https://book.orthanc-server.com/plugins/python.html#performance-and-concurrency
Best regards,
Alain.
Hello,
Basically the issue appears when using basic multiprocessing in a python script. I made a basic script that reproduces the error when calling the API. The script will still work but there is going to be an error in the console and it will only use 1 core.
Best regards,
Théophilus.
bug_reproduction.py (792 Bytes)
Hi Théophilus,
What message do you get ? I do not see anything wrong on the logs on my side ? How do you know that it is running on a single core ?
I1025 07:16:03.930290 HttpServer.cpp:1262] (http) GET /bug_test
I1025 07:16:03.930396 OrthancPlugins.cpp:2443] (plugins) Delegating HTTP request to plugin for URI: /bug_test
I1025 07:16:04.080467 HttpServer.cpp:1262] (http) GET /app/images/favicon.ico
I1025 07:16:42.467378 HttpServer.cpp:1262] (http) GET /bug_test
I1025 07:16:42.467488 OrthancPlugins.cpp:2443] (plugins) Delegating HTTP request to plugin for URI: /bug_test
I1025 07:16:42.640806 HttpServer.cpp:1262] (http) GET /app/images/favicon.ico
Hello Alain,
It seems that to see it you need to have ‘ORTHANC__PYTHON_VERBOSE: “true”’ activated in your environment variables.
Best regards,
Théophilus
Well, here’s the output I get. I do not see any error message in there. Just the python interpreter that is loading a module the first time the code is executed.
orthanc_1 | I1025 13:51:07.154407 LuaScripting.cpp:868] Starting the Lua engine
orthanc_1 | I1025 13:51:18.687211 HttpServer.cpp:1244] (http) GET /bug_test
orthanc_1 | I1025 13:51:18.687319 OrthancPlugins.cpp:2443] (plugins) Delegating HTTP request to plugin for URI: /bug_test
orthanc_1 | I1025 13:51:18.687364 PluginsManager.cpp:161] (plugins) Creating Python object of class OrthancPluginRestOutput
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/pool.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/pool.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/pool.cpython-39.pyc’
orthanc_1 | # /usr/lib/python3.9/pycache/queue.cpython-39.pyc matches /usr/lib/python3.9/queue.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/pycache/queue.cpython-39.pyc’
orthanc_1 | # extension module ‘queue’ loaded from '/usr/lib/python3.9/lib-dynload/queue.cpython-39-x86_64-linux-gnu.so’
orthanc_1 | # extension module ‘queue’ executed from '/usr/lib/python3.9/lib-dynload/queue.cpython-39-x86_64-linux-gnu.so’
orthanc_1 | import ‘_queue’ # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f856ed3c850>
orthanc_1 | import ‘queue’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856ed23b80>
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/util.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/util.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/util.cpython-39.pyc’
orthanc_1 | import ‘multiprocessing.util’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856ed3c940>
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/connection.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/connection.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/connection.cpython-39.pyc’
orthanc_1 | # extension module ‘multiprocessing’ loaded from '/usr/lib/python3.9/lib-dynload/multiprocessing.cpython-39-x86_64-linux-gnu.so’
orthanc_1 | # extension module ‘multiprocessing’ executed from '/usr/lib/python3.9/lib-dynload/multiprocessing.cpython-39-x86_64-linux-gnu.so’
orthanc_1 | import ‘_multiprocessing’ # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f856c4cc8e0>
orthanc_1 | import ‘multiprocessing.connection’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856ed453a0>
orthanc_1 | import ‘multiprocessing.pool’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856ed67e20>
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/queues.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/queues.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/queues.cpython-39.pyc’
orthanc_1 | import ‘multiprocessing.queues’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856ed235e0>
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/synchronize.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/synchronize.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/synchronize.cpython-39.pyc’
orthanc_1 | import ‘multiprocessing.synchronize’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856c4de520>
orthanc_1 | # /usr/lib/python3.9/multiprocessing/pycache/popen_fork.cpython-39.pyc matches /usr/lib/python3.9/multiprocessing/popen_fork.py
orthanc_1 | # code object from ‘/usr/lib/python3.9/multiprocessing/pycache/popen_fork.cpython-39.pyc’
orthanc_1 | import ‘multiprocessing.popen_fork’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f856c4e6910>
orthanc_1 | I1025 13:51:18.732324 PluginsManager.cpp:161] (plugins) Calling method OrthancPluginAnswerBuffer() on object of class OrthancPluginRestOutput
orthanc_1 | I1025 13:51:19.001566 HttpServer.cpp:1244] (http) GET /favicon.ico
orthanc_1 | I1025 13:51:19.006081 HttpServer.cpp:1244] (http) GET /app/images/favicon.ico
orthanc_1 | I1025 13:53:47.801291 HttpServer.cpp:1244] (http) GET /bug_test
orthanc_1 | I1025 13:53:47.801354 OrthancPlugins.cpp:2443] (plugins) Delegating HTTP request to plugin for URI: /bug_test
orthanc_1 | I1025 13:53:47.801381 PluginsManager.cpp:161] (plugins) Creating Python object of class OrthancPluginRestOutput
orthanc_1 | I1025 13:53:47.833326 PluginsManager.cpp:161] (plugins) Calling method OrthancPluginAnswerBuffer() on object of class OrthancPluginRestOutput
orthanc_1 | I1025 13:53:48.021371 HttpServer.cpp:1244] (http) GET /app/images/favicon.ico
I just tried myself and it seems you are right, it does multiprocess but it seems to be really slow, on this simple algorithm the executions takes at least twice the time it would take compared to running the program myself on the same computer. Do you have any ideas why and/or a solution?
Best regards,
Théophilus
bug_reproduction.py (1.05 KB)
Thanks Alain for your help, we were confused about the warning message and several bugs we had that occurs which were not related to multithreading.
@Theophilus, I think the performance drop you have is because of Docker, with docker you virtualizing a linux machine on the top of your Mac OS while running the script locally you avoid this virtualization step.
(https://abletech.nz/article/performance-comparison-for-docker-for-mac-vm-linux-docker-and-linux-docker-native/)
Using docker on a linux system shows great performances, multiprocessing works well.
