For a recent project I had to write my own Orthanc plugin. To build this plugin I needed to build Orthanc from source. The official docker-images are assembled based on pre-built binaries. So I could not use them.
Orthanc Dockerfile
The first step is create a Dockerfile that compiles Orthanc from source. Looking through the compilation instructions and inspired by the official docker images I have assembled the following Dockerfile:
FROM debian:stable AS builder
WORKDIR /root
ENV ORTHANC_VERSION 1.6.0
RUN apt update && apt install -y build-essential unzip cmake mercurial \
uuid-dev libcurl4-openssl-dev liblua5.1-0-dev \
libgtest-dev libpng-dev libjpeg-dev \
libsqlite3-dev libssl-dev zlib1g-dev libdcmtk2-dev \
libboost-all-dev libwrap0-dev libjsoncpp-dev libpugixml-dev distcc
# pull source from upstream
RUN hg clone -b Orthanc-${ORTHANC_VERSION} --stream https://bitbucket.org/sjodogne/orthanc Orthanc
# distcc
ARG JOBS=4
ARG DISTCC_HOSTS=localhost/4
RUN echo "${DISTCC_HOSTS}" > /etc/distcc/hosts
ARG XXX=1
RUN mkdir OrthancBuild
RUN cd OrthancBuild && cmake -DALLOW_DOWNLOADS=ON \
# -DSTATIC_BUILD=ON \
-DCMAKE_CXX_COMPILER_LAUNCHER=distcc \
-DUSE_SYSTEM_CIVETWEB=OFF \
-DUSE_GOOGLE_TEST_DEBIAN_PACKAGE=OFF \
-DDCMTK_LIBRARIES=dcmjpls \
-DCMAKE_BUILD_TYPE=Release \
~/Orthanc
RUN cd OrthancBuild && make -j ${JOBS}
FROM debian:stable-slim
# locales
RUN apt-get update && apt install -y locales wget
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen
RUN locale-gen
RUN mkdir -p \
/etc/orthanc \
/var/lib/orthanc/db \
/usr/local/sbin/ \
/usr/local/bin/ \
/usr/local/share/orthanc/plugins/
COPY --from=builder /root/OrthancBuild/Orthanc /usr/local/sbin/
COPY --from=builder /root/OrthancBuild/OrthancRecoverCompressedFile /usr/local/sbin/
COPY --from=builder /root/OrthancBuild/lib* /usr/local/share/orthanc/plugins/
VOLUME [ "/var/lib/orthanc/db" ]
EXPOSE 4242
EXPOSE 8042
ENTRYPOINT [ "Orthanc" ]
CMD [ "/etc/orthanc/" ]
# https://groups.google.com/d/msg/orthanc-users/qWqxpvCPv8g/Z8huoA5FDAAJ
ENV MALLOC_ARENA_MAX 5
Docker build
Build the image with
docker build . -t orthanc:latest
Above command takes around 6min (excluding the remote dependencies) to complete on my Macbook pro. Since I’m expecting to build more regularly, I started to look around to speed up the process. I found the following options:
- Use a different docker image as cache
- Use experimental RUN –mount
- Use ccache, alone this has no advantage for docker builds because the cache directory is flushed for every build.
- Use distcc to offload some compilation operations to external host
- Finally I decided to combine
distccwith local caching ofccache.
The idea is not new. First resource I found was very useful to show distcc usage in kubernetes. However it wasn’t enough as I didn’t think the solution is very elegant. So then I found a second article with cleaner solution on combining distcc with ccache.
cinaq/distcc
As a result I have combined both articles into a simple to use distcc-ccache-kubernetes setup. It’s documented at cinaq/distcc-docker.
With cinaq/distcc running in Kubernetes, we can now compile Orthanc 2 times as fast with:
docker build . --build-arg DISTCC_HOSTS=distcc.dev/4 --build-arg JOBS=4 -t orthanc:latest
Here we pass 2 build arguments:
- DISTCC_HOSTS: distcc.dev/4 where
distcc.devis the hostname of our distcc (cluster) and 4 is number of jobs to send - JOBS: 4, denotes the total number of jobs
makewill run in parallel. We should be able to increase this more. We left this value the same for both test cases to confirm distcc and ccache do speed up the overal build process.
We can observe the logs of distcc service:
kubectl logs distcc-deployment-5d6fb547d7-z2rlv
distccd[11] (dcc_job_summary) client: 10.1.28.1:59075 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:4342ms /usr/lib/ccache/c++ /root/Orthanc/Core/Cache/MemoryCache.cpp
distccd[10] (dcc_job_summary) client: 10.1.28.1:59074 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:7109ms /usr/lib/ccache/c++ /root/Orthanc/Core/Cache/MemoryObjectCache.cpp
distccd[12] (dcc_job_summary) client: 10.1.28.1:59090 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:4178ms /usr/lib/ccache/c++ /root/Orthanc/Core/Cache/MemoryStringCache.cpp
...
And the cache being populated:
kubectl exec -it distcc-deployment-5d6fb547d7-z2rlv -- /bin/ls -l /cache
total 68
drwxr-xr-x 13 distcc distcc 4096 May 10 19:37 0
drwxr-xr-x 10 distcc distcc 4096 May 10 19:36 1
drwxr-xr-x 6 distcc distcc 4096 May 10 19:36 2
drwxr-xr-x 10 distcc distcc 4096 May 10 19:37 3
drwxr-xr-x 11 distcc distcc 4096 May 10 19:37 4
drwxr-xr-x 9 distcc distcc 4096 May 10 19:35 5
drwxr-xr-x 9 distcc distcc 4096 May 10 19:35 6
drwxr-xr-x 6 distcc distcc 4096 May 10 19:37 7
drwxr-xr-x 11 distcc distcc 4096 May 10 19:37 8
drwxr-xr-x 14 distcc distcc 4096 May 10 19:37 9
drwxr-xr-x 13 distcc distcc 4096 May 10 19:37 a
drwxr-xr-x 11 distcc distcc 4096 May 10 19:37 b
drwxr-xr-x 13 distcc distcc 4096 May 10 19:37 c
-rw-r--r-- 1 distcc distcc 16 May 10 19:32 ccache.conf
drwxr-xr-x 10 distcc distcc 4096 May 10 19:37 d
drwxr-xr-x 9 distcc distcc 4096 May 10 19:36 e
drwxr-xr-x 11 distcc distcc 4096 May 10 19:37 f
The new build takes 3min to complete to build Orthanc from source.
Conclusions
Honestly I was expecting much higher speed up. But 50% time reduction is very nice nonetheless. I really liked breathing new fresh life into mature tools like distcc and ccache in a modern stack.
Future work
- Scale up number of pods running
distcccould speed up the process further. Not sure how distcc will react because there would be multiple distcc instances behind a single LoadBalancer host. - dive deeper into
distccfor optimal parameters could also result in speed up. - allow usage of
Persistent Volumeinstead ofEmptyDirso that caches are persisted longer than the pod lifetime. - package this up into a helm chart.