All Articles

OpenRestyをopentracing付きでdocker buildする

OpenRestyをOpentracing付きでビルドする

NginxにOpentracingを導入する必要があってアレコレやって成功したメモ書きです。

DockerのNginxにOpentracingを入れるには 公式Docker Imageがありますが、OpenRestyを併用する場合はやっぱり自分でイメージをビルドしないといけなくて、openresty/docker-openresty を使いますが、これをベースにopentracingを入れるのに少し苦労しました。

今回はOpentracingの送信先にはdatadogを使っていて、その場合、

  1. dd-opentracing-cpp
  2. opentracing-cpp
  3. nginx-opentracing

の順に3つのプラグインを入れる必要があります。で、dd-opentracing-cpp のREADMEを参考にビルドをしていくとopentracing-cppのバージョンが違うと言われて動かず。またalpineでやるとlib64周りのエラーでこれまた動かず。結局xenialのイメージを使いました。

最終的に成功した Dockerfile を下記に貼っておきます。OpenSSLバージョンなんかも少し調整しています。

# Dockerfile - Ubuntu Xenial
# https://github.com/openresty/docker-openresty
ARG RESTY_IMAGE_BASE="ubuntu"
ARG RESTY_IMAGE_TAG="xenial"
FROM ${RESTY_IMAGE_BASE}:${RESTY_IMAGE_TAG}
LABEL maintainer="Evan Wies <evan@neomantra.net>"
# Docker Build Arguments
ARG RESTY_IMAGE_BASE="ubuntu"
ARG RESTY_IMAGE_TAG="xenial"
ARG RESTY_VERSION="1.15.8.2"
ARG RESTY_LUAROCKS_VERSION="3.2.1"
ARG RESTY_OPENSSL_VERSION="1.1.1f"
ARG RESTY_PCRE_VERSION="8.43"
ARG RESTY_J="1"
ARG RESTY_CONFIG_OPTIONS="\
--with-compat \
--with-file-aio \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_geoip_module=dynamic \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_image_filter_module=dynamic \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-http_xslt_module=dynamic \
--with-ipv6 \
--with-mail \
--with-mail_ssl_module \
--with-md5-asm \
--with-pcre-jit \
--with-sha1-asm \
--with-stream \
--with-stream_ssl_module \
--with-threads \
"
ARG RESTY_CONFIG_OPTIONS_MORE="--add-dynamic-module=/tmp/nginx-opentracing/opentracing"
ARG RESTY_LUAJIT_OPTIONS="--with-luajit-xcflags='-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT'"
ARG RESTY_ADD_PACKAGE_BUILDDEPS="git cmake wget"
ARG RESTY_ADD_PACKAGE_RUNDEPS=""
ARG RESTY_EVAL_PRE_CONFIGURE=""
ARG RESTY_EVAL_POST_MAKE=""
# These are not intended to be user-specified
ARG _RESTY_CONFIG_DEPS="--with-pcre \
--with-cc-opt='-DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' \
--with-ld-opt='-L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' \
"
LABEL resty_image_base="${RESTY_IMAGE_BASE}"
LABEL resty_image_tag="${RESTY_IMAGE_TAG}"
LABEL resty_version="${RESTY_VERSION}"
LABEL resty_luarocks_version="${RESTY_LUAROCKS_VERSION}"
LABEL resty_openssl_version="${RESTY_OPENSSL_VERSION}"
LABEL resty_pcre_version="${RESTY_PCRE_VERSION}"
LABEL resty_config_options="${RESTY_CONFIG_OPTIONS}"
LABEL resty_config_options_more="${RESTY_CONFIG_OPTIONS_MORE}"
LABEL resty_config_deps="${_RESTY_CONFIG_DEPS}"
LABEL resty_luajit_options="${RESTY_LUAJIT_OPTIONS}"
LABEL resty_add_package_builddeps="${RESTY_ADD_PACKAGE_BUILDDEPS}"
LABEL resty_add_package_rundeps="${RESTY_ADD_PACKAGE_RUNDEPS}"
LABEL resty_eval_pre_configure="${RESTY_EVAL_PRE_CONFIGURE}"
LABEL resty_eval_post_make="${RESTY_EVAL_POST_MAKE}"
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
gettext-base \
libgd-dev \
libgeoip-dev \
libncurses5-dev \
libperl-dev \
libreadline-dev \
libxslt1-dev \
make \
perl \
unzip \
zlib1g-dev \
${RESTY_ADD_PACKAGE_BUILDDEPS} \
${RESTY_ADD_PACKAGE_RUNDEPS}
RUN if [ -n "${RESTY_EVAL_PRE_CONFIGURE}" ]; then eval $(echo ${RESTY_EVAL_PRE_CONFIGURE}); fi \
&& cd /tmp \
&& git clone https://github.com/DataDog/dd-opentracing-cpp \
&& cd dd-opentracing-cpp \
&& sh ./scripts/install_dependencies.sh \
&& mkdir .build && cd .build \
&& cmake .. && make && make install \
&& cd /tmp \
&& git clone https://github.com/opentracing/opentracing-cpp.git \
&& cd opentracing-cpp \
&& git checkout v1.5.1 \
&& mkdir .build && cd .build && cmake .. && make && make install \
&& cd /tmp \
&& git clone https://github.com/opentracing-contrib/nginx-opentracing.git \
&& cd /tmp \
&& curl -fSL https://www.openssl.org/source/openssl-${RESTY_OPENSSL_VERSION}.tar.gz -o openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
&& tar xzf openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
&& cd openssl-${RESTY_OPENSSL_VERSION} \
&& if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.1" ] ; then \
echo 'patching OpenSSL 1.1.1 for OpenResty' \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-1.1.1c-sess_set_get_cb_yield.patch | patch -p1 ; \
fi \
&& if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.0" ] ; then \
echo 'patching OpenSSL 1.1.0 for OpenResty' \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/ed328977028c3ec3033bc25873ee360056e247cd/patches/openssl-1.1.0j-parallel_build_fix.patch | patch -p1 \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-1.1.0d-sess_set_get_cb_yield.patch | patch -p1 ; \
fi \
&& ./config \
no-threads shared zlib -g \
enable-ssl3 enable-ssl3-method \
--prefix=/usr/local/openresty/openssl \
--libdir=lib \
-Wl,-rpath,/usr/local/openresty/openssl/lib \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install_sw \
&& cd /tmp \
&& curl -fSL https://ftp.pcre.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz \
&& tar xzf pcre-${RESTY_PCRE_VERSION}.tar.gz \
&& cd /tmp/pcre-${RESTY_PCRE_VERSION} \
&& ./configure \
--prefix=/usr/local/openresty/pcre \
--disable-cpp \
--enable-jit \
--enable-utf \
--enable-unicode-properties \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install \
&& cd /tmp \
&& curl -fSL https://github.com/openresty/openresty/releases/download/v${RESTY_VERSION}/openresty-${RESTY_VERSION}.tar.gz -o openresty-${RESTY_VERSION}.tar.gz \
&& tar xzf openresty-${RESTY_VERSION}.tar.gz \
&& cd /tmp/openresty-${RESTY_VERSION} \
&& eval ./configure -j${RESTY_J} ${_RESTY_CONFIG_DEPS} ${RESTY_CONFIG_OPTIONS} ${RESTY_CONFIG_OPTIONS_MORE} ${RESTY_LUAJIT_OPTIONS} \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install \
&& cd /tmp \
&& rm -rf \
openssl-${RESTY_OPENSSL_VERSION}.tar.gz openssl-${RESTY_OPENSSL_VERSION} \
pcre-${RESTY_PCRE_VERSION}.tar.gz pcre-${RESTY_PCRE_VERSION} \
openresty-${RESTY_VERSION}.tar.gz openresty-${RESTY_VERSION} \
nginx-opentracing \
opentracing-cpp \
&& curl -fSL https://luarocks.github.io/luarocks/releases/luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz -o luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
&& tar xzf luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
&& cd luarocks-${RESTY_LUAROCKS_VERSION} \
&& ./configure \
--prefix=/usr/local/openresty/luajit \
--with-lua=/usr/local/openresty/luajit \
--lua-suffix=jit-2.1.0-beta3 \
--with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1 \
&& make build \
&& make install \
&& cd /tmp \
&& if [ -n "${RESTY_EVAL_POST_MAKE}" ]; then eval $(echo ${RESTY_EVAL_POST_MAKE}); fi \
&& rm -rf luarocks-${RESTY_LUAROCKS_VERSION} luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
&& if [ -n "${RESTY_ADD_PACKAGE_BUILDDEPS}" ]; then DEBIAN_FRONTEND=noninteractive apt-get remove -y --purge ${RESTY_ADD_PACKAGE_BUILDDEPS} ; fi \
&& DEBIAN_FRONTEND=noninteractive apt-get autoremove -y \
&& mkdir -p /var/run/openresty \
&& ln -sf /dev/stdout /usr/local/openresty/nginx/logs/access.log \
&& ln -sf /dev/stderr /usr/local/openresty/nginx/logs/error.log
# Add additional binaries into PATH for convenience
ENV PATH=$PATH:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin
# Add LuaRocks paths
# If OpenResty changes, these may need updating:
# /usr/local/openresty/bin/resty -e 'print(package.path)'
# /usr/local/openresty/bin/resty -e 'print(package.cpath)'
ENV LUA_PATH="/usr/local/openresty/site/lualib/?.ljbc;/usr/local/openresty/site/lualib/?/init.ljbc;/usr/local/openresty/lualib/?.ljbc;/usr/local/openresty/lualib/?/init.ljbc;/usr/local/openresty/site/lualib/?.lua;/usr/local/openresty/site/lualib/?/init.lua;/usr/local/openresty/lualib/?.lua;/usr/local/openresty/lualib/?/init.lua;./?.lua;/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/openresty/luajit/share/lua/5.1/?.lua;/usr/local/openresty/luajit/share/lua/5.1/?/init.lua"
ENV LUA_CPATH="/usr/local/openresty/site/lualib/?.so;/usr/local/openresty/lualib/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so"
# Copy nginx configuration files
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
COPY nginx.vh.default.conf /etc/nginx/conf.d/default.conf
CMD ["/usr/local/openresty/bin/openresty", "-g", "daemon off;"]
# Use SIGQUIT instead of default SIGTERM to cleanly drain requests
# See https://github.com/openresty/docker-openresty/blob/master/README.md#tips--pitfalls
STOPSIGNAL SIGQUIT
view raw Dockerfile hosted with ❤ by GitHub

ポイントは以下の部分:

  ARG RESTY_CONFIG_OPTIONS_MORE="--add-dynamic-module=/tmp/nginx-opentracing/opentracing"

  ...

  && git clone https://github.com/opentracing/opentracing-cpp.git \
  && cd opentracing-cpp \
  && git checkout v1.5.1 \

  ...

opentracing-cpp は最新版だとバージョンの相違で動かないので、 v1.5.1 のタグを参照させないとダメでした。

Layerを減らすために RUN をつなげていると、途中でビルドに失敗したときにやり直しがしんどいですね。 一回のビルドに数分かかるので、こちらを参考にしてもらうと良いかもしれません。

これでNginxもDatadogのAPMから観測できるようになるのでパフォーマンスモニタリングが捗ります。

現場からは以上です。

Published 4 Apr 2020