shadowsocks libev 3.0 cross compiling for tomato shibby

shadowsocks-libev出到3.0了

edit：
如果不想看下面这一大段垃圾伤眼，可以直接去下载lz编译好的http://oglopss.github.io/ctng-ss-jekyll/

同样使用之前弄出来的ct-ng工具链

• Use crosstool-ng 1.21 toolchain

  git clone https://github.com/crosstool-ng/crosstool-ng.git
  cd crosstool-ng
  git checkout tags/crosstool-ng-1.21.0
  ./bootstrap
  ./configure
  make && make install
  

• compile toolchain
  config files gist

  mkdir -p ~/config
  cd ~/config
  
  # copy my ctng.config as .config
  # copy uclibc.config as uclibc.config
  
  ct-ng build.4
  

• clone shadowsock-libev

  cd ~/Downloads# 连submodule一起clone下来
  git clone --recursive https://github.com/shadowsocks/shadowsocks-libev.git
  cd shadowsocks-libev
  git checkout tags/v3.0.5
  

  如果clone的时候没有连带submodule一起,可以

  # git >= 1.8.2
  git submodule update --recursive --remote
  
  # git >= 1.7.3
  git submodule update --init --recursive
  
  # 如果想把submodule升级到最新，无视repo里的设置
  git pull --recurse-submodules
  

• 把toolchain加到PATH里

  export PATH=/home/oglop/x-tools/mipsel-unknown-linux-uclibc/bin:$PATH
  

• libsodium

  cd ~/Downloads/shadowsocks-libev
  export LIBSODIUM_VER=1.0.11
  # 好像这地址挂了
  # wget https://download.libsodium.org/libsodium/releases/libsodium-$LIBSODIUM_VER.tar.gz
  wget --backups=1 https://github.com/jedisct1/libsodium/releases/download/$LIBSODIUM_VER/libsodium-$LIBSODIUM_VER.tar.gz
  tar xvf libsodium-$LIBSODIUM_VER.tar.gz
  pushd libsodium-$LIBSODIUM_VER
  CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib  ./configure --prefix=$HOME/Downloads/libsodium-install --host=mipsel-uclibc-linux
  
  make
  make install
  
  popd
  

• mbedTLS

  export MBEDTLS_VER=2.4.2
  wget https://tls.mbed.org/download/mbedtls-$MBEDTLS_VER-gpl.tgz
  tar xvf mbedtls-$MBEDTLS_VER-gpl.tgz
  pushd mbedtls-$MBEDTLS_VER
  CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib make SHARED=1 CFLAGS=-fPIC
  
  make install DESTDIR=$HOME/Downloads/mbedtls-install
  
  popd
  

  这里有个问题是如果在make后面不加 SHARED=1 CFLAGS=-fPIC 最后就不会产生对libmbedcrypto.so.0的依赖，但是可执行文件会大一点，因为静态链接进去了大概，但依然会有libev的依赖

• udns

  export UDNS_VER=0.4
  wget http://www.corpit.ru/mjt/udns/udns-$UDNS_VER.tar.gz
  tar xvf udns-$UDNS_VER.tar.gz
  pushd udns-$UDNS_VERCC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib ./configure
  
  make
  # 因为作者没在makefile里放install
  popd
  

  其中conftest过不了，因为是cross compiling,所以去config.lib里做如下修改 跳过conftest

  if [ -n "$CC" ]; then
  -    if ac_run $CC -o conftest conftest.c && ac_run ./conftest; then
  +    if ac_run $CC -o conftest conftest.c; then
  ac_result "\$CC ($CC)"
  else
  ac_result no
  ccld="$cc"
  if [ -n "$LDFLAGS" ]; then ccld="$ccld $LDFLAGS"; fi
  if [ -n "$LIBS" ]; then ccld="$ccld $LIBS"; fi
  -  if ac_yesno "whenever the C compiler ($ccld)
  -           can produce executables" \
  -     ac_compile_run <<EOF
  -int main() { return 0; }
  -EOF
  -  then :
  -  else
  -    ac_fatal "no working C compiler found"
  -  fi
  

• pcre

  export PCRE_VER=8.40
  wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-$PCRE_VER.tar.gz
  tar xvf pcre-$PCRE_VER.tar.gz
  cd pcre-$PCRE_VERCC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib ./configure --host=mipsel-uclibc-linux --prefix=$HOME/Downloads/pcre-install
  
  make
  make install
  

• libev

  git clone https://github.com/enki/libev.git
  cd libev
  CPPFLAGS=-I$HOME/Downloads/udns-0.4 LDFLAGS=-L$HOME/Downloads/udns-0.4  CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib ./configure --prefix=$HOME/Downloads/libev-install --host=mipsel-uclibc-linux
  make
  make install
  

• shadowsocks-libev
  因为configure没有给libev和udns的配置参数，所以放在CPPFLAGS和LDFLAGS里了

  ./autogen.shCPPFLAGS="-I$HOME/Downloads/udns-0.4 -I$HOME/Downloads/libev-install/include"  LDFLAGS="-Wl,-rpath,/jffs/lib -L$HOME/Downloads/udns-0.4 -L$HOME/Downloads/libev-install/lib"  CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib ./configure --disable-ssp --prefix=$HOME/ss-install --with-pcre=$HOME/Downloads/pcre-install --with-sodium=$HOME/Downloads/libsodium-install --with-mbedtls=$HOME/Downloads/mbedtls-install --disable-documentation --host=mipsel-uclibc-linux
  
  make
  make install
  
  

  注意上面lz把 /opt/lib 放在了 rpath的最前面，这是为了让装有entware的路由器找到 libcrypto.so里的MD5 symbol, 以及libev.so lz把/jffs/lib加到了rpath里面，原因见下文

  lz试过

  -Wl,-z,origin -Wl,-rpath,\$\$ORIGIN':/lib:/usr/lib
  

  不过似乎没效果，也许是v138里的uclibc不支持rpath里用$ORIGIN, 那就还是自己用tomatoedit把.so文件放进/usr/lib吧

• 然后在$HOME/ss-install下可以见到编译出来的成果,把他们strip小一点然后upx压缩一下

  # 不要压./ss-nat因为他是个shell script
  find . ! -path './ss-nat' -type f | xargs mipsel-unknown-linux-uclibc-strip# upx 一下
  find . ! -path './ss-nat' -type f | xargs ~/Downloads/upx-3.91-amd64_linux/upx
  
  Ultimate Packer for eXecutables
  Copyright (C) 1996 - 2013
  UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013
  
  File size         Ratio      Format      Name
  --------------------   ------   -----------   -----------
  172376 ->     64116   37.20%  linux/mipsel   ss-local
  150008 ->     56140   37.42%  linux/mipsel   ss-tunnel
  212296 ->     78252   36.86%  linux/mipsel   ss-server
  91372 ->     37552   41.10%  linux/mipsel   ss-manager
  158488 ->     58948   37.19%  linux/mipsel   ss-redir
  --------------------   ------   -----------   -----------
  784540 ->    295008   37.60%                 [ 5 files ]
  
  

  压缩之前

  $ ll --block-size=K
  total 2916K
  -rwxr-xr-x. 1 oglop oglop 663K Feb 16 00:57 ss-local
  -rwxr-xr-x. 1 oglop oglop 339K Feb 16 00:57 ss-manager
  -rwxr-xr-x. 1 oglop oglop   6K Feb 16 00:57 ss-nat
  -rwxr-xr-x. 1 oglop oglop 594K Feb 16 00:57 ss-redir
  -rwxr-xr-x. 1 oglop oglop 737K Feb 16 00:57 ss-server
  -rwxr-xr-x. 1 oglop oglop 567K Feb 16 00:57 ss-tunnel
  
  

  压缩之后

  $ ll --block-size=K
  total 308K
  -rwxr-xr-x. 1 oglop oglop 63K Feb 16 00:58 ss-local
  -rwxr-xr-x. 1 oglop oglop 37K Feb 16 00:58 ss-manager
  -rwxr-xr-x. 1 oglop oglop  6K Feb 16 00:57 ss-nat
  -rwxr-xr-x. 1 oglop oglop 58K Feb 16 00:58 ss-redir
  -rwxr-xr-x. 1 oglop oglop 77K Feb 16 00:58 ss-server
  -rwxr-xr-x. 1 oglop oglop 55K Feb 16 00:58 ss-tunnel
  

• 拷到路由器上测试一下,lz的路由器是tomato shibby AIO v138,插了优盘，装了entware的

  root@unknown:/jffs/ss# ldd ./ss-server
  libev.so.4 => /jffs/lib/libev.so.4 (0x2aac1000)
  libm.so.0 => /lib/libm.so.0 (0x2aae1000)
  libsodium.so.18 => /usr/lib/libsodium.so.18 (0x2aaf8000)
  libmbedcrypto.so.0 => /jffs/lib/libmbedcrypto.so.0 (0x2ab56000)
  libpcre.so.1 => /usr/lib/libpcre.so.1 (0x2abf0000)
  libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2ac3c000)
  libpthread.so.0 => /lib/libpthread.so.0 (0x2ac5b000)
  libc.so.0 => /lib/libc.so.0 (0x2ac7e000)
  ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)

  由此看来，默认情况下AIO已经带了大部分需要的lib,(entware的都在/opt/lib下），还少一个libev,少libev和libmbedcrypto， 如果你在 configure ss的时候没加-Wl,-rpath,/jffs/lib 就会找不到libev和libmbedcrypto，此时你要么用tomatoedit把这个so加到固件里，然后重刷，要么可以试试用rpath修改可执行文件的rpath路径

  因为lz的路由器有usb口可以插优盘，而且lz装有entware,所以lz的libev是用opkg install libev直接从repo里装的，省去了一些麻烦.

  root@unknown:/tmp/home/root# opkg install libev
  Installing libev (4.20-1) to root...
  Downloading http://entware.wl500g.info/binaries/mipselsf/libev_4.20-1_mipselsf.ipk.
  Configuring libev.
  

  edit:
  经过尝试把 /opt/lib 加到 rpath里似乎不行，对于>=2.6.3的ss来说，会找不到某个symbol,所以还是需要把编译libev时产生的.so嵌入到/usr/lib下，或者可以试试自定义个路径加到/etc/ld.so.conf里，然后保存进nvram,或者加个/etc/ld.so.conf.d/ss.conf，如果不想重刷路由器的话

  最后…

  root@unknown:/tmp/home/root# ./ss-server -h
  shadowsocks-libev 3.0.5
  
  maintained by Max Lv <max.c.lv@gmail.com> and Linus Yang <laokongzi@gmail.com>
  
  usage:
  
  ss-server
  
  -s <server_host>           Host name or IP address of your remote server.
  -p <server_port>           Port number of your remote server.
  -l <local_port>            Port number of your local server.
  -k <password>              Password of your remote server.
  -m <encrypt_method>        Encrypt method: rc4-md5,
  aes-128-gcm, aes-192-gcm, aes-256-gcm,
  aes-128-cfb, aes-192-cfb, aes-256-cfb,
  aes-128-ctr, aes-192-ctr, aes-256-ctr,
  camellia-128-cfb, camellia-192-cfb,
  camellia-256-cfb, bf-cfb,
  chacha20-ietf-poly1305,
  salsa20, chacha20 and chacha20-ietf.
  The default cipher is rc4-md5.
  
  [-a <user>]                Run as another user.
  [-f <pid_file>]            The file path to store pid.
  [-t <timeout>]             Socket timeout in seconds.
  [-c <config_file>]         The path to config file.
  [-n <number>]              Max number of open files.
  [-i <interface>]           Network interface to bind.
  [-b <local_address>]       Local address to bind.
  
  [-u]                       Enable UDP relay.
  [-U]                       Enable UDP relay and disable TCP relay.
  [-6]                       Resovle hostname to IPv6 address first.
  
  [-d <addr>]                Name servers for internal DNS resolver.
  [--reuse-port]             Enable port reuse.
  [--fast-open]              Enable TCP fast open.
  with Linux kernel > 3.7.0.
  [--acl <acl_file>]         Path to ACL (Access Control List).
  [--manager-address <addr>] UNIX domain socket address.
  [--mtu <MTU>]              MTU of your network interface.
  [--mptcp]                  Enable Multipath TCP on MPTCP Kernel.
  [--key <key_in_base64>]    Key of your remote server.
  [--plugin <name>]          Enable SIP003 plugin. (Experimental)
  [--plugin-opts <options>]  Set SIP003 plugin options. (Experimental)
  
  [-v]                       Verbose mode.
  [-h, --help]               Print this message.
  
  

• compile simple-obfs

  git clone https://github.com/shadowsocks/simple-obfs
  cd simple-obfs
  git checkout tags/v0.0.3
  git submodule init && git submodule updateLDFLAGS="-Wl,-rpath,/jffs/lib -L$HOME/Downloads/libsodium-install/lib -L$HOME/Downloads/udns-$UDNS_VER -L$HOME/Downloads/libev-install/lib" CFLAGS="-I$HOME/Downloads/libsodium-install/include -I$HOME/Downloads/udns-$UDNS_VER -I$HOME/Downloads/libev-install/include" CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib ./configure --host=mipsel-uclibc-linux --prefix=$HOME/Downloads/obfs-install --disable-ssp --disable-documentation
  make && make install
  
  

  obfs

  root@unknown:/jffs/ss# ./obfs-server -h
  
  simple-obfs 0.0.3
  
  maintained by Max Lv <max.c.lv@gmail.com>
  
  usage:
  
  obfs-server
  
  -s <server_host>           Host name or IP address of your remote server.
  -p <server_port>           Port number of your remote server.
  -l <local_port>            Port number of your local server.
  -r <addr>:<port>           Forward traffic to this remote server address.
  --obfs <http|tls>          Enable obfuscating: HTTP or TLS (Experimental).
  
  [-a <user>]                Run as another user.
  [-f <pid_file>]            The file path to store pid.
  [-t <timeout>]             Socket timeout in seconds.
  [-c <config_file>]         The path to config file.
  [-n <number>]              Max number of open files.
  [-b <local_address>]       Local address to bind.
  
  [-6]                       Resovle hostname to IPv6 address first.
  
  [-d <addr>]                Name servers for internal DNS resolver.
  [--fast-open]              Enable TCP fast open.
  with Linux kernel > 3.7.0.
  [--mptcp]                  Enable Multipath TCP on MPTCP Kernel.
  
  [-v]                       Verbose mode.
  [-h, --help]               Print this message.
  
  root@unknown:/jffs/ss# ./obfs-local  -h
  
  simple-obfs 0.0.3
  
  maintained by Max Lv <max.c.lv@gmail.com>
  
  usage:
  
  obfs-local
  
  -s <server_host>           Host name or IP address of your remote server.
  -p <server_port>           Port number of your remote server.
  -l <local_port>            Port number of your local server.
  --obfs <http|tls>          Enable obfuscating: HTTP or TLS (Experimental).
  --obfs-host <host_name>    Hostname for obfuscating (Experimental).
  
  [-a <user>]                Run as another user.
  [-f <pid_file>]            The file path to store pid.
  [-t <timeout>]             Socket timeout in seconds.
  [-c <config_file>]         The path to config file.
  [-n <number>]              Max number of open files.
  [-b <local_address>]       Local address to bind.
  
  [--fast-open]              Enable TCP fast open.
  with Linux kernel > 3.7.0.
  [--mptcp]                  Enable Multipath TCP on MPTCP Kernel.
  
  [-v]                       Verbose mode.
  [-h, --help]               Print this message.
  
  

ps:
在travis ci上compile mbedtls的时候遇到奇怪的问题

# 如果使用下面的命令, 在笔记本上一切正常
CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib make SHARED=1 CFLAGS=-fPIC

总是提示没有看到-fPIC参数
/home/travis/x-tools/mipsel-unknown-linux-uclibc/lib/gcc/mipsel-unknown-linux-uclibc/5.2.0/../../../../mipsel-unknown-linux-uclibc/bin/ld: aes.o: relocation R_MIPS_HI16 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
aes.o: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make[1]: *** [libmbedcrypto.so.0] Error 1
make: *** [lib] Error 2

# 必须改成这样才行
CC="mipsel-unknown-linux-uclibc-gcc -fPIC" CXX="mipsel-unknown-linux-uclibc-g++ -fPIC" AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib make SHARED=1

如果mbedtls不编译成动态的(不加SHARED=1 CFLAGS=-fPIC ),编译出来的可执行文件会大一些，但是同时也少了一个依赖，不过lz还是希望可执行文件小一点，所以文末提供的下载是上面所示的更小的版本

total 532K
-rwxr-xr-x 1 travis travis 119K Feb 17 08:21 ss-local
-rwxr-xr-x 1 travis travis  37K Feb 17 08:21 ss-manager
-rwxr-xr-x 1 travis travis   6K Feb 17 08:21 ss-nat
-rwxr-xr-x 1 travis travis 114K Feb 17 08:21 ss-redir
-rwxr-xr-x 1 travis travis 133K Feb 17 08:21 ss-server
-rwxr-xr-x 1 travis travis 112K Feb 17 08:21 ss-tunnel

文件下载:

http://oglopss.github.io/ctng-ss-jekyll/

需要注意的是对于>=2.6.3的版本，会有如上ldd结果显示的依赖，所以还需要将两个.so文件放入/jffs/lib或者用tomatoedit加入固件中

ps:
目前ss-server >=2.6.3的版本在tomato shibby v138上 会有如下错误，无法正常使用

root@unknown:/jffs/ss# gdb -ex run --args /jffs/ss/ss-servre -c /jffs/ss/ss-server-config.json -u -v
2017-03-11 08:10:02 INFO: UDP relay enabled
2017-03-11 08:10:02 INFO: initializing ciphers... aes-256-cfb
2017-03-11 08:10:02 INFO: tcp server listening at 0.0.0.0:18388
2017-03-11 08:10:02 INFO: udp server listening at 0.0.0.0:18388
2017-03-11 08:10:02 INFO: running from root user
2017-03-11 08:10:08 INFO: accept a connection
Bus error
Program received signal SIGSEGV, Segmentation fault.
0x0041b88c in cork_error_set_printf ()
(gdb) bt
#0  0x0041b88c in cork_error_set_printf ()
#1  0x0041c7d0 in cork_ip_init ()
#2  0x004112e8 in server_recv_cb () at server.c:795
#3  0x2aac2a70 in ev_invoke_pending () from /jffs/lib/libev.so.4
#4  0x2aac6818 in ev_run () from /jffs/lib/libev.so.4
#5  0x00413044 in main () at server.c:1890

For higher version of crosstool-ng, the following patch maybe needed for libev

sed -i "/#define HAVE_EVENTFD 1/s/^/\/\/ /" ./config.h
# sed -i "s/epoll_create1/epoll_create/g" ./ev_epoll.c

and for libcork, force disable tls

sed -i -e 's/\(#define CORK_CONFIG_HAVE_THREAD_STORAGE_CLASS  \)1/\10/' ./libcork/include/libcork/config/gcc.h

perp

root@unknown:/jffs/ss# perpls
[+ +++ +++]  ss         uptime: 18s/18s  pids: 11703/11701
[+ +++ +++]  ss-server  uptime: 14s/14s  pids: 11707/11706

参考:

compile udns-0.0.9 on an MIPSEL-architecture

How to run travis-ci locally

GNU Linker 和 RPATH

How To Automatically Skip Wget SSL Certificate Check

Easy way pull latest of all submodules

rpath=$ORIGIN not having desired effect?

Building Portable Binaries – Attempting to self-contain application dependencies

Recompile with -fPIC option, but the option is already in the makefile

(静态)交叉编译shadowsocks-libev