Monday, August 5, 2013

Running Couchbase 2.1.1 on SmartOS

I assume most of my readers knows about my love to the Solaris operating system and its descendants such as SmartOS. I've been on vacation for a couple of weeks now, and during my vacation I noticed some comments on previous blog posts that people had tried to build Couchbase on their SmartOS system without success so I figured I should create a new blog post where I walked through the steps needed.

To make reproducible steps for people who are interested, I decided to ensure that the blog post includes all the steps needed (including creating the environment). The first thing we need to do is to log into our server and update the list of available datasets. I started off this morning by downloading smartos-20130725T202435Z-USB.img.bz2 and created a bootable USB stick and booted my server.

With my SmartOS server running the (as of today) latest bits I imported the dataset I was going to use for my build with the following command:

[root@smartos ~]# imgadm import 9eac5c0c-a941-11e2-a7dc-57a6b041988f

And created a the vm with the following setup:

[root@smartos ~]# cat | vmadm create
  "alias" : "couchbase",
  "autoboot": true,
  "brand": "joyent",
  "dns_domain" : "",
  “resolvers” : [ "" ],
  "image_uuid" : "9eac5c0c-a941-11e2-a7dc-57a6b041988f",
  "hostname" : "cbbuilder",
  "max_physical_memory": 4096,
  "nics": [
      "nic_tag": "admin",
      "ip": "",
      "netmask": "",
      "gateway": ""

Listing all of my vm's shows:

[root@smartos ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
1200e3a9-a9cc-49e5-b9f0-bed2ec3b005d  OS    4096     running           couchbase

The first thing I did was to log in and set the password for the root user and create my own user to use during the build process:

[root@smartos ~]# zlogin 1200e3a9-a9cc-49e5-b9f0-bed2ec3b005d
[Connected to zone '1200e3a9-a9cc-49e5-b9f0-bed2ec3b005d' pts/5]
Last login: Mon Aug  5 08:22:26 on pts/3
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; SmartMachine (base64 13.1.0)

[root@cbbuilder ~]# passwd root
[root@cbbuilder ~]# useradd -g 10 -s /usr/bin/bash \
-d /home/trond -m trond
[root@cbbuilder ~]# passwd trond

Now that I've got my own user I logged in as that user over an ssh shell and became root and installed all of the packages I need to build Couchbase:

[trond@cbbuilder ~]$ pfexec su -
[root@cbbuilder ~]# pkgin -y in libtool-base autoconf \
                                automake scmgit-base gcc47 \
                                gnupg gmake libevent icu \
                                py27-expat snappy erlang \
[root@cbbuilder ~]# wget --no-check-certificate \
                         -O/opt/local/bin/repo \
[root@cbbuilder ~]# chmod a+x /opt/local/bin/repo

I'll be installing Couchbase to /opt/couchbase, so lets go ahead and create that:

[root@cbbuilder ~]# mkdir /opt/couchbase
[root@cbbuilder ~]# chown trond /opt/couchbase

There are a few dependencies Couchbase use that don't exist in the pkgin repository. Let's go ahead and build them and install them into /opt/couchbase.

[trond@cbbuilder ~]$ wget --no-check-certificate \

[trond@cbbuilder ~]$ gtar xfz gperftools-2.1.tar.gz
[trond@cbbuilder ~]$ cd gperftools-2.1

[trond@cbbuilder ~/gperftools-2.1]$ ./configure --enable-minimal \
--enable-shared \
--disable-static \
[trond@cbbuilder ~/gperftools-2.1]$ gmake install
[trond@cbbuilder ~/gperftools-2.1]$ cd ..
[trond@cbbuilder ~]$ wget --no-check-certificate -Ov8.tar.gz \
[trond@cbbuilder ~]$ gtar xfz v8.tar.gz
[trond@cbbuilder ~]$ cd v8-3.19.0
[trond@cbbuilder ~/v8-3.19.0]$ gmake dependencies
[trond@cbbuilder ~/v8-3.19.0]$ gmake x64 library=shared -j 4
[trond@cbbuilder ~/v8-3.19.0]$ cp out/x64.release/ \
[trond@cbbuilder ~/v8-3.19.0]$ cp include/* /opt/couchbase/include/

To avoid passing too many arguments when we're invoking make we can add them into ~/.couchbase/build/Makefile.extra:

[trond@cbbuilder ~]$ mkdir -p ~/.couchbase/build
[trond@cbbuilder ~]$ cat > ~/.couchbase/build/Makefile.extra
OPTIONS += LDFLAGS="-R/opt/local/lib -L$(PREFIX)/lib -R$(PREFIX)/lib"
OPTIONS += CXX="g++ -L/opt/local/lib -I/opt/local/include" 
OPTIONS += CC="gcc -I/opt/local/include -L/opt/local/lib"
memcached_EXTRA_OPTIONS += --enable-tcmalloc-minimal

We need to "configure" git before we can start use it to download the source code:

[trond@cbbuilder ~]$ git config --global "trond.norbye@localhost"
[trond@cbbuilder ~]$ git config --global "Trond Norbye"
[trond@cbbuilder ~]$ mkdir compile && cd compile
[trond@cbbuilder ~/compile]$ repo init -u git:// -m released/2.1.1.xml
[trond@cbbuilder ~/compile]$ repo sync

Unfortunately there is a problem with one of the exceptions being thrown in Couchbase that cause a crash on SmartOS, so we need to "patch" one file. Its not hard, just add the following 3 lines of code:

[trond@cbbuilder ~/compile/ep-engine]$ git diff
diff --git a/src/couch-kvstore/ b/src/couch-kvstore/
index 931fb30..a48f271 100644
--- a/src/couch-kvstore/
+++ b/src/couch-kvstore/
@@ -515,6 +515,9 @@ void CouchKVStore::getPersistedStats(std::map &stats)
     char *buffer = NULL;
     std::string fname = dbname + "/stats.json";
+    if (access(fname.c_str(), F_OK) == -1) {
+        return;
+    }
     std::ifstream session_stats;
     session_stats.exceptions (session_stats.failbit | session_stats.badbit);
     try {

With that in place we can build Couchbase with the following command:

[trond@cbbuilder ~/compile]$ gmake PREFIX=/opt/couchbase

When make completes /opt/couchbase should contain a successful build of Couchbase 2.1.1, and at this time you should probably go ahead and create your startup scripts etc. We can try to emulate a cluster by starting 2 nodes on the same machine by running the following command:

[trond@cbbuilder ~/compile]$ cd ns_server
[trond@cbbuilder ~/compile/]$ ./cluster_run -n 2

And in another terminal we can build the cluster by executing:

[trond@cbbuilder ~/compile/]$ ./cluster_connect -n 2

Now tune your browser to the IP address of your server at port 9000 and enjoy your cluster.

Happy hacking!



  1. This comment has been removed by the author.

  2. I wrote a followup adding SMF scripts as

  3. Fantastic! I successfully tested this on the 'java:13.1.0' & 'base64:1.9.1' images.
    I put the entire build procedure into a bash script to make it easier to fire-and-forget:

  4. Very cool! I'm excited to try this out. Any plans to put in dtrace hooks?

    1. The plans are there, but I don't have a timeframe ;-)

  5. One update and one problem...

    I discovered that the URL for Google's 'repo' code has changed

    This is where I found that info:!topic/repo-discuss/4EsDRDRK5Lk

    With the new repo code I recently tried to run this procedure on an image of this type:
    17c98640-1fdb-11e3-bf51-3708ce78e75a base64 13.2.1 smartos smartmachine

    It is failing on:

    libtool: link: gcc -m64 -I/opt/local/include -L/opt/local/lib -shared -fPIC -DPIC -Wl,-z -Wl,text -Wl,-h -Wl, -o .libs/ src/.libs/libcouchstore_la-arena.o src/.libs/libcouchstore_la-btree_modify.o src/.libs/libcouchstore_la-btree_read.o src/.libs/libcouchstore_la-collate_json.o src/.libs/libcouchstore_la-couch_db.o src/.libs/libcouchstore_la-couch_save.o src/.libs/libcouchstore_la-crc32.o src/.libs/libcouchstore_la-couch_file_read.o src/.libs/libcouchstore_la-couch_file_write.o src/.libs/libcouchstore_la-db_compact.o src/.libs/libcouchstore_la-iobuffer.o src/.libs/libcouchstore_la-json_reduce.o src/.libs/libcouchstore_la-llmsort.o src/.libs/libcouchstore_la-tree_writer.o src/.libs/libcouchstore_la-mergesort.o src/.libs/libcouchstore_la-node_types.o src/.libs/libcouchstore_la-reduces.o src/.libs/libcouchstore_la-strerror.o src/.libs/libcouchstore_la-util.o src/.libs/libcouchstore_la-couch_index.o src/.libs/libcouchstore_la-os.o -Wl,-z -Wl,allextract ./.libs/librfc1321.a ./.libs/libbyteswap.a -Wl,-z -Wl,defaultextract -R/opt/local/lib -R/opt/local/gcc47/lib -R/opt/local/lib -R/opt/local/gcc47/lib -R/opt/couchbase/lib -L/opt/local/lib /opt/local/lib/ -L/opt/local/gcc47/lib /opt/local/gcc47/lib/ -lpthread -L/opt/couchbase/lib -licuuc -licudata -licui18n -lm -m64 -O2 -m64
    ld: fatal: file /opt/local/gcc47/lib/ wrong ELF class: ELFCLASS32
    ld: fatal: file processing errors. No output written to .libs/
    collect2: error: ld returned 1 exit status
    gmake[1]: *** [] Error 1
    gmake[1]: Leaving directory `/home/chaimk/compile/couchstore'
    gmake: *** [make-install-couchstore] Error 2

    I'm sure we want it to use '/opt/local/gcc47/lib/amd64/' which exists but instead it is using the 32 bit version which is odd since the -m64 flag is set everywhere, or so it seems.
    Any ideas?

    1. I would _GUESS_ that this is being included by /opt/local/lib/ These #$%^& .la files is nothing but trouble ;-) I would just remove that file, and it would probably work perfectly fine :-)

    2. I got rid of '/opt/local/lib/' and later had to remove '/opt/local/lib/' as well and the compilation competed.

  6. One more update. The 2013Q2 pkgsrc repository includes both of these versions of Erlang:
    erlang-16.1 (default erlang install candidate)
    However, compiling with erlang-16.1 does not create a working couchbase 2.1.1 so you need to specify erlang- in the list of dependencies on the server where you compile. Couchbase seems to run with either version of Erlang but it may be safer to use erlang- Note: If erlang-16.1 was already installed you must manually uninstall it. Simply trying to install erlang- will not cause a downgrade to take place.

    I have updated my script ( with all the changes that I mentioned here and above.

  7. Thanx for the installation instructions.
    I only had one problem, the Makefile.extra example in the gist on github says:
    OPTIONS += LDFLAGS="-R/opt/local/lib -L\$(PREFIX)/lib -R\$(PREFIX)/lib -m64"
    it should be without the \ after -L and -R .
    Took my 2hours to figure it out :) In your blog post it's correct, I guess it was a copy/paste mistake :)