Best In Class Atom Feed

Best In Class: Google Code Jam #1

2012-04-15T09:45:41Z

These Google challenges are typically a great suit for functional languages as the point is often to arrive at an algorithm which you can apply to the problem at hand. When reviewing the solutions, its interesting to compare high/low level languages as well as imperative/funtional languages. After having solved the problem, I went to find other solutions on Google and here is one I came across written in Java: Java Solution

The Java Solution

When reading the solution I notice the following:

The developer spent 1.5 hours on it
A large part of the code is spent on reading/writing files
The solution itself is handwritten - the program does not learn from the data

The Problem

You can read the official explanation here: Challenge #1

The short version is this: Google provides you with an unreadable language for which you must develop a translator. In order to make the translator you get 3 lines of pre/post translated text as well as 3 character mappings. The task is then to discover the remaining 24 mappings.

Repl Play

Since we're trying to produce mappings we need to construct a hash map where the keys are english characters and the values are the Googleresesian counterparts, so lets begin by loading the sample:

(def input ["ejp mysljylc kd kxveddknmc re jsicpdrysi"
            "rbcpc ypc rtcsra dkh wyfrepkym veddknkmkrkcd"
            "de kr kd eoya kw aej tysr re ujdr lkgc jv"])

(def google ["our language is impossible to understand"
             "there are twenty six factorial possibilities"
             "so it is okay if you want to just give up"])

In order to produce the initial mapping all we have to do is traverse each character in each string and store the relation:

(->> (map #(map (fn [i g] (hash-map i g)) %1 %2) input google)
     flatten
     (apply merge))
{\space \space, \a \y, \b \h, \c \e, \d \s, \e \o, \f \c, \g \v, \h \x, \i \d, \j \u, \k \i, \l \g, \m \l, \n \b, \o \k, \p \r, \r \t, \s \n, \t \w, \u \j, \v \p, \w \f, \x \m, \y \a}

The outer map walks the strings, the inner map walks the characters of each string, returning a hash-map with the key/val pair. All that's needed is now to manually add the mapping given by Google in the explanation of the problem:

(def dict (assoc *1 \z \q))

The *1 is simply short-hand for "the last result of evaluation", in this case the mappings. With the dictionary, dict, defined, we can look up simply by calling it as a function

(dict \a)
> \y

Here we see that an "a" translates to a "y". To make a translate function all we have to do is call dict on every character and then return the resulting sequence of characters as a string:

(defn translate [in]
  (->> (map #(dict %) in) (apply str)))

Now all of the work is basically done and all we need to do is apply our translate function to the dataset and properly format our output:

(def input (-> "A-small-practice.in" reader line-seq rest))
> input

The input file to Google is cast to a reader then read as a sequence of lines, dismissing the first line which simply contains an integer which we don't need.

With a sequence of strings in input we could simply run (map translate input) and that would produce the correct translation, however to properly format the output we need to add an index and some text:

(doseq [[idx txt] (map #(vector %1 %2) (range) s)] 
                          (println (str "Case #" (inc idx) ": " (translate txt))))
Case #1: our language is impossible to understand
Case #2: there are twenty six factorial possibilities
Case #3: so it is okay if you want to just give up
Case #4: xoggk d yl wxo vkkvgo ekso uyl wonw sywy
Case #5: xkf yto akj
Case #6: yabba dabba yabba dabba yabba dabba yabba dabba yabba dabba yabba dabba yabba dabba yabba dooooooooo
Case #7: a b c d e f g h i j k l m n o p q r s t u v y w x y z now i know my abcs
Case #8: next time wont you sing with me
Case #9: for those who speak in a tongue do not speak to other people
Case #10: nobody understands them since they are speaking mysteries in the spirit
Case #11: this is so exciting i have to go the bathroom
Case #12: it was the best of times it was the blurst of times
Case #13: let lips do what hands do
Case #14: this here is gunpowder activated twenty seven caliber full auto no kickback nailthrowing mayhem
Case #15: i have bested fruit spike and moon now i shall best you the guy
Case #16: oh hai im in ur computer eating your cheezburgers and googleresing your textz
Case #17: an eye for an eye and a pigeon for a pigeon
Case #18: all your base are belong to error the spoony bard
Case #19: you pissed off the chicken lady
Case #20: now is the summer of our lack of discontent
Case #21: by the pricking of my thumbs something wicked this way comes
Case #22: in a world of direwolves and lions sometimes the rarest creature is a friend
Case #23: greetings cheese popsicle the number you have dialed is currently out of porkchops
Case #24: why do programmers always mix up halloween and christmas
Case #25: im commander shepard and this is my favorite problem on the google code jam
Case #26: f of two equals f of one equals one
Case #27: for i between three and n f of i equals f of i minus one plus f of i minus two
Case #28: how are you holding up because im a potato
Case #29: dr zaius dr zaius dr zaius dr zaius ooooooooooooh dr zaius
Case #30: whoooooooooooooooooooaaaaaaaaa i know c plus plus

Conclusion

Working in a high-level language allow you to express your thoughts very quickly and succintly. Over time, working with high-level languages, changes the way you think about problems allowing you more easily to see the bigger picture. To this end Clojure has been an excellent companion.

The above program took only a few minutes to write and never made it as far as a file, it was simply written in the repl while playing with the data.

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: Building a Social Media site

2011-01-09T10:30:26Z

Preface

After many requests I've decided it would be a good idea to build a Clojure Driven Social Media Community website using some of the hottest tools available today to help us pull together. If you've read my old "Reddit Clone in 10 minutes and 91 lines of Clojure" this is on a very similar thread, however the core is now 98 lines and the tools have been updated. Lets get to it.

Frontpage

We begin by simply getting a webserver (Jetty) up and running and set to render our frontpage. One of the great things about developing webapps using Ring/Moustache is that you can construct your source such that when loaded on your development machine, it'll spawn a Jetty instance, but when loaded as a war file it will run as a Servlet within Tomcat. This makes for insanely rapid development/deployment. First, we'll start building our namespace which contains all of our web templates - I usually start by building a master page, which all (most) other pages are derived from:

HTML

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Clojure Driven Social Site</title>
    <script type="text/javascript" src="/scripts/jquery.js"></script>
    <script class="import" type="text/javascript"></script>
    <link rel="stylesheet" href="/css/main.css" type="text/css" media="screen" />
    <link class="style"  rel="stylesheet" type="text/css" media="screen" />
  </head>

  <body>
    <div class="header">
      <img src="/images/logo.jpg"/>
    </div>

    <div id="content">
    </div>

    <div class="footer">
      <a href="/">home</a>
      <a href="/submit">submit a link</a>
      <a id="loginout"/>
    </div>

  </body>
</html>

This HTML serves as a skeleton where Enlive will inject the dynamic components. Notice the places holders for scripts, styles and content which Enlive will look for. What we want now, is a function which takes 3 arguments: The scripts and styles to be loaded in head and the content for the div tag with that id. In addition, we need to pass a 4th argument, which is the user session so that the Login link, becomes a Logout link if the user is already logged in - Enlive makes this trivial:

(ns socialsite.templates
  (:use net.cgrand.enlive-html))

(deftemplate page "page.html" [session styles scripts cnt]
  [:link.style]   (clone-for [style styles]
                    (set-attr :href style))
  [:script.import] (clone-for [script scripts]
                    (set-attr :src script))
  [:a#loginout]   (content
                   (if (seq (:nick session))
                     {:tag :a, :attrs {:href "/logout"}, :content ["logout"]}
                     {:tag :a, :attrs {:href "/login"}, :content ["login"]}))
  [:div#content]  (content cnt))

As you can see, deftemplate defines a template with the name 'page' based off the HTML file page.html and takes our 4 arguments which we can use in the body. In the body I use clone-for whenever I need to duplicate an item a certain number of times, set-attr when I need to set attributes and content when Im setting content. Enlive is fairly well documented, but for now all you need to know is that clone-for works exactly like for, so the above says "clone for every style in styles" etc.

If you're new to Enlive you're probably wondering about the way Im setting the login/logout text and its strictly a performance concern. Instead of passing an html-snippet, Im passing Enlives native datastruct, here's how I arrived at it:

socialsite.templates> (html-snippet "<a href="/login">Login</a>")
({:tag :a, :attrs {:href "/login"}, :content ("Login")})

Simply replace the lists with vectors or quoted lists and put that in your templates. With the template in place, we can write our handler:

(defn view-frontpage
  [r]
  (->> (page (:session r) nil nil "Welcome")
       response))

The call to (page) returns the HTML as a seq of strings, passing that to response returns a hashmap which Ring understands. All thats left is defining a route to serve this and launch the server:

(def routes
  (app
   (wrap-file "resources")
   [""] view-frontpage))

(doto (Thread. #(run-jetty #'routes {:port 8080})) .start)

Notice that Im passing the routes argument as a var using #'. This forces Clojure to re-read the value on every read, meaning if I dynamically redef my routes while Im developing the results will show in the browser on the next reload - a must for interactive development. Here's the result:

Looks good - Partly because I used the wrap-file middleware which loads my images/css/scripts etc. You'll notice a big difference from my original Reddit clone in that I have now achieved a 100% separation between code and HTML. Originally I used Hiccup which is a bit like PHP for Clojure in that you seamlessly mix HTML and code. If you want to dig deeper with Enlive, I suggest reading this tutorial.

Database

My original clone persisted everything in memory which was conveniet at the time. However with the release of ClojureQL 1.0.0 it seems fitting to work with a real db backend. So here's what you can do if you have MySQL installed:

mysql> create database social;
mysql> grant all privileges on social.* to "social"@"social" identified by "social";
mysql> CREATE TABLE `posts` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `title` varchar(255) NOT NULL,
  `upvotes` int(11) NOT NULL DEFAULT '0',
  `downvotes` int(11) NOT NULL DEFAULT '0',
  `submitter` int(11) NOT NULL,
  `url` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;
mysql> CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `fullname` varchar(255) NOT NULL,
  `nick` varchar(255) NOT NULL,
  `password` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
mysql> insert into posts (title,url,submitter) VALUES ("ClojureQL 1.0.0 released!", "http://clojureql.org", 1);
mysql> insert into users (fullname,nick,password) VALUES ("Lau Jensen", "Lau", "social");

That will give you a new database called social, a user/pass social/social and two tables posts and users as well as your first user. As a convenience for you guys, I've put this SQL statement in the INSTALL.sql file in the Git repo linked at the bottom. (ps: This is not how you should design a production database!)

Now we need to render the post(s) on the frontpage. In order to do that, we need to join the two tables together and sort them on the amount of upvotes each post has - ClojureQL makes this easy: First we define our connection object and reference our tables

(def db   {:classname   "com.mysql.jdbc.Driver"
           :subprotocol "mysql"
           :user        "social"
           :password    "social"
           :subname     "//localhost:3306/social"})

(def posts (table db :posts))
(def users (table db :users))

Then we simply emit the HTML as before:

(defn view-frontpage
  [r]
  (->> (frontpage @(-> (join posts (project users [:nick])
                             (where (= :posts.submitter :users.id)))
                       (sort [:upvotes#desc])))
       (page (:session r) ["/css/frontpage.css"] nil)
       response))

Now the frontpage snippet gets an ordered list of all the posts, which we just need to inject into our skeleton. First the skeleton:

<body>
  <div class="post">
    <div class="voting">
      <div class="upvote">
        <img src="/images/up.png"/>
      </div>
      <div class="votes"/>
      <div class="downvote">
        <img src="/images/down.png"/>
      </div>
    </div>
    <div class="title"/>
    <div class="submitter"/>
  </div>
</body>

This is quite simple. We have 2 buttons to use for voting a title and a submitter. To fill in these tags we will use another of Enlives features namely the snippet, which allows me to pass a chunk of Enlive-HTML to a template:

(defsnippet frontpage "frontpage.html" [:body :> any-node]
  [posts]
  [:div.post]
  (clone-for [{:keys [id title url nick upvotes downvotes]} posts]
    [:div.votes]     (-> (- upvotes downvotes) str content)
    [:div.title]     (content
                      {:tag :a, :attrs {:href url}, :content [title]})
    [:div.submitter] (-> (str "submitted by " nick) content)))

Let see what we've got:

Perfect. Now we need to make the voting work, which unfortuntely involves Javascript - Although I like JS it would be nice with a ClojureDSL for that as well:

<script type="text/javascript">
    function vote(id, uri, output) {
        $.ajax({url: uri, type: 'POST', data: {id: id},
                success: function(data) {
                    if (data.substring(0, 2) == "OK") {
                        $(output).html(data.substring(4));
                    } else {
                        Alert("Your vote could not be submitted");
                    }
                }
               });
    }

    function attach_voter(objs, dir) {
        $(objs + ' img').each(function() {
            $(this).click(function() {
                vote($(this).parent().parent().attr("pid"),
                     dir,
                     $(this).parent().parent().find("div.votes"));
            }).css("cursor", "pointer");
        });
    }

    $(function() {
        attach_voter("div.upvote", "/vote/up");
        attach_voter("div.downvote", "/vote/down");
    });
  script>

This is a pretty naive implementation:The function vote calls an URL in the backend, receives a response and updates the votes count with whatever it gets back. The function attach_voter finds all image tags under the select objs and adds an onclick event which simply calls vote. This is clever, because if you only want to allow your users to vote once, simply let the backend keep track of the votes and it will always reply with the correct count.

Voting

Now we can add the backend implementation:

(defn vote
  [{id :form-params} direction]
  (let [predicate (where (= :id (get id "id")))]
    (update-in! posts (where predicate)
     (if (= "up" direction)
       {:upvotes   (inc (-> (select posts (where predicate))
                            (pick! :upvotes)))}
       {:downvotes (inc (-> (select posts (where predicate))
                            (pick! :downvotes)))}))
    (-> (format "OK: %s" (-> (select posts (where predicate))
                             (aggregate [["(upvotes - downvotes)" :as :sum]])
                             (pick! :sum)))
        response)))

Basically this function takes the arguments supplied by the JS and a direction in which to vote, ie. up or down. Then it updates the row in posts where predicate is true. If direction is "up" it increments the upvotes, otherwise increments the downvotes. The function pick! is particularily helpful as it derefences our table, checks if theres a single hit and then pulls out the value of the keyword supplied as the argument - this lets us call inc directly on the result. Once thats done, we want to return the score which is simply upvotes minus downvotes - notice how ClojureQL easily lets me slip in a string and alias that.

Now we need moustache to call this function when somebody votes. As you saw in the JS voting is done by POSTing to either /vote/up or /vote/down, here's how to catch that with Moustache:

(def routes
  (app
   (wrap-file "resources")
   [""]               view-frontpage
   ["vote" direction] (wrap-params (delegate vote direction))))

In a production system, I wouldn't do it like this - I would use the same URI for both calls and carry the direction in the POST data. But this lets me show off more of Moustaches syntax and the delegate function - You could use partial, but then you would have to re-arrange the argument order. The ["vote" direction] will match "/vote/*" where * can be anything except another slash - whatever is passed goes into the var direction.

Submitting posts

Now we need to allow users to submit posts, which follows the outline above:

Make a HTML skeleton
Define an Enlive snippet
Write a backend handler

Since the first 2 are exceedingly trivial, I'll simply demonstrate the handler:

(defn accept-submission
  [req]
  (let [params (-> req :form-params)
        user   1];(-> req :session :id)]
    (conj! posts {:title     (params "title")
                  :url       (params "url")
                  :submitter user})
    (response "OK")))

Because I haven't written the 'login' functionality yet, I've hardcoded the user ID to 1. Then I simply conj! the hasmap containing the values from the post onto posts and reply "OK". Normally, you would want to filter/sanitize your results and add some error handling but thats beyond the scope of this post. Next you might want to group both the POST and GET replies on the URI "/submit", here's how to do it:

(def routes
  (app
   (wrap-file "resources")
   [""]               view-frontpage
   ["submit"]         {:get  view-submission-page
                       :post (wrap-params accept-submission)}
   ["vote" direction] (wrap-params (delegate vote direction))))

Passing Moustache a map, instead of a function lets you group related functionality on the same URI.

Logging in/out

All we need now is to give users the ability to log in and out. In order to do that, we need to perform the usual steps ending with the handler:

(defn authenticate-user
  [{params :form-params}]
  (let [user  @(select users (where (and (= :nick     (params "usr"))
                                         (= :password (params "psw")))))]
    (if (= 1 (count user))
      (assoc (redirect "/") :session (first user))
      (redirect "/login?err=true"))))

Ring has a middleware called wrap-session, which when enabled keeps a :session key in every request which you can use to store information about each session. Here I simply select the row from the database which has the correct username/password combination and if I get a hit then I associate that record with the session - If I dont get a hit, I redirect the client to the login page and show an error. This brings us to the final piece of the puzzle...

Middlewares

One of the great delights of working with Ring is its middlewares. They act as functions which work on each request and then passes the request down the line, possibly modified. One middleware which we definitely need now, is a security middleware which rejects unauthorized users from pages they're not allowed to see, here's a naive way to achieve that:

(defn wrap-security
  [app]
  (fn [req]
    (let [uri (:uri req)]
      (if (or (-> req :session :nick seq)
              (= uri "/")
              (= uri "/login"))
        (app req)
        (redirect "/?err=Not logged in")))))

If the user is logged in or is trying to access one of the two public URIs the application handles the request as normally, but if not it redirects to the frontpage. The entire middleware stack for this app looks like so:

(def routes
  (app
   (wrap-file   "resources")
   (wrap-reload '[socialsite.templates])
   (wrap-session)
   (wrap-security)

   [""]               view-frontpage
   ["logout"]         logout
   ["login"]          {:get view-login-page
                       :post (wrap-params authenticate-user)}
   ["submit"]         {:get  view-submission-page
                       :post (wrap-params accept-submission)}
   ["vote" direction] (wrap-params (delegate vote direction))))

wrap-file allows me to serve static files from the resources directory, wrap-reload reloads the Enlive templates on every pageview, very handy for development, very no-no for production. The final two I've already commented on - Lets see what we ended up with:

There you go, your own Clojure Driven Social Media website to help keep the community engaged and connected :)

The source code for this app is on Github, to launch it simply follow these simple steps:

$ git clone http://github.com/LauJensen/SocialSite.git
$ cd SocialSite
$ mysql -u root -p < INSTALL.sql 
Enter password: *******
$ cake repl
user=> (use 'socialsite.core)
nil

The INSTALL.sql file will create the social/social user/pass on your local MySQL installation as well as the social database which is populates with two tables. Once you issue the use command in the repl, you can access the site on http://127.0.0.1:8080 - Here you can login with my user "Lau" / "social" - Enjoy!

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: ClojureQL - 1.0.0 now in beta

2010-11-18T01:50:40Z

Preface

Over 2.5 years ago I set out to develop THE database abstraction library and after 2.5 years I felt painted into a corner, so I deleted the whole thing and re-wrote it in less than 3 weeks (with a little help from my friends).

The ClojureQL that Im presenting today is the public 1.0 beta release and hopefully it will meet all of your production needs. I've prepared this small screencast to get you started and I'll discuss the details below - If the quality of the video is unsatisfactory please download the original file from Vimeo instead of using the Flash player:

ClojureQL (Beta) from Lau Jensen on Vimeo.

ClojureQL 1.0 - Bringing Relational Algebra, back to Relational Databases

ClojureQL 1.0 is has been a long time coming, but I feel confident that the interfaces and primitives of this implementation are whats needed to provide the ultimate SQL integration experience. ClojureQL is similar in intent to Rubys Arel. The current implementation is the refined experience of the previous 2.5 years boiled down into about 700 lines of pure Clojure.

If you want the finer details of why the first version of ClojureQL got scrapped, read this post. As I was designing this re-write I was approached by Justin Balthrop (ninjudd), of Cake fame, who said "What you've got is 90% of the way towards getting true composability, relational algebra (RA) primitives will get you all the way". I read up on RA and looked at some examples and knew that Justin was on to something, RA was definitely the key to making a truly composable library (well, that and the lack of macros), so off we went. The ClojureQL you've just seen demonstrated is totally RA based.

Getting started

To get ClojureQL either include these artifacts in your project:

Cake / Lein:

[clojureql "1.0.0-beta1-SNAPSHOT"]

Maven:

<dependency>
  <groupId>clojureql</groupId>
  <artifactId>clojureql</artifactId>
  <version>1.0.0-beta1-SNAPSHOT</version>
</dependency>

Or simply clone the Github Repo.

git clone git@github.com:LauJensen/clojureql.git

If you run into trouble that you think you yourself might be the cause of, you can get help on #clojureql on irc.freenode.net where you'll find both myself and Justin. If you've uncovered an issue with ClojureQL itself, please report it on Github and include instructions on how to reproduce.

Documentation

I'll outline the various functions/interfaces that you need to know below, but generally you should refer to the Github page which will serve as documentation henceforth!

Shadows

In a few instances ClojureQL shadows functions from clojure.core and in two of these instances they behave different:

conj! - In clojure.core this works on transients, in ClojureQL it only works on tables
disj! - Same as above

Besides these 2 we shadow take, drop and sort but these all work as you would expect. If you call them on a Table they return a new table, if you call them on a collection they pass your arguments on to the clojure.core version. No mathematical operators or and/or are shadowed. You can make your code clearer by using ClojureQL :as cql.

The interfaces

There are only a few functions that you need to get comfortable with in order to start composing powerful queries:

Table

Table lets you construct a reference to a table in a SQL database. As its arguments you can specificy the connection-object (keyword or hashmap, required), the table name (keyword or hash-map if you're aliasing).

Deref

When you dereference a table object (either (deref table) or @table) the query of that table is compiled and executed.

Select / Where

The select statement is not to be confused with SQL SELECT command. Select actually corresponds to WHERE from MySql. There are 2 ways in which you can build a select statement.

1) Using the 'where' macro:

clojureql.core> (where (and (> :id 5) (<= :wage 100)))
"((id > 5) AND (wage <= 100))"

Which accepts column references as keywords, standard mathematical operators and and/or. Or you can:

2) Write out the datastructure which 'where' compiles manually by suffixing every function with a star* (this is the only thing which 'where' does for you). To do that, you must first use/require clojureql.predicates:

clojureql.core> (and* (>* :id 5) (<=* :wage 100))
"((id > 5) AND (wage <= 100))"

Rename

Rename is helpful when you need to alias a column after table construction. Here are 2 variants of renaming, both valid:

clojureql.core> (-> (table nil :users)
                    (project [:id :name])
                    (rename {:id :idx})
                    to-sql)
"SELECT users.id,users.name FROM users AS users(idx,name)"
clojureql.core> (-> (table nil :users)
                    (project [[:id :as :idx] :name])
                    to-sql)
"SELECT users.id AS idx,users.name FROM users"

Note that a nested vector always denotes aliasing as does hashmaps. To rename a table, do the following:

clojureql.core> (-> (table nil {:users :u1})
                    to-sql)
"SELECT * FROM users u1"

Join

Join compiles to a regular join using either USING or ON depending on your predicate. As arguments it takes 2 tables and a predicate. If your predicate is a keyword it will compile to USING(keyword), if its a predicate built with the 'where' macro, it will be taken literally:

clojureql.core> (-> (join (-> (table nil :t1) (project [:c1]))
                          (-> (table nil :t2) (project [:c2]))
                          :id)
                    to-sql)
                    
"SELECT t1.c1,t2.c2 FROM t1 JOIN t2 USING(id)"

And here's the equivalent join with a custom predicate

clojureql.core> (-> (join (-> (table nil :t1) (project [:id :c1]))
                          (-> (table nil :t2) (project [:id :c2]))
                          (where (= :t1.id :t2.id)))
                    to-sql)
                    
"SELECT t1.id,t1.c1,t2.id,t2.c2 FROM t1 JOIN t2 ON (t1.id = t2.id)"

It's also a possibility to join a table unto itself using aliasing, like this next example where we find an employees manager:

clojureql.core> (-> (select (table {} {:employees :p})
                            (where (= :name "John")))
                    (join (table {} {:employees :b})
                          (where (= :p.manager :b.id)))
                    to-sql)
"SELECT p.*,b.* FROM employees p JOIN employees b ON (p.manager = b.id) WHERE (name = 'John')"

Outer-join

Outer-join does pretty much as advertised, it compiles to an outer-join either LEFT, RIGHT or FULL. It takes as its arguments 2 tables, the type of join as a keyword (:left|:right|:full) and a predicate (again, either keyword or clause):

clojureql.core> (-> (outer-join (table nil :t1)
                                (table nil :t2)
                                :left
                                (where (= :t1.id :t2.id)))
                    to-sql)
                    
"SELECT t1.*,t2.* FROM t1 LEFT OUTER JOIN t2 ON (t1.id = t2.id)"

Aggregate

Aggregate lets you select aggregates from your tables in a composable fashion (they dont need to be defined up front). It simply takes a table and a collection of aggregates. All aggregates in ClojureQL follow this syntax for singular fields: :function/field and this for multiple fields: :function/field1:field2:fieldn. If you need to get even fancier, use strings they are supported literally:

clojureql.core> (-> (table nil :users)
                    (aggregate [:count/* :avg/salary])
                    to-sql)
"SELECT count(*),avg(users.salary) FROM users"
clojureql.core> (-> (table nil :users)
                    (aggregate [:count/* :max/income:expenses])
                    to-sql)
"SELECT count(*),max(users.income,users.expenses) FROM users"
clojureql.core> (-> (table nil :users)
                    (aggregate ["corr(x + y)"])
                    to-sql)
"SELECT corr(x + y) FROM users"

Take/Drop

Just as with your regular collections in Clojure, you can use take/drop on tables and they act exactly the same way:

clojureql.core> (-> (table nil :users)
                    (take 4)
                    (drop 2)
                    to-sql)
"SELECT users.* FROM users   LIMIT 2,2"

Are you wondering why it says LIMIT 2,2 and not 2,4? Well consider this:

clojureql.core> (->> (range 10) (take 4) (drop 2))
(2 3)

ClojureQL handles sequences exactly as in Clojure (btw. that all works in the same namespace).

Sort

Sort takes a table, a column to ORDER BY and a direction (keyword, :asc|:desc):

clojureql.core> (-> (table nil :users)
                    (sort [:id#asc])
                    to-sql)
"SELECT users.* FROM users ORDER BY users.id ASC"

Conj!

Conj! inserts a record or records into the table. A record is simply a hashmap and you can put multiple hashmaps inside a vector:

clojureql.core> @(-> (table db :users)
                     (project [:id])
                     (conj! {:name "Frank"}))
({:id 1} {:id 4} {:id 5} {:id 6} {:id 7} {:id 8} {:id 9} {:id 10} {:id 11} {:id 12})

Disj!

disj! removes a rename which matches the supplied predicate:

clojureql.core> @(-> (table db :users)
                     (project [:id])
                     (disj!  (where (= :name "Frank"))))
({:id 1} {:id 4} {:id 5} {:id 6} {:id 7} {:id 8} {:id 9} {:id 10})

Update-In!

update-in! again does as advertised, it finds the row(s) matching the predicate and update the information on them. If no rows match the predicate a new one is created:

clojureql.core> @(-> (table db :users)
                     (project [:id])
                     (update-in! (where (= :name "Frank"))
                                 {:name "John"}))
({:id 1} {:id 4} {:id 5} {:id 6} {:id 7} {:id 8} {:id 9} {:id 10} {:id 13})

With-Results

With-results is an alternative to deref which lets you pass a body that will be lazily evaluated on the open resultset. ~~It takes a table first, then a placeholder for the results, then the body of code~~: Updated: It takes a binding vector, where you bind results to a table:

clojureql.core> (with-results [rs users]
                  (doseq [r rs]
                         (prn r)))
{:id 1}
{:id 4}
{:id 5}
{:id 6}
{:id 7}
{:id 8}
{:id 9}
{:id 10}
nil

If you pass the optional :fetch-size argument to the database object you code will be called on each chunk, allowing you to work with huge datasets.

With-Connection

If you've constructed a table with nil as the connection object, you can simply wrap any database interaction with that table in (with-connection db-obj ..).

Final words

I would like to thank both Meikel Brandmeyer and Nicolas Buduroi for helping me learn those tough lessons on the first, now scrapped, version of ClojureQL. I would also like to thank Christophe Grand who served as an inspiration to me in Frankfurt and finally a huge thank you goes out to Justin Balthrop who has fought the good design fight with me on every step of the way towards ClojureQL(2) 1.0!

Also my thanks go out to Chris Houser who helped review the screencast, provide feedback and even contributed a slogan:

"ClojureQL - bringing Relational Algebra back to Relational Databases"

I hope you all take this library out for a spin and send some feedback (good, bad, cruel, performance related, all is accepted) back to me. Thanks and have fun!

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: ClojureQL - Revolutions

2010-11-03T05:19:18Z

Preface

Two and half years ago I approached fellow Clojurian Meikel Brandmeyer to learn whether or not he would be interested in developing an advanced Clojure version of SchemeQL. We had 2 big dreams:

Being able to construct SQL statements using only pure Lisp
...and having those statements be valid on any backend (MySQL, Postgres, Oracle, you name it)

Both of us were just learning Clojure so the natural first step was to hammer out the fatal word "defmacro". After a while of stepping on each others toes we decided to modularize the project, so that one of us could concentrate on the 'frontend' (user input) and the other on the backend (compiler output). Our first cut on the frontend received some insightful critique from Zef here, which we took to heart and started improving upon - The problem was of course, the macros. Meikel volunteered to re-write the frontend to a system which relies on a Monad type of queries, but he quickly realized how much work this entailed and simply gassed out, so ClojureQL was at a stand still.

The backend was quite advanced and even from its early days it could compile our own AST format into valid SQL for MySql, Sqlite, Derby and PostgresSQL. There were however two problems with this model.

It takes more work than any of us have time for, to maintain a myriad of backends
We started out too young.

Then what happend?

Christophe happend

Christophe Grand is somewhat of an expert when it comes to designing DSLs. He's designed Enlive and Moustache, trained professionals in DSL design across Europe and even educated the Americans on DSLs - So lets just say, he knows what he's talking about.

Besides being a borderline genius, Christophe is also my co-instructor at the Conj Labs events and recently we just wrapped up such an event in Frankfurt. After 4 fast paced days, covering oceans of material, speaking, eating and drinking Clojure non-stop we finally ended up at a small round table in Frankfurt International Airport. Here we had about 4 hours just to chit/chat and since its impossible to tire of chatting about Clojure I wanted Christophes take on the direction ClojureQL was going in.

Christophe looked down for a moment while he pondered ClojureQLs architecture and finally he looked up and said "I don't think I would have begun the way you did, I wouldnt go in that direction". In my mind that basically translated into "rm -rf clojureql/", so he went on using yet another of his 70s shows analogies

"Do you remember that 70s show about a Robot that acted like a Human? And how it was like 80% or 90% perfect human, but all anybody ever saw was the 10% that it was lacking? That's how it's going to be with ClojureQL. You might be able to make the backend stuff 90% transparent to the users, but they will see the 10% that's missing"

So to me that literally meant "rm -rf clojureql/", so thats what I did.

Then what happend?

A simpler design

Deleting ClojureQL and picking it right back up again has been pleasant. Thankfully I've learned a lot about Clojure and DSL design since we put in the first lines of code 2.5 years ago, so its been a very rapid process of prototyping ideas and it's ongoing! Ultimately here's what I want to end up with:

A simple, Clojuresque, interface for working with databases
A compiler which works only on datastructures, not macros
A compiler which lets users seamessly escape to raw SQL statements, for those odd syntax quirks of the various DBMs

It's not done and here's what's driving the effort:

I'm looking through applications I've developed trying to find better ways of expressing them from top to bottom
I'm requesting feedback from anybody and everybody who wants slick SQL integration for Clojure

Current Status

So with the fancy backend compiler that targets multiple DBMs out of the way, I am free to concentrate about the interface. Here's what I got jotted down so far:

Tables

Tables are at the heart of any SQL abstraction layer, so in ClojureQL they're first class citizens. Once you have defined a link to a table, you don't need to worry about specifying colums for every query, database connections, etc. Simply grab the name and act on it:

user> (def users (table db-con :users [:id :name :title]))
#'user/users

A link to a table is like a reference type, it isn't its value but it just holds it. Dereferencing it at any point in time peeks at that value:

user> @users
({:id 1, :name "Lau Jensen", :title "Dev"} {:id 2, :name "Christophe", :title "Design Guru"} {:id 3, :name "sthuebner", :title "Mr. Macros"})

Typically in Clojure you can have two variations on a function. Take for example 'conj' which conjoins an item unto a copy of the original collection and compare it to 'conj!' which mutates a transient. In the same way ClojureQL provides conj! and disj! which both 'mutate' the table:

user> @(disj! users {:id 3})
({:id 1, :name "Lau Jensen", :title "Dev"} {:id 2, :name "Christophe", :title "Design Guru"})"Mr. Macros"})

user> @(conj! users {:name "Frank" :title "Dev"})
({:id 1, :name "Lau Jensen", :title "Dev"} {:id 2, :name "Christophe", :title "Design Guru"} {:id 4 :name "Frank" :title "Dev"})

Notice how the returns from both calls have to be dereferenced to be read. This makes it easier to chain calls with the -> macro. In addition there are 2 less interesting helpers which I'll just mention as well:

user> (take users 1)
({:id 1, :name "Lau Jensen", :title "Dev"})

user> (-> (disj! users (either (= {:id 3}) (= {:id 4})))
          (sort :desc))
({:id 2, :name "Christophe", :title "Design Guru"} {:id 1, :name "Lau Jensen", :title "Dev"})

The first example is a simple LIMIT statement, but the second shows off a chained call and also the special predicate compiler which returns a statement that matches either id=3 or id=4.

Finally, you have some freedom to do selects, here I'll demo both the escape and compiled version:

user> (select users (where "id=%1 OR id>=%2" 1 10))
({:id 1, :name "Lau Jensen", :title "Dev"})

user> (select users (where (either (= {:id 1}) (>= {:id 10}))))
({:id 1, :name "Lau Jensen", :title "Dev"})

Use version1 if you have some very special syntax which the compiler cant generate and use version2 if you prefer to have a cleaner expression of your query. Finally, I provide a way of expressing joins which turns out to be quite simple since the table objects know about all the colums:

user> (join users salary #{:users.id :salary.id})
({:wage 100, :title "Dev", :name "Lau Jensen", :id 1} {:wage 200, :title "Design Guru", :name "Christophe", :id 2} {:wage 300, :title "Mr. Macros", :name "sthuebner", :id 3})

Everything you see here and (especially) below is still in flux. It might change as this is a design phase, but I'm working quite quickly so I hope to cement a good design soon - But again, input is more than welcome!

Predicates

Building predicates is the tricky part of SQL abstraction and also where we got burned the most in the early versions by relying heavily on macros - This time around I dare you to find a defmacro in the abstraction code! :)

Basically what I have so far is an expression compiler and some helper functions to make that compiler more accessible. Here's how it works:

user> (compile-expr [:or [:eq {:a 5}] [:gt {:b 5}]])
"(a = 5 OR b > 5)"

^{(thanks to Chousuke from #clojure for inspiration on the data-design)}

The compiler simply walks a vector of vectors looking for special keywords denoting actions and colums. Colums have a little extra syntax in that they can be aggregates denoted by a period, like so:

user> (compile-expr [:or [:eq {:a 5}] [:gt {:avg.b 5}]])
"(a = 5 OR avg(b) > 5)"

But of course, any sane person would rather write SQL directly than use such a syntax so here's the same code with the added layer of sugar:

user> (compile-expr (either (= {:a 5}) (> {:avg.b 5})))
"(a = 5 OR avg(b) > 5)"

As a user you should never call compile-expr yourself, I only show it here for demonstration purposes. Typically you'll use some function that requires matching and it will call compile-expr for you. Two obvious example are disj! and where.

user> (where (both (> {:a 5}) (<= {:b 10})))
"WHERE (a > 5 AND b<=10)"

user> (where (both (> {:a 5}) (either (<= {:b 10}) (> {:avg.y 100.0}))))
"WHERE (a > 5 AND (b<=10 OR avg(y) > 100.0))"

Both of these can be passed to a (select user ...) call, or the expression can be fed to disj! without the (where) call.

Onwards!

So, if you're very sad and disgruntled that I deleted ClojureQL 0.9.7 Frontend 2.0 and you really wanted to see that make it all the way to version 1.0 don't dispair! The repo is still live on Gitorious with both the master and frontend-2.0 branches so if you have mountain moving power, by all means, move it!

If you're psyched about this type of SQL abstraction and want to contribute there's a number of ways you can do it:

Send me design specs if you have good ideas
Attach patches if you have Clojure-fu
Convert some of your programs to use ClojureQL and get in touch with me if something doesn't work
...and tell me if something is so great that I absolutely musn't delete it

If you're content just experimenting I recommend cloning the repo (which is now back on Github), adjusting the 'db' var in core.clj to match your local MySQL installation and then load the file core.clj and run '(test-suite)'.

Hope you enjoyed the read.

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: Taking Uncle Bob to school

2010-10-08T09:23:11Z

Preface

Uncle Bob is a man shrouded in mystery, a man of opinions and contradictions - a developer and agile enthusiast. I hadn't heard about Uncle Bob until recently when he popped up on my radar as he was giving a talk about Clojure in Amsterdam. I asked around and learned that he wrote a book called "Clean code" and that he likes Ruby, which just makes no sense to me at all - How do you unify a tool as unstable, unpredictable, untamed as Ruby (second only to Perl) with Clean Code? Beats me, but maybe I should read his book.

Recently I saw that Uncle Bob had written an Orbit Simulator, so I figured I'd read through it point out a few areas where the code could be more idiomatic, to help out Bob and other Ruby developers looking to take on Clojure.

Orbital Overview

The Orbit Simulator is found here. Its very easy to run as Uncle Bob has made the package compilable to uberjar and thereby executable. Basically it generates about 500 objects all located at random positions around the sun and then simultates their orbit around the sun, checking for collisions, applying forces, that kind of thing. This particular simulation performs awfully because from top to bottom its expressed using functional sequences, which is great. You can optimize this a bit, but not a lot, without loosing the purely functional but for a more performant version you'd have to rewrite the core to something a little less functional, which would defeat the purpose of the demo.

Idiomatic Lisp

There are many things specific to Clojure that I want to run through, but before we get to that we need to examine the very basics: Our Lisp heritage. As the fantastic Mr. Fogus talked about in his blogpost on Community Standards, it's important not to alienate yourself from the community around you by inventing new odd personlized formats for your code. It will undoubtedly mean that a great deal of Clojurians will have a hard time looking at your code. Notice how Mr. Fogus uses his own brand of comma/period placement in that blogpost, that's how silly he thinks homemade Lisp formats are and I agree.

Uncle Bob has unknowingly put himself on Mr. Fogus bad side, because he writes Lisp like so:

(defn random-position [sun-position]
  (let [
        r (+ (rand 300) 30)
        theta (rand (* 2 Math/PI))
        ]
    (position/add sun-position (position/make (* r (Math/cos theta)) (* r (Math/sin theta))))
    )
  )

Now to the untrained eye I dont know what that looks like. But to an eye trained on Lisp, it looks awful. Here's the idiomatic version:

(defn random-position [sun-position]
  (let [r     (+ (rand 300) 30)
        theta (rand (* 2 Math/PI))]
    (position/add sun-position (position/make (* r (Math/cos theta)) (* r (Math/sin theta))))))

Thats the kind of mistake that is easy to make when you take on a new language, however I've been kind of enough to clean up every instance in his codebase :)

Namespaces

Lets have a look at namespace declarations. In Clojure we declare namespaces using the very versatile ns macro. Here's Uncle Bobs code:

(ns orbit.world
  (:import (java.awt Color Dimension)
    (javax.swing JPanel JFrame Timer JOptionPane)
    (java.awt.event ActionListener KeyListener))
  (:use clojure.contrib.import-static))

(import-static java.awt.event.KeyEvent VK_LEFT VK_RIGHT VK_UP VK_DOWN)
(require ['physics.object :as 'object])
(require ['physics.vector :as 'vector])
(require ['physics.position :as 'position])

Although you see the ns macro doing both imports and uses here, it actually does require as well so here's a more idiomatic version:

(ns orbit.world
  (:import (java.awt Color Dimension)
           (javax.swing JPanel JFrame Timer JOptionPane)
           (java.awt.event ActionListener KeyListener))
  (:use clojure.contrib.import-static)
  (:require [physics.object   :as object]
            [physics.vector   :as vector]
            [physics.position :as position]))

(import-static java.awt.event.KeyEvent VK_LEFT VK_RIGHT VK_UP VK_DOWN)

Concurrency Semantics

The code does a good job picking which places are immutable and which can mutate and be shared between threads. Particularily the world itself is a reference type as it must be exposed to the renderer. Unfortunately Uncle Bob decided to use the STM (Refs) for this job, which is a sub-optimal choice and here's why:

Synchronous coordinated change: Refs

Refs are great for when you need coordinated change, ie. the ability to look at two things at once, coordinating their change of state. When you need coordination there's no way around refs, but when you don't you should really find another way as the STM adds quite a bit of overhead to every transaction.

Synchronous uncoordinated change: Atoms

For synchronous uncoordinated change, atoms are your best pick. They're fast, they automatically respin if need be and they enforce all of the concurrency safety switches you need to forget about race conditions. For simply exposing the world to the renderer, atoms will do just fine.

Unsynchronous uncoordinated change: Agents

When you want something to go off, into another thread and go about its business and you don't care when or how it happens and require no coordination, agents are you best friend. Changes are still atomic, however not instant from the view of the caller.

Rewriting for atoms

Atoms are only slightly different in use from the STM. They use swap! instead of alter and don't require the surrounding dosync. Lets start with Uncle Bobs code:

(defn update-world [world controls]
  (dosync
   (alter world #(object/update-all %))))

The first thing to notice, is that controls is being passed as an argument, which it doesn't need to be. I assume thats legacy code. Secondly, from the docs it's clear that alter takes a ref as its first argument and a function as its second. But what is not being leveraged in this piece of code is that you can add an arbitrary amount of parameters to that function, like:

(defn update-world [world controls]
  (dosync
   (alter world object/update-all)))

Rewriting that to atoms looks like so:

(defn update-world [world controls]
  (swap! world object/update-all))

Here's another example which makes it clear how much is gained by using the functions idiomatically

(defn magnify [factor controls world]
  (dosync
   (let [sun-position (:position (find-sun @world))
         new-mag (* factor (:magnification @controls))]
     (alter controls #(assoc % :magnification new-mag))
     (alter controls #(assoc % :center sun-position))
     (alter controls #(assoc % :clear true)))))

Written idiomatically, also using atoms looks like:

(defn magnify [factor controls world]
  (dosync
   (let [sun-position (:position (find-sun @world))
         new-mag      (* factor (:magnification @controls))]
     (swap! controls assoc
            :magnification new-mag
            :center        sun-position
            :clear         true))))

Looks better, performs better and is more idiomatic.

Idiomatic Conditionals

There are 3 main ways of doing conditionals in Clojure: cond, condp and case. The Orbit Sim uses cond rather heavily and in place where condp makes alot more sense. condp is like cond, but makes the code more readable when the predicate is the same for all dispatch routes. Here's the original code:

(defn color-by-mass [{m :mass}]
  (cond
   (< m 1)  Color/black
   (< m 2)  (Color. 210 105 30)
   (< m 5)  Color/red
   (< m 10) (Color. 107 142 35)
   (< m 20) Color/magenta
   (< m 40) Color/blue
   :else (Color. 255 215 0)))

Here we see, that all predicates have something to do with m being smaller than something else. In condp we have to swap the position of the arguments, but that still gives us cleaner code. Notice the implicit :else at the bottom:

(defn color-by-mass [{m :mass}]
  (condp > m
    1  Color/black
    2  (Color. 210 105 30)
    5  Color/red
    10 (Color. 107 142 35)
    20 Color/magenta
    40 Color/blue
    (Color. 255 215 0)))

In this case, the win is decent, but in the next example it's huge:

(defn- quit-key? [c]
  (= \q c))

(defn- plus-key? [c]
  (or (= \+ c) (= \= c)))

(defn- minus-key? [c]
  (or (= \- c) (= \_ c)))

(defn- space-key? [c]
  (= \space c))

(defn- trail-key? [c]
  (= \t c))

(defn handle-key [c world controls]
  (cond
    (quit-key? c) (System/exit 1)
    (plus-key? c) (magnify 1.1 controls world)
    (minus-key? c) (magnify 0.9 controls world)
    (space-key? c) (magnify 1.0 controls world)
    (trail-key? c) (toggle-trail controls)))

Style-wise I think Uncle Bob prefers to err on the verbose size, which is good in terms of clarity - And indeed his code is very easy to follow. However in this case I think he took it took far. My rewrite would look like this:

(defn handle-key [c world controls]
  (condp = c
    \q     (System/exit 1)
    \+     (magnify 1.1 controls world)
    \-     (magnify 0.9 controls world)
    \space (magnify 1.0 controls world)
    \t     (toggle-trail controls)))

That saves us about 15 lines of code and sacrifices little or no clarity. In the above example all the test-constants are literals so you could re-write this using case instead of condp and that would give you constant time dispatch. Here's the faster version which also has an implicit else value (nil), for when the user presses an unmapped key. If you dont have this, you'll see exceptions.

(defn handle-key [c world controls]
  (case c
    \q     (System/exit 1)
    \+     (magnify 1.1 controls world)
    \-     (magnify 0.9 controls world)
    \space (magnify 1.0 controls world)
    \t     (toggle-trail controls)
    nil))

Being a little minimalistic

There are two reasons that the code is so slow. The first is that fact that it allocates an insane amount of objects. For the collision test Uncle Bob allocates all possible combinations of the 500 objects on every single frame, which takes ages (1 second) - And later the GC has to come clean this up again. The second reason, is that the code is doing a gazillion lookups since everything is put in structs. For the most part, this makes a lot of sense in terms of readability, but in two cases I think it goes too far: position.clj and vector.clj. These 2 files do pretty much the same thing, which is operations on coordinates. Most of us understand [x y] and dont need {:x x, :y y} right? So if I was writing something like that, I'd keep it slim when dealing with coordinates.

This simplifies a lot of things like:

(defn origin? [p]
  (and
    (zero? (:x p))
    (zero? (:y p))))

Which then simply becomes:

(defn origin? [p]
  (every? zero? p))

(defn add [p q]
  (struct position
    (+ (:x p) (:x q))
    (+ (:y p) (:y q))))

Which becomes:

(defn add [[x1 y1] [x2 y2]]
  [(+ x1 x2) (+ y1 y2)])

And finally, some of the functions in vector are generating anonymous functions everytime they're called, which is also quite a slow-down. So to speed things up a little change

(defn distance [p q]
  (letfn [(square [x] (* x x))]
    (Math/sqrt
      (+
        (square (- (:x p) (:x q)))
        (square (- (:y p) (:y q)))))))

into

(defn distance [[x1 y1] [x2 y2]]
  (Math/sqrt
   (+
    (Math/pow (- x1 x2) 2)
    (Math/pow (- y1 y2) 2))))

And finally, be dynamic!

Whenever you're writing code in Clojure make sure you have a good interactive development setup. Coding, compiling to uberjar, running is not a good idea. Coding, compiling, restarting repl is not good. It makes sense to be able to restart the program at will, cleaning up memory, re-evaluating functions while running, never even closing the JPanel. I think Uncle Bobs code was not intended for further development, but rather presentation so I won't use it as an example, but a real time-saver is to also ensure a good interactive setup.

Conclusion

Uncle Bob - We're glad to have you aboard, I hope you have lots of fun with Clojure and that the tips provided here are appreciated. To the rest of you - I dont write the standards for Clojure development, but you'll find them here: Guidelines. If I've missed any core idioms in this post, please point them out.

Generally I will say, that in demonstrating Clojure the Orbit Simulator does a very good job. Usually what people struggle with initially is adapting to the functional paradigmes, which Uncle Bob seems to have gotten right in the first try! There may be a little too much use of loop (where reduce could be used instead), but generally Im very impressed with the app. Hope to see more high quality demos from you in the future! You can find my modifications: here.

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: A hackers Blackberry

2010-10-06T07:26:06Z

Preface

One of the things which is really important to me is my toolkit - Im committed to always working with the best, most efficient, most supportive tools for the job. Recently I purchased the renowned developers keyboard Das Keyboard, and recently I shipped it back to the supplier. The main reasons were

It didn't make me faster/more precise
The sound of it is deafening

But one of the tools that I have been the most happy with is my trusty Black Berry so here's a report from the outskirts of BB land.

Getting the right smartphone

When I first opened shop I just wanted a good phone which did emails great - I chose the IPhone. After using it for a while both the phone and the company that sells them didn't sit well with me, so I shipped it back. I decided to try the android instead, which was truly a disappointment. The android OS and the quality of the HTC phones could best be described as half-baked. So onwards to Black Berry (after a short stop at Nokia).

Black Berry for Hackers

If you're a typical corporate consumer you'll buy your Black Berry and your expensive Black Berry dataplan and you'll be happy. If you're like me, you don't want to pay for bloat. When we were setting up our German Conj Labs session we had found a hotel which looked really nice, had all the trimmings, fair pricing etc. But taking a closer look at the paper-work, I noticed that they wanted something like 14€ per day, per attendant for internet access. So we emailed them asking to void that ridicously fee - they wouldn't... so we found another hotel. Its not that we couldn't afford it, its that we wont pay for sillyness - same goes with phones and subscriptions.

The Black Berry dataplan gives you:

Push Email
Blackberry messaging
Online surfing
Downloadable apps over the air

But what if I told you, that you could get all of that without a Black Berry dataplan? You can! With the possible exception of BB Messaging, I didn't look into it.

Getting online

I contacted my phone company to get the details needed for getting the Blackberry online - They said it wasn't possible. Then I got the instructions for a Nokia phone instead and lo and behold, they worked:

Go to options
Go to TCP/IP
Enable APN and set the APN: to whatever they would use for Nokia/Samsung or whatever. (in my case its 'wap')

Now you're able to browse the web using EDGE, which isn't too bad, though 3G would have been nice, but 3G is impossible since BB doesn't let us configure the proxy settings and so on for the APN. This alone lets you download lots of great apps from various sites around the web.

To get the most out of your browsing I recommend getting Opera Mini. But now for the all important.....

Push Email - My Funambol

Having push email is an absolute must for me and since the device is capable of it, why shouldn't I have it? Where the Black Berry solution costs you some money, My Funambol is completely free for the first 90 days - After the 90 days are up you have to either become a active/contributing member of the community or switch to another solution. If you stay, its still free, if you change provider it might cost you money, but look through this list of Funambol customers and see what your options are.

Setting up My Funambol is so easy it must be illegal somewhere. Once you've created your account, log in to your main screen:

On the far left you see my email account(s). Here you simply add your account by filling in the server/login details etc and Funambol provides you with a sync url. It also supports synchronizing pictures and events.

On the far right, you have links which you can follow to install the client software on your phone directly from the web. Once the you install the contact synchronization, you get this screen on your phone:

Simply navigate to the item you want to sync and click the mouse button - Almost instantly, your contacts, calendar appointments, notes or whatever will popup in the Web UI - Cool :)

Wish List

There are a couple of things which My Funambol leaves me wanting, which may or may not be in the pipe

Integration with the BB notification system, blinking LED
Customizable notifications
Notifications that respect bedside mode

But thats really it, other than that this free app really delivers what you'd expect!

Conclusion

Blackberry + My Funambol lets you reap the benefits of an expensive Blackberry dataplan without paying the costs - beautiful. And the application itself is so well crafted, so easy to use that its inspirational for me as a developer. Go check it out :) My next pet-project will likely be to put Clojure on the Blackberry!

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: Best In Class - The tour

2010-09-08T08:30:20Z

Preface

Some time ago I generally described the process of baking Best In Class and afterwards released the source code. At present I hear and see many questions relating to Web Development which could be answered by reviewing the sourcecode for Best In Class (primarily admin.clj and templates.clj). So to save you guys the trouble of having to install the entire site yourselves, I've prepared a small screencast. If you want to hang around after the screencast, there's a general talk about baking below.

Best In Class - The tour from Lau Jensen on Vimeo.

Avoiding performance craters

The old Best In Class was PHP driven. On every request that came in PHP would look into a database and fetch content, menus, comments etc, mix it with HTML and serve it. PHP is quite fast at doing this, but it still took so much time that during traffic spikes the site would becomes unresponsive and the backend nearly unuseable. I actually received a few emails from my hosting provider asking if everything was OK, because the app was tearing up the disks. Although it should be said, that back then I used to blog twice a week pretty consistently, which obviously puts more stress on the server than the sporadic posts that I find time for nowadays.

So talking about scaling a simple blog seems silly - Unlike an application a blog should just sit there and serve its pages without any need for special scaling initiatives right? Right! So here's how to do that:

Don't use two databases

The old site (or any Wordpress/Drupal/Typo3 based site) actually used 2 databases. First there was a MySQL backend which contained all of the information about the site in an easily accessible format (SQL). Then secondly, once that was compiled a second version existed of the fused data and HTML. Why not just leave out the first database and simply rely on the second, the filesystem?

Well for one - There is actually dynamic content

Like comments, twitter updates, etc. But if you simply update the HTML file atomically instead of adding more data to the database, that will effectively give you the same result as using a backend SQL. This works well for comments, footers, headers etc. For something like a Twitter stream, simply write some Javascript to fetch it in the browser. One example is my comment box. It needs to be dynamic but the page which fetches it doesn't. Its simply brought in via Ajax from the Moustache backend. All that Moustache needs to know is who's calling, so thats baked into each page as well:

(defsnippet post "blogpost.html" [:body :> any-node]
  [{:keys [title body thumb date comments link]}]
  [:h1#title]          (content title)
  [:img#thumb]         (set-attr :src (str thumb-prefix thumb))
  [:div#pubdate]       (content date)
  [:a#addcomment]      (set-attr :href (str "/cmt/url=" link))
  [:div#post]          (append (html-snippet body))
  [:div.comment]       (clone-for [{:keys [author email date comment]} comments]
                                  [:div.author]       (content author)
                                  [:div.date]         (content date)
                                  [:div.message :pre] (content comment)))

The line which selects [:a#addcoment] matches any links (a tags) with the id 'addcomment' just like CSS and then sets the href. Once you get used to the prefixed colons, this reads and acts exactly(!) like CSS selectors and its all performed automatically whenever I call this template.

Second - The filesystem is unsafe!

I received a certified beatdown in #clojure from a fellow who was very protective of his data, juding that simply relying on the filesystem was an incredibly dangerous way to go. My oppinion is, that all disks will eventually crash. The time of the crash is determined by the disks usage and SQL is heavier on Disk I/O than a static site. Second, usually what matters in these situations is your backup routine and your failover - Best In Class can in its entirety be backed up using rsync, can you say the same about an SQL database? Bottomline, the filesystem is completely safe and even an SQL database needs a working filesystem beneath it. In fact I'd say by simply using the filesystem as your database, you've removed a point of failure and reduced the complexity of your application.

Third - It can't possibly work!

When I reworked Best In Class I just knew that I was unhappy with the current setup and its performance, so after evalutating my options I decided to spend a few days on reworking the site using Enlive - There has been zero regrets. It was a fast rewrite and much of the time was spent writing a conversion routine for exporting all of the old blogposts from Wordpress to the new format. I decided to put all of the source online, so that an ambitious Clojure hacker moving from any kind of blogging platform could find some useful snippets. Also, did you know that Enlive lets you completely separate presentation from code? There is zero code in the HTML and zero HTML in the code, so I could work out the entire design in HTML upfront and just inject data using the CSS selectors.

Make life a little easier

Even if performance wasn't an issue, this would still be an optimal way to go in many cases. You can forever say goodbye to database maintenance, updates, heavy backup routines etc. The entire site is stored neatly in a folder which is ready to be tarballed or rsynced to a backup location - And the entire site can be reployed using nothing more than rsync - Truly an easier life.

The Moustache backend is usually the part of Best In Class that I tweak the most, adding features here and there. The wonderful thing about a completely static site, is the fact that I can restart the backend at runtime without any users being affected. Only in the rare situation that a user hits the 'Add comment button' in the exact same moment that I'm restart will somebody notice and I haven't received any complaints yet.

Final thoughts

Best In Class is faster than ever. And as a bonus - Did you notice how fast the admin panel came up? When I enter that URL all of the stats are compiled at that moment, and it takes about 1 second. The old PHP solution (which admittedly did a little more statistics, but not a lot) would take so long that sometimes my request would time out. If you want to see how it's done, the source code is still on: Github.

About the author:

Lau Jensen is the founder and owner of Best In Class a danish consultancy company which specializes in Clojure development.

Lau is also one of the instructors of driving the Conj Labs initiative. If you would like to be notified once new blogposts are published, you can follow Lau on Twitter.

Best In Class: Developer Productivity - The Red Pill

2010-08-26T10:19:22Z

Preface

"I know why you're here. I know what you've been doing... why you hardly sleep, why you live alone, and why night after night, you sit by your computer. You're looking for, it I know because I was once looking for the same thing. And when I found it, I knew what I'd been searching for. I was looking for an answer. It's the question that drives us. It's the question that brought you here. You know the question, just as I did." --- The (sorta) Matrix

Last time I blogged, I had just returned from the first Conj Labs in Brussels and coincidentally I've been tied up with Clojure development, so that now just as we open registrations for Conj Labs Frankfurt am I able to blog again. My last blogpost was meant as an inspiration to adapt your work environment to make you the most productive - I purposely didn't move into the 'how' as I just wanted to get people interested. It worked it seems so now I'm back to wrap up the series with some productivity tips as well as configs.

Top 10 Productivity Boosters

^{(by request)}

I've tried to think about some of the deliberate choices I've made to function better in my job, but Im sure your mileage will vary.

1. Get around 7 hours of sleep every single night

If your sleep isn't good you will suffer mentally and energywise. Catching up on a bad nights sleep is trickier than it sounds, so the key is to stick to a routine.

2. Avoid caffeine entirely (coffee & tea)

Ive been a long time coffee drinker (read: bottomless coffee pit), but after I returned from Conj Labs Brussels I considered the effect of the caffeine in my system and how I got up in the morning and felt like I was in a daze until I got my first cup. Taking that thought further I reasoned that after having slept through the first 2 hours of the night, the next 5 would be spent by my body craving the next cup, which was how I would wake up. Since I only have 2 settings (off or on) I quit coffee over night, cold turkey. It took 3 - 4 days before I mentally was on top again, but it took 14 days for the headaches to go away (they were bad). Now that its over and done with Im not touching that stuff again, Im mentally alert and sharp from the moment I roll out of bed till I get back in - No additives needed!

3. Maintain your tools

What good is a lumberjack who doesn't sharpen his axe? Only a little more than a developer who doesn't exercise his body. In our younger years most of us felt immortal, nothing we ever did gave us any lasting marks despite our parents warnings (at least I hope you were as fortunate as me), but as we grow older it becomes clear that the years take their toll. Lack of exercise is a killer - You'll feel it in your bones, when typing, in your energy levels, in your ability to focus - Everything works better if you maintain your primary tool: Your body. Personally I try to work in some exercise either into my lunch-break or in the evening - It takes a little time, but its worth it.

4. Dont be vain

In all honesty, one of the reasons I wanted to try out Linux way back when, was because of some of the Compiz/Beryl visual effects for the desktop - this is vanity. One of my friends recently saw my desktop (in all its Awesome Emacsy splendor) and his comment was "upgrade that ****" because he didn't feel that it was as pretty as his new Windows 7 installation, which it isn't. But in the time he has booted to his Desktop I've already answered 3 emails, said good morning on #clojure and written my first few lines of code - Vanity slows you down, whether its your choice of VM, OS (read: OSX or Windows) or anything else - If you want to be productive, aim for productivity not prettiness.

5. Avoid Social Networks

One of the strongest weapons for building quality systems is the ability to concentrate over prolonged periods of time. If you get regular Facebook updates, Twitter updates, or any other kind of updates which steals your attention, even if its just for a few seconds, I'm willing to bet that you're working at 50% of your full capacity. Why? Because even though it takes you 5 seconds to read a twitter update, I'm guessing it takes you 5 minutes to regain full mental focus. If your updates are coming in at a rate around 1 update every 5 minutes you're never running at full speed. If you are connected to something like twitter, which I am, I recommend that you check it once in a while - I do it in the morning or evening, but typically not during the day - and not every day. There are 2 exceptions: If my phone rings or if I get an email I usually reply ASAP whenever possible, since customers shouldn't have to wait.

6. Work off a TODO list

I showed this in my last screencast as well, how I organize my TODO bullets into categories A, B or C. A means 'will loose significant value if not done today', B means 'important, but will not loose significant value if not done today', C means 'optional'. I cannot stress how important this bullet is - On its own it might overshadow the other 9 in the short-term. When working as a Project Manager I have seen several developers who select some minor task to work on, and everytime they reach a milestone they look around, not knowing what to do, then start reading online newspapers or chit-chatting. Sure there's a time and a place for that, but if you're doing it consistently everytime you've put down 50 lines of code, you have a problem. Working off a TODO list means, that when you're in need of a high level of productivity you cross one item off the list and move on to the next! For many developers, huge wins in terms of productivity are available when the downtime between 2 tasks is cut out.

7. Use a tiling WM

If you're not on a tiling WM, you're constantly switching between the mouse and keyboard. Everytime you want to click an item with the mouse you have to find it visually, take aim, click, maybe miss, click again. In most cases you hit the first time, but you've wasted time context switching and taking aim. When I was younger I had shoulder/neck pains which I think came from using the mouse too much - These days I never have pains.

8. Use Emacs for Everything

This was a central theme in my last blogpost: Integrate as much as you can into Emacs: Code editing, HTML editing, Emailing, reading Twitter streams, file management, Git integration, Day planning and whatever else you can think of. Have a uniform interface and heavy integration between your tools speeds you up enormously.

9. ...Use Conkeror for the rest

Whether you like it or not, you probably spend quite a bit of time in a webbrowser - Either finding javadocs, searching for libraries or something similar. When I switched from Chrome to Conkeror my ability to browse became almost equal to my ability to read and think. As soon as I could think of which link to click, the page was already loading - Thats the power of a keyboard-based browser, why settle for less? (preemptive strike: Some would argue that the lack of Firebug integration etc, would be an argument for not using Conkeror, but notice I said use it for 'browsing', I use many different browsers for debugging/testing)

10. Always keep looking

I didn't start out with neither Arch, Awesome or Emacs, but a continual focus on optimizing my tools eventually got me here - Who knows where I'll be in 10 years. Right now Im trying to learn the NEO keyboard layout, because Im told it greatly reduces the distance your fingers have to travel when typing, thus speeding you up and minimizing the risk of getting RSI.

DISCLAIMER

You might be thinking after reading the above, that Im a sadistic terminator who is determined to run my colleagues/employees into the ground, sweating, sobbing, broken. Nothing could be further from the truth. The truth is, relax time is important and fun time is important. But I find that I can enjoy all of these the most if I've put in a good amount of work first. So if I am going to work for something like 3 - 4 hours in a row, doesn't it make sense to make those hours as productive as possible without wrecking yourself trying to cut down on breaks, sleep, family time and what not? I think a lot of the common problems that developers struggle with after years in the field can be avoided by adapting the techniques above, but as always I welcome input.

Watch out for the pitfall

So what to do, when you've applied all of the above tips and your back is still against the wall timewise. Those who have the resources in terms of colleagues or coworkers tend to try and delegate. Delegation can be like taking a loan to buy something which you really cant afford: It comes back and bites you and ends up costing more than you wanted to pay. Why?

I think primarily because people don't delegate intelligently enough. I recognize two kinds of delegation: Gopher Delegation and Delegation:

Distinguishing

Gopher delegation is the type of delegation that sounds like 'please go to this store, pick up these 3 items, come back and put them on the table in the cantina, arrange the plates, forks and glasses around the table, evenly distributed in the exact amount of people attending the lunch meeting'. This type of delegation is in stark contrast to true delegation which sounds like 'Please make the necessary preparations for our lunch meeting at 12:00'. The key thing to learn, is that gophers need gopher delegation and trusted employees/colleagues need regular delegation.

In the above example, the gopher given the first task would probably forget the knives - he was only asked to get forks and since the entire task had been nicely cut out into smaller tasks he felt no responsibility to go above and beyond, he simply follows orders. For gophers, this is what you want, fortunately there aren't too many of them.

The trusted employee might feel free to pick up more than the 3 items, might clean the table, or apply any number of improvements to your original plan (which he wasn't told) because he feels its his task and his responsebility to work out a great solution. For Conj Labs we work out a number of lab exercises in advance of the course. Imagine I was to ask Christophe to prepare such a lab. Would I get the best result by telling him exact scope of the exercise, where to inject explanatory slides, which functions to use, etc etc? Or would it be better to say 'Christophe, I would love a lab on DSLs can you try to come up with something?' - Let me tell you, he has not yet failed to surprise me :)

In the past I have been bit by delegating important tasks to gophers without being clear enough, and Im sure I have choked trusted employees creativity and intelligence by being too specific - Both are enormous errors where delegations ends up being a pain instead of giving you some freedom with your time. But now for the practical stuff:

Emacs keys worth knowing:

I promised to list some of my most used keyboard bindings, so I've asked my fingers and here is the result. Since this is almost an inexhaustable topic, I'll keep it brief and open up the comments section :)

Small helpers

Zap-To-Char (M-z): Kills all characters up to and including the one you supply as an argument

Recenter (C-l): Try hitting it 3 times and see what happens each time (that's C-lower-case L)

Query-replace (M-%): Regex replace, (y to replace, n to skip, ! to replace all, works on regions as well)

Goto line (M-g g): Jumps to a specific line

These three guys go together and I use them constantly for rearranging text:

Kill-Line (C-k): Kills an entire line

Kill-Region (C-w): Kills a region (region: Selected area)

Kill-Ring-Save (M-w): Copies a region to the clipboard

Paste-Ring (C-y): Pastes whatevers in the kill-ring

Paste-again (M-y): Keeps replacing what you just pasted with the next item in the kill ring

For repetitive tasks, these are a must

record-macro-start (C-x (): Records all keystrokes until you stop recording

record-macro-end (C-x )): Stops recording

play-macro (C-x e): Plays the last recorded macro

play-macro-on-region (C-x C-k r): Play its only on the selected region

play-macro-n-times (C-u 10 C-x e): Plays the macro 10 times

save-macro-with-name (C-x C-k n): Give it a name, refer to it later

M-x insert-kbd-macro: Lets you save the Lisp code of your macro for use in future sessions

Optimizations

M-x paredit: Will disable paredit if you enabled it by accident

Configs

I've put my configs on Github - I dont plan on updating them, so they are there now and can be used as inspiration.

Emacs:

Nothing too special here. There's my swank setup, which has a very customized classpath, this is the times where I want to fiddle or contribute to some project - Most of my development goes on after calling M-x swank-clojure-project. If someone is pinging me in #clojure I hit F12 which makes the channel fullscreen, and once Im done I hit F12 again and the original window configuration is restored completely. At the very bottom Ive added some repos to technomancys new version of ELPA, which I never finished testing (sorry).

Awesome (rc.lua): The awesome config (rc.lua) is looted from anrxc (#archlinux)

This setup will likely save you half a lifetime - Installing Awesome takes a few seconds but configuring your way out of the lua madness (indices start at 1) takes quite some time. Thankfully #archlinux is a good place to get help. There's nothing too specific about this setup, except for when you hit M-q then a small Emacs-Orgmode-Remember window pops up, which allows you to quickly take notes. This feature wont work unless you use (parts of) my emacs config and you need to change a few paths. Finally, your battery is most likely not named as mine, so to get battery stats in the top bar fix the string on line 50 in /awesome/vicious/bat.lua. The entire config goes into ~/.config/awesome. (yes I know its ridicously hacky, I love using Awesome, not configuring it)

Wanderlust:

There are several Email applications for Emacs - They all suck, Wanderlust sucks the least. I use Wanderlust sporadically for sending emails and also for reading emails - I do however always keep a thunderbird icon in the tray to alert me to new emails, as this is one of the (many?) bugs of Wanderlusts IMAP integration - It doesn't notify you of new emails. Word of caution: If you subscribe to the Wanderlust ML, you can only unsubscribe by sending an email from within Wanderlust - Not knowing this, cause a little pain and a lot of flame. When they get the bugs ironed out, I might switch 100% to a fetchmail/wanderlust combo but its not quite ready yet.

Note, that when you're composing an email, hit C-h to convert it to HTML using org-mode-htmlize.

You can find the files: here

Conclusion

Its important to be productive - I see it as driving a car where there's no speedlimits, why not see how fast it can go? I hope I was able to inspire some of you to revisit your setup and to start asking the question "How productive can we become?".

There's of course a final very definite way to be more productive and that is to use a screwdriver for screws instead of hammer, and to use Clojure for development of quality software. If you would like to learn more about Clojure and how to use it professionally, I'll recommend you to join us at the next installment of Conj Labs - This time in Frankfurt, Germany. Once again I'm teaming up with the fantastic Christophe Grand (author of Enlive and Moustache) to provide 3 days of Clojure training - We hope to see you there. As a new thing, we have acquired the help of InnoQ who are helping us in spreading the word, so we hope to see many Germans (as well as foreigners) there.

Best In Class: Trail blazing innovators

2010-07-07T07:17:42Z

Preface

The first ever ConjLabs was held the week before last in Brussels and in attendance were some of the more innovative developers of Europe. After each of them got back home, my feeling is that people really were able to step up to the next level of their Clojure game after those 3 days. One of the guys asked me "Well how do you market an event like this?" and the answer is simple: We didn't. The first Conj Labs was similar to raising a magnet which draws in natural born innovators and see who would show up.

Innovators in action

The crowd consisted of experienced developers who were well established in a variety of languages, one of them even knew Perl. In common, they all shared a desire to improve on their current skillset in the hopes of producing better software. One guy said "My ambition is to build small beautiful programs, that are simple and easy to maintain and my hope is, that Clojure is the language which will let me do that." (thanks Walter). I think he expressed an ambition which we all shared, but during the week it occured to me that I was amongst a special brand of gentlemen. For a large part of my carreer I have worked primarily with people who have found something which lets them complete tasks, be it C, C++, Java or whatever. And once they've attained a certain level of mastery over one of these languages, they disregard the glaring problems and productivity inhibitors and push on to complete the assignment on their desk. The magnet-marketing trick had effectively brought in a different crowd.

Another attendant related how he works on a massive code-base for a world wide corporation, where he was tag-teaming with hundreds of developers in C++ in order to add incremental changes to their product. I asked "Its not very mentally challenging is it?" and where many developers might have said yes (and felt it), he said no. He was looking to build projects in a language which would let him be work on the problem and not the limitations/traditions/ceremonies of the language.

Innovative weaponry

At the lab there were many different setups. We had Danish keyboards, French keyboards, Dutch keyboards and many more. Almost all of the attendants were wielding Macbooks and OSX while the instructors were on Lenovos. One guy remarked "Its weird to see someone as smart as Christophe using Windows" and I couldn't agree more. So in the name of higher education I agreed with Christophe to have him firmly planted in Emacs for the duration of the Labs. Why? Because Emacs is a black hole which eventually sucks in all productivity killers allowing you to reach previously impossible measures of productivity. The stumbling block, is that you need to crave innovation and productivity, otherwise you might not be able to look past the old-fashioned interface and enormous help file. C-h t however, lets you hit the ground running (emacs tutorial).

I've prepared a small screencast to demonstrate a few of the reasons why Emacs is the IDE of choice for Clojure development, as well the for many other languages. The main pull of Emacs is nothing Clojure specific, rather its the ability to unify all of your tools within the same program:

Emacs - IDEverything from Lau Jensen on Vimeo.

There are a few keys to obtaining über productivity which I try to live by:

Stay on the keyboard and off the mouse! (for everything)
- That means use Emacs for everything and Conkeror for the rest
Keep as much functionality as possible in Emacs
- That gives you a very uniform interface, your fingers will know what to do
Make friends with Emacs keyboard bindings
- Since you can now use them for everything, it makes sense to obtain Emacs mastery

I know that there are many would disagree and claim that Mouse-driven IDEs and XML based configs are the way to move forward, but I firmly believe that such convictions stem from a lack of exposure. If you're already an Emacser, I recommend reviewing your toolbox and seeing how much you can stuff into Emacs. If you're not, I highly recommend throwing yourself off the deep end, you'll be surprised a how quickly you start swimming fast while your colleagues are still in float-mode.

I've made the following deliberate choices:

I run Arch linux because it's a slim, safe and stable distro with just around zero bloat
I use Awesome as my WM because it allows me to stay off the mouse entirely
I use Emacs as my IRC client, Email client, Addressbook manager, Dayplanner, Code editor, Revision Control interface, and much more
I use Tilda for fast shell access
I used to run Gnome-do, but I havent installed that on Arch yet since Awesome has a similar feature but with zero bloat
I used to run all my chat-networks through bitlbee and straight into Emacs but I havent installed that on Arch - I might though.

You don't have to do it like I do it as long as you get the big wins from the first list, but I'll admit Ive never seen an effective setup on neither OSX or Windows - However, if one such exists feel free to add it in the comments.

Conclusion

The first Conj Labs was fantastic and everybody walked away with a lot of tools/skills to get them to the next level of their Clojure game. We are already mounting the next session which will be in Germany, if you want to be notified when the details are in place either follow me on twitter, watch conj-labs.eu or send me an email.

If I the interest is high enough, I might do a separate blogpost demonstrating my config files :)

Best In Class: Now Open Sourced

2010-06-04T00:26:34Z

Preface

Last month I did a remake of BestInClass.dk and afterwards I was very tied up preparing for the Conj Labs session here in June. But now that we're catching up, I figured I would clean the code-base and release it.

One big win to this approach is that you can fiddle with the backend all you want and once you're ready to update, just rsync the code to the server and call restart on the jetty script. The only downtime your users will experience is between the closing of the socket and until Jetty is up again, and only users who are clicking the comment button at that exact time will notice!

Caveat

One of the things which can come in handy when you're rapidly testing web-handlers, is Moustache. I used it heavily during the first days of development and I've left the webview.clj file in the repo, as a reminder to others not to let your prototyping grow out of hand. If you want to do some prototyping which mimics what I did, then feel free to use it otherwise just remove it.

Security

You might be wondering how the backend is secured, since there's no standard auth lib available for ring/moustache. The backend opens an interface which only answers to calls from 127.0.0.1, meaning the webserver it self. To reach it you have to get through a reverse proxy pass, which filters the reponses based on certain criteria, which amount to this: If you're not me, you're not getting proxied. So almost all security is at the webserver level.

Points of Interest

The primary bestinclass.clj file was used by me when I deployed bestinclass.dk for the first time. Its a one off event since it goes through all of the :WP:POST entries in the Wordpress backup file and generates the html version of those posts. Afterwards, the static pages "Forside", "Services" etc are generated and finally the atom feed is generated. All of these emissions of html/xml call templates or snippets found in the templates.clj file. The file should more approriately be used to start the admin interface now, but adjust it to your needs.

While the generation of the site shows of Enlive better half namely "templates" then the Wordpress import module shows of "selections" or scraping. Some of the selectors are a little complex and I must admit I've consulted with both Christophe and David Nolen to arrive at the result. However it stands as a very good demo of just how powerful Enlive is.

Backend features

The backend features as not-so minimalistic editor:

There's also some statistics and an interface for comment moderation. Since I added the custom captcha I have not received one single spam comment - not one! If you want to test the comment interface yourself without running the frontend through a webserver, you can post like this:

$ curl -d "url=someurl&name=Lau&email=mymail&captcha=10&cid=0&comment=hi" http://127.0.0.1:8080/cmt/

The url is only used to determine which file to write the comment in, and it poses a potential security hole as you could manually post to /../../../some-config and then really make my life miserable. So it checks if you're doing any such thing and if its determined that you're trying to crack my server, the dogs are dispatched immediately. If you need to test with something that has spaces and linebreaks just run that string through java.net.URLEncoder/encode and paste it to curl.

If the comment comes through with the correct captcha answer and the URL isn't malformed/manipulated, then the server sends me an email (please change the recipient on your local installation). It does so via a postfix server running on "localhost" - any smtp server will do. See email.clj for more details.

TODO

There's still work to be done. Name the load-times on the admin interface needs to be improved (its not hard) and the editor should be integrated in the admin interface. As it is I go to /editor and once I've completed the entire post I browse to the /admin interface (waiting once again for the stats) and publish.

In closing

I'll let the code speak for itself. If you like this approach to webapplications, please feel free to use the code, you can find it here: Github

If you would like to get an update when I post on this website, either subscribe to the RSS feed or follow me on Twitter.