With the Manifesto firmly established lets then turn our attention to the mental aspects involved. The ability to write beautiful code requires more than just understanding the components and qualities thereof. Lets go down through the corridors of thought, comparing ourselves to the code we produce.
I dislike provocation for the sake of provocation, stirring up tempers for amusement. Conversely I enjoy challenges and indeed to challenge others. Today I hope to challenge some of you to dig deeper than you have before. This post is not for Lispers or Clojurians as much as those who have not yet enjoyed the freedom which is given at far end of the learning curve. A learning curve which unfortunately discourages many.
What inhibits learning and why do many stick to the comfortable paths? Comfort is a natural inhibitor in that many people are not willing to put in work in order to acquire new abilities. If you are in that category then this post is not for you as I don't have such a command over the English language as to motivate you beyond your comfortable resting place - The price you pay for comfort is exceedingly high, as we shall soon see.
There are 2 other categories of individuals who have not yet begun their mental ascension towards code mastery, who can benefit from this post. First and foremost those of you who have struggled heavily on the learning curve of Lisp and not found the motivation to push through to the liberty of expression. Secondly, there is a much larger group who refuse to embark on the journey for the reason of hating Lisp and all that comes with it. For this second group Leo Tolstoy has with surgical precision pinpointed the cause of your condition and by embracing this diagnosis, you will be able to move on:
We come here upon what, in a large proportion of cases, forms the source of the grossest errors of mankind. Men on a lower level of understanding, when brought into contact with phenomena of a higher order, instead of making efforts to understand them, to raise themselves up to the point of view from which they must look at the subject, judge it from their lower standpoint, and the less they understand what they are talking about, the more confidently and unhesitatingly they pass judgment on it.
- Leo Tolstoy
Before you wield the power of Lisp or Functional Programming you have not been lifted to the required level of understanding and in my estimation once you wield that power, you will not want to go back.
Lisp has been used extensively for the study of Artificial Intelligence for a good reason. It helps, with superior ability, to translate a humans thought into the understanding of the computer. All languages are eventually compiled to machine-code for the sake of execution and since machine-code is a very poor reflection of a human thought our only option for unifying the two is to adapt the language.
On the other end of the spectrum we come across C and its relatives. When you write C code, you first imagine what your program must do to solve a problem, then you manually convert that to imperative code, namely C. The produced C-code reflects your original thoughts similar to how a muddy pool reflects the Atlantic Ocean. You have willingly converted your thoughts to a robotic manuscript which hopefully will do what you thought it would do, under all circumstances.
Consider the following
def length(s):
n = 0
while (s != []):
n += 1
s = s[1:]
return n
You pass a collection of some sort and it loops through it, incrementing n while there's still items. It works, its not buggy but your original thought is somewhat obfuscated now. Imperative programming tells the computer how to solve a problem, robotically. Functional programming tells the compiler what the problem is, elegantly.
length :: [Int] -> Int
length [] = 0
length (x:xs) = 1 + length(xs)
If you understand that x:xs simply means a list with a head and a tail then that's almost how you would explain it to your uncle
If the list is empty the length is zero, but if it has items then every item increases the length by 1
But why is it so important, that our language reflects our thoughts?
Recently I came across an article which in some ways served as the memoirs of E.W. Dijkstra who has been in the game for over 50 years. Reading it was like reading a book of quotes and the mans insights were profound, starting with
But if I start to analyse the thinking habits of myself and of my fellow human beings, I come, whether I like it or not, to a completely different conclusion, viz. that the tools we are trying to use and the language or notation we are using to express or record our thoughts, are the major factors determining what we can think or express at all! The analysis of the influence that programming languages have on the thinking habits of its users, and the recognition that, by now, brainpower is by far our scarcest resource, they together give us a new collection of yardsticks for comparing the relative merits of various programming languages.
- E.W. Dijkstra
What he is touching on here is at the very core of what I'm trying to say with this post. If you spend your days and nights adjusting your brain bit by bit to an imperative language you narrow your horizon. The depth with which you can reason about behavior and algorithms becomes increasingly shallow. In the previous section I said that the developer converts his thoughts to a sequence of commands (imperative code) , but that's only for the beginners. Experienced C, PHP, Java developers etc do this almost like second-nature, they start to think in ways that are restricted and we ironically refer to this as experience, despite its detrimental effect on the mind.
Ideally we would primarily develop software using languages which only acts as a relatively small indirection between our minds and the machine-code. This will regularly let us train the part of the brain which produces algorithms, architectures, solutions based on insight and creativity and not the restrictions and customs of our language.
If I were to put my thoughts down in a RAW format, you'd see a mix of alpha/beta waves and a vast list of neurons firing, the strength of their bindings determining the outcome. It's not something which means anything to you, because you don't have the same bindings, so to speak. What I'm about to demonstrate is a language which for new-comers is hard to read, easy to misunderstand and ideally suited for use in industry. But it's the best way in which I can demonstrate how to get thoughts jotted down and tested - Enter J.
Let's say that I'd like to produce a surface rendering of the Cos/Sin values of all digits from 0 - 50 dampened with a factor 0.2 in 3D. How would you do that in C? You have to find a graphics library, get the main function written out, that's gotta run a few loops, track cos and sin values etc. Or what about PHP? Or Go? Or Java? Or Python? All of these trigger almost similar thoughts regarding the approach and why shouldn't they? Once you've seen one Algol68 derived language you've pretty much seen them all.
Instead have a look at my 1-line solution in J, the first of two 1-liners you'll see today:
'TYPE surface' plot 1 2 o./ 0.2 * i.50
Which produces the following surface graph
[caption id="attachment_555" align="aligncenter" width="399" caption="Click to enlarge"]
[/caption]
As you behold the direct and short line between my thought, the representative code and the resulting graph consider Mr. Dijkstras quote above - What we can express in code affects what we can think. Now I can think in graphs with the small indirection of powerful expressions that pack a hard punch due to their succinct notation. That notation now becomes my tool for thinking, allowing my mind to run extra laps around the problem at hand.
J takes a lot of practice to master, because all it's functions have both dyadic and monadic variants and can be modified using either inflections or adverbs - Put more simply: The function + can be used monadically as +5 to produce the positive integer 5. It can also be used dyadically as 5 + 5 to produce 10. It can also be used in conjunction with foldRight / as +/ 1 2 3 to produce 6. That makes J tricky and time consuming to learn. Fortunately, that's beside the point of this post.
I recently picked up J as a method of expanding my ability to transform thoughts into programs and I have not been disappointed. Despite the required investment of time there is a measurable outcome. The honorable Tracy Harms has been instrumental in softening the learning curve and here I will try to take you through his Cellular Automaton.
Rosetta Code defines the task like so:
Assume an array of cells with an initial distribution of live and dead cells, and imaginary cells off the end of the array having fixed values.
Cells in the next generation of the array are calculated based on the value of the cell and its left and right nearest neighbours in the current generation. If, in the following table, a live cell is represented by 1 and a dead cell by 0 then to generate the value of the cell at a particular index in the array of cellular values you use the following table:
000 -> 0 # 001 -> 0 # 010 -> 0 # Dies without enough neighbours 011 -> 1 # Needs one neighbour to survive 100 -> 0 # 101 -> 1 # Two neighbours giving birth 110 -> 1 # Needs one neighbour to survive 111 -> 0 # Starved to death.
The output should then be something along these lines:
_###_##_#_#_#_#__#__ _#_#####_#_#_#______ __##___##_#_#_______ __##___###_#________ __##___#_##_________ __##____###_________ __##____#_#_________ __##_____#__________ __##________________
Despite the many rules you can implement a simple solution, because the sum of all rules which result in life is 2 and all rules which result in death is non-2.
Before I share and expound Mr. Harms solution, please prepare yourself by reading the solutions written in C and Java. They will serve as representatives of the old mindset which we are trying to forsake in order to liberate our minds, enabling us to reach higher levels of insight than we could before.
Mr. Harms solved it in this way:
life1d=: '_#'{~ (3(2=+/\) 0,],0:)^:a:
As you read that try to mentally overlay the C and Java solutions and imagine the route from the developers thoughts to the produced code. I can't tell you how long that route is, but the longer the better. If it's short that means that the developer has been consumed by his language. If you are disgusted by what you see, cling to Mr. Tolstoys diagnosis as I break this solution down going from left to right.
Mr. Harms starts out easy
life1d=:
Thats just a definition, but in J we use =: for globals and =. for locals. He knows that what he wants to produce is a list of binaries similar to 0 1 0 0 1 1 0 0 1, but he needs it rendered in a way where 0 becomes _ and 1 becomes #, so he says
'_#' {~
And here J shows some power, which I can best demonstrate by teaming J up with Clojure:
2 { 1 2 3 => 3 equal to (nth [1 2 3] 2)
1 2 3 {~ 2 => 3 equal to (nth [1 2 3] 2)
The tilde simply reverses the order of the arguments, so in effect what he's doing is picking indices from a literal list '_#'
'_#'{~ 0 1 1 0 equal to (map #(nth [\_ \#] %) [0 1 1 0]))
_##_
Now that he's handled the rendering in just 6 characters he moves on to the rules. We can start out by producing the sum of a list of binaries by using foldRight as I mentioned earlier:
+/ 0 1 0 1 2
But in order for this to work with the cellular automaton, we need to partition it in chunks of 3, so we use \ dyadically with the left argument being our chunk-size. To show you this step I have to box the result using < but that's of little importance
3 <_ strong="strong" _01="_01" _----------="_----------" _1="_1" _0="_0">
Now it's a small thing to combine our partitioning \ with our tool for summing up a list and asking if the result is 2 or not:
3 (+/\) 0 1 0 1 NB. Summing up the windows 1 2 3 (2=+/\) 0 1 0 1 NB: Asking if the sum equals 2 0 1
So we're coming up on our tenth character and we have the rendering, partitioning and application of the rules. The program now looks like this:
life1d =: '_#'{~ (3 (2=+/\)
So next up we need to compile the right hand side argument for (2=+/\) which cant just be the raw input because we risk some nasty corner conditions, so to avoid that we pad the right hand side (RHS) argument with zeros. In Clojure we use % to refer to the anonymous RHS argument but in J the equivalent is ] and since J is infixed you also have [ to get the left hand side argument. And when you know that comma , is shorthand for append then padding is a breeze:
(0,],0:) 1 0 1 0
A lonely 1, padded in zeros. The sharp reader notices the colon: behind the last 0 and the really short explanation is: Dont worry about it. Our program now looks like so
life1d=: '_#'{~ (3(2=+/\) 0,],0:)
So what's left? We'll there's really only one thing and that's producing more than a one-line result. The simulation needs to run until the result stabilizes, ie. the nth result equals the n-1.th result. In J you use the Power ^: function for determining the number of executions
double =: +: double double double 5 40 double^:3 (5) 40
So function^:n args executes function, n times iteratively starting with args. In J you always get a few goodies for free though, so ^:a: means run until result stabilizes. And so our end result is what Mr. Harms started out with:
life1d=: '_#'{~ (3(2=+/\) 0,],0:)^:a:
Now once again I would welcome you to mentally overlay the classic imperative solutions, realizing the awesome expressiveness of J. Looking at this code I'm not sure what fascinates me the most:
Don't get me wrong, I don't encourage software development while grocery shopping but it's a sure sign that you've begun unifying your thoughts and the computers effectively once you have the ability to do so. If a C programmer said "Oh I had a mental picture of these 75 lines before even hitting the keyboard" he's most likely lying through his teeth. He still has to run through the motions of breaking down comprehensions into loops, factoring out routines, getting types just right etc etc. So at the end of the day, what would you say has filled his mind? The finer points of a cellular automaton, or the finer points of C?
Foy Savas recently proclaimed 'Hate is a lie'. Despite the fact that hate is a very real thing, he brings up a good point saying that it serves to simplify the world around us. Mr. Savas relates how he quit his day-job as a Ruby developer because his boss wanted him to write code in PHP instead. Few people would blame him for making that decision, but why is it that developers are generally so agitated by PHP?
Mr. Savas reminded me, that as a Hypertext PreProcessor, PHP isn't bad. It's actually pretty good. If what you need to do is just slap a few files in an Apache directory and have them modify some html output, then PHP does the trick. So what's the problem?
The problem is that PHP in all its simplicity, with all of its restrictions and pitfalls warps the minds of its users. People take this little Preprocessor and get a kick out of its non-existent learning-curve. Instantly they can make small applications which query databases and automates simple processes. Some will be entertained for years by this, but while they're having fun their mind stagnates. After 5 years they're still just loading libraries and rendering HTML in new and colorful ways.
So when the owner of a now untrained mind is faced with a larger, more complicated problem than just preprocessing html and he's only worked in PHP for the past couple of years. What does he do? He builds a large project using a Hypertext Preprocesser, Ouch! In the wake of such a decision naturally comes a lot of pain, frustration, bugs and more of the same flavor. PHP isn't meant for larger projects and using it on that scale is not helping its reputation along, but what should worry the imperative crowd more than the number of bugs they produce is the effect their daily programming has on their mind. In the case of Mr. Savas, I understood that his boss actually wanted him to write an application which demanded some multi threaded functionality in PHP rather than Ruby - Recognize a problem?
When we observe certain development communities, surely a pattern of in-the-box thinking emerges in many of these. The first thing that struck me about the Clojure community was their friendliness and secondly their deep technological insights and ability to think outside the box and subsequently extend the box a little for the rest of us. And if you are rightfully blown away by the deep insights of many Clojurians (Rich Hickey perhaps being the best example), you'll notice the exact same thing in the J community perhaps even more so. They've long ago forsaken the "Why does this pointer operation segfault" or "Why can't I cast this type to an array", instead they focus on the problems at hand.
I have now shared with you some thoughts which stem from personal experience. I am definitely not saying that every C, PHP or Java programmer is unintelligent and has a small horizon but many do and I think most are headed in that direction unless they intentionally put in work to avoid getting there. How else would you explain the catastrophic quality of much of the software floating around?
If you have read this post and now feel offended, please don't. Taking offense is a waste of time. I cannot judge where you are now on the path towards code mastery but I hope you see that Clojure and FP offer you a quick route to improving both your development skills and your mind, though it requires some work and persuasion. J as a late comer in my personal arsenal takes its seat in this category as well. If you are one of the classic in-the-box run-of-the-mill nothing-new-under-the-sun solve-everything-with-a-loop kind of developer, then the good news is that you have unused potential just waiting to be used.
I will give the final word to Mr. Dijkstra
LISP has jokingly been described as "the most intelligent way to misuse a computer". I think that description is a great compliment because it transmits the full flavor of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.
- E.W. Dijkstra
