Jan 16

Yurt: Mount’s Local Real (e)State

Components or states in mount are kept in Clojure vars, which means that they are accessible from any other namespaces that :require them. They can be made private of course but nothing stops a developer from accessing a private var in Clojure by its full name: i.e. #’any.namespace/any-var.

The flip side is that they “lie near”: they are really easy to use. Can usage of these vars be abused? Of course. Can any “other state management solutions” be abused? Of course. I like my fine balance between “easy” things for development and “simple” things for the architecture.

Several, Local and Simultaneous

In several reddit discussions, people pointed out that since mount keeps components in Clojure vars, those are singletons, hence you can’t have more than one.

While I honestly don’t know how often I would want to have more than one database connection to

* the same database
* with the same host / port / sid
* the same credentials and
* the same schema

(because if any of the above is not the same it would be a different resource / component all together) The example people gave would always come down to this one use case where you want to run a development “system” and a test “system” (potentially more than one) in the same REPL.

The way I do it today is simply running:

boot watch speak test

in a different REPL.

The need to run multiple systems in the same REPL might have something to do with the limitation of the framework (i.e. component), since you can’t just start and stop parts of a system, which means you can’t iterate quickly with sub systems in the REPL. So instead you end up predefining and running several sub systems simultaneously. But this is just an assumption.

I personally never needed to run multiple “systems” within the same REPL. But I truly believe there are multiple great ways to do things, an I simply can’t just dismiss the fact that people have different development flows.

So I sat down and started to think.

Yurts are Comfy

Mount relies on the Clojure compiler to tell it what states are defined and what order they should be started / stopped in. So, I thought, I can just use this intel, create a “system” and simply detach it from vars. In other words, using the intel mount has and only using the vars for the bootstrap I can spawn as many local systems as needed.

I don’t think this should be mount’s core functionality, plus I really like the way things work with vars and namespaces, but it would be really cool to have these local systems that people can use for testing, whether it is from REPL or for running tests in parallel.

That’s how Yurt was born. The docs go over Yurt’s API (i.e. blueprint, build, destroy) and have a couple of examples of running multiple Yurts simultaneously in the same REPL.

This of course comes down to a choice of either using vars directly or encapsulating components in Yurts, or maybe even both (?) in the same application. But what’s cool about either choice, there is still “nothing to buy”: no full app buy in.

Jan 16

Swapping Alternate Implementations with Mount

Getting ready for a talk at Clojure Remote gave me an excuse to work on several mount example apps that, I feel, should help others to start with mount, as well as just present certain ways to structure applications.

Sending SMS over Web

Using a great twilio library I wrote a small web app that sends texts over web (i.e. via HTTP POST). It serves as a good example of how to test mount states by swapping them with stubs/mocks.

This application has 3 states:

* config which is loaded from an external file

* web-server a Jetty web server

* send-sms which, once started, becomes a function that sends texts

The app receives an HTTP POST request and sends an SMS message:

(POST "/sms/:from/:to/:msg" [from to msg]
    @(send-sms {:from from
                :to to
                :body msg}))))

The send-sms is a mount state that is setup with Twilio credentials that come from the config:

(defn create-sms-sender [{:keys [sid auth-token]}]
  (fn [{:keys [from to body]}]
    (twilio/with-auth sid auth-token
        (twilio/sms from to body)))))
(defstate send-sms :start (create-sms-sender 
                            (:sms config)))

When an HTTP request is sent:

$ curl -X POST "http://localhost:4242/sms/+15104266868/+17180000000/mount%20is%20fun%20:)"

it gets wrapped to an SMS payload and is passed to Twilio which successfully delivers it:

I am sure you noticed, but the Twilio phone number this SMS is sent from is:

+1 (510) 42 MOUNT :)

Testing the App

Notice that while send-sms is a state, when started, it becomes just a function that takes args and passes them to Twilio as an SMS payload. Which means that if it is needed to be replaced during testing, it can be replaced with a test function that, for example, receives an SMS and puts in on a core.async channel:

(fn [sms] 
  (go (>! sms-ch sms)))

One thing to note, the real twilio/send-sms returns a future, which means it might be dereferenced somewhere in the codebase, and it is just safer to stay true to the “real thing”, so we’ll return a future as well:

(fn [sms] 
  (go (>! sms-ch sms))

Now all that needs to be done is to create a test state and let mount know to use it instead if the real one. We can do it from within a test file:

(def sms-ch (chan))  ;; can also be a state
(defstate send-sms :start (fn [sms] 
                            (go (>! sms-ch sms))

And in order to use it we would start mount with this test state instead of the real one:

(mount/start-with {#'app.sms/send-sms #'test.app/send-sms})

This way the application will be started as usual, but instead of the real send-sms state, it would use this one from a test.app, where all SMS messages are going to be sent to a core.async channel instead of Twilio.

This means a test can read from this channel and see if what was meant to be sent was actually sent:

(deftest sms-endpoint-should-send-sms
  (mount/start-with {#'app.sms/send-sms #'test.app/send-sms})
  (http/post (post-sms-url "mars" "earth" "we found a bottle of scotch!"))
  (is (= "we found a bottle of scotch!"
      (:body (<!! sms-ch)))))

Check out the working test to get a visual on how all the above pieces come together.

And here is more details on swapping alternate implementations from mount docs.

Narrowing Down the Scope

While working on this blog post I realized that start-with can be improved, so I created an issue that, once completed, will make start-with to take an instance for replacement, rather than a var, which would allow to narrow down the scope during testing with alternate implementations:

(let [sms-ch (chan)
      send-sms (fn [sms] (go (>! sms-ch sms)))]
  (mount/start-with {#'app.sms/send-sms send-sms})
  ;; testing.. and

It would shift the responsibility to a developer to cleanup the instance when needed, but this is something that we are doing in tests already.

Jan 16

Disturbance in The Force

I’d like to lay out a couple of thoughts here to discuss further to get more insight from people with different mindsets / opinions / facts / views. Reasons are mostly selfish: I know what I don’t know and I’d like to understand it better.

Folds and Braids

As Clojure developers we are in this “Cult of Simple”. Simple “was defined to us” :) as one fold/braid/twist, which does not mean “one thing”, and is really about the interleaving, not the cardinality. We also know that simple is “objective”, we can look at it and see the “number of folds”.

I don’t think it’s such a bad cult to be in, but.. It seems that the above “simple definition of simple” is now taken to an extreme where, as definition, it is interleaved / complected with real problems and is losing its intended power.

The definition of simple is now used as a shield, rather than a tool.

It might have to do with the very subjective definition of “one fold”. It is easier to understand what “one fold” is when thinking about “primitives”: i.e. Rich’s example of “Sets vs. Lists”, where Sets are simpl(er) since Lists introduce order. But it is not as clear what “one fold” is in a more “complected problems” whether these are business problems or tool libraries.

Keep Your Functions Close, but “Just In Case” Closer

Clojure protocols are super powerful, and I would say “simple”.

I don’t think Clojure records are powerful, but good (in my experience) for type driven polymorphism, and they hold fields a bit more efficiently than maps. In reality “type driven polymorphism” would be the only reason I would use them for.

If “type driven polymorphism” is all that’s needed I would first reach out to deftype instead of records, since records complect data with types: two folds :)

However, since intuitively, building solutions with protocols feels a lot more extensible, robust and flexible, I think they get applied where a simple set of functions over multiple namespaces should have been used instead. This problem, I think, is caused by the invisible seduction of “easy”, which is defined to us as “lie near”:

since my solution should be robust and extensible, I’ll use records and protocols, since I know they will make it so“.

In other words, for initial application design: “records and protocols lie near”.

That is not to say that protocols or types should be avoided, quite the contrary, dynamic type dispatch, libraries with (internal) abstractions, host interop: all great cases to use and love them.

But I don’t think it is wise, in the Clojure Universe, to make people create records / types and use protocols when they need to use your library to develop products. This is not the absolute truth, but for most cases, when developing a business application, I would rather use a Clojure function: one fold :)

Humor Driven Development

I spend a couple of years working in Scala. An implicit type in Scala was one of the most common causes of confusion. It is used everywhere internally in the language itself, as well as advocated to use in every day Scala programs. This and many other examples, teach us that “implicit” is “complex”. The flip side of that is where the problem lies. In Clojure “formal circles” the “inferential” belief is that “explicit” is “simple”.

I believe that neither implicit nor explicit can be applied to simple without a context. And no, a “well implied implicit” does not mean complex. And no, explicit does not mean simple.

For example, I need to create a local scope and bind some values: if I am given a choice to write an identity monad or to use a Clojure let binding, I would choose the let binding, because it is a great syntactic implicit. An identity monad would also work, but being explicit here does not buy me any simplicity.

An interesting quality of a good implicit, by the way, is “automatic” understanding of what’s implied: that’s how we laugh :)

Respect and Doubts

This brings me to the overuse of “explicit formalism” in Clojure. On one hand it cannot be subjectively complex, since we know that “simple” is objective. On the other hand it can, because a “single fold” can be defined very differently in the explicitly formal solution by me and by people who created it.

I say: let’s listen to each other, rather than teach and preach.

Jan 16

Clojure Mindsets

After Dan’s post, we had a great discussion on reddit that was not a religious discussion, but a “civil and thought-provoking” discussion. And that, in my mind, is the greatest power of Clojure.

And The Winner is…

Yes, we have not established which is better mount or component, but that was never the point. The great summary of the discussion was a realization that @yogthos and @weavejester made at the end:

Overall, it sounds like both approaches are actually fairly similar: Mount uses namespaces and vars where Component uses records and protocols, but the purpose is broadly the same.

This is interesting for two major reasons:

* It advocates the proper usage of Component with using protocols over records/components

* It draws a clear parallel between Mount and Component, so it is a lot easier to choose which one fits your coding style / (formal vs. simple) mindset.

Libraries vs. Frameworks

Since I am obviously biased towards Mount :)..

Component manages protocols and records, and in order to do that it requires a whole app buyin, which makes it a framework. Mount does not need to manage namespaces and vars, since it is very well managed by the Clojure Compiler. Which makes it a library, which I prefer. You may not, and I respect that.

Math vs. Music

Another thing I keep noticing is that “formally” inclined people prefer records and protocols for many solutions, and for them it is easier to reason about applications since it fits better into the formal mindset.

What I disagree with is equating “more formal” with simpler. It is simpler for a formal mindset, yes, but it is not simpler for me. Explicit formalism complects things for my mindset. I do not believe there is an objective truth here, since we are just different. But I believe Clojure has both: which allows for both mindsets to coexist.

What I like protocols for is an optimized dynamic dispatch, creating libraries (which use protocols internally), and a way to tame the expression problem if and when it arises. Protocols make it simple for me.

“Frogs All Look The Same”

And to make it more interesting, I am currently on the train going back to Philly from the great NYC Lisp meetup, where Gerald Sussman presented a way to write systems that evolve and expand to handle any future functionality that is thrown at them (“extensible generics”).

He drew an interesting parallel between software systems and biology saying that “many biological mutations are fatal, but as a result we end up with extremely robust systems“, doesn’t sound very formal to me 😉

Jan 16

Don’t Fear: The Quite Global Mutable State

Global Mutable State is bad.
Global Mutable State is bad.
Global Mutable State is bad.

But which state and in what context?

Your Oracle is Bad

Do you use a database? Because if you do, it is a Global Mutable State. Is it bad? I don’t think so, it’s very useful.

The mutability concept is “bent” by databases like Datomic since it introduces a time dimension, but even in Datomic, for most cases, you only need to know “the latest” value, and it is going to mutate. Yes, over time, but nevertheless, for an application that cares about a stock price: it mutates. This is just how we understand the world: it is our own Global State and it Mutates over time, and it’s great!

Your Oracle Connection is Bad

Do you ever connect to a database? Because if you do, it is a Global Mutable State. Oh, wait, it should not be global, it should be well encapsulated and only used in functions / components / classes / modules / local bindings that need it right? We should narrow it down as much as possible.

All correct. But… narrow it down all you want, if you rely on this database connection in 42 places in your codebase and this connection dies, it affects all 42 places, no matter how well “localized” they are. So why not embrace this fact and just admit that this is a Global Mutable State: e.g. you can call connection.disconnect() in any of these 42 places, and it is going to affect all the rest. If it is easier, we can call it a Quite Global Mutable State.

Your Configuration is Bad

Do you rely on any configuration files that you load from Zookeper / etcd / file system? Ah.. but they are not mutable. Ah.. but they are.

Say you support reloading these configs at runtime, a pretty common pattern, and quite useful for certain systems. Every time you mutate it and reload, it affects all components of an application that rely on it. “Yes, but don’t put it as a top level mutable reference, so a beginner programmer can mutate it by mistake, and then it is not easy to reason about it”… Ok, somewhat a fair point => you can still keep it as a global reference, if you need, but point it to an immutable map: done.

Your Salary is Bad

Some questions with the same answers as above:

* Is your network socket connection bad?
* Is your global threadpool bad?
* Is your queue bad?
* …

But this is not to say that any state is naturally a Global Mutable State, or that we, as programmers should not worry about it, quite the contrary, we should take a great care about the “business”/”problem” state.

Say you have an application that deducts taxes from peoples’ salaries, and does so concurrently. If you have your “next salary” as a Global Mutable State, you might end up borrowing for the mortgage this month.. computer error, oops.

The point is: resources, I/O, external storage, etc.. are all examples of the real Quite Global Mutable State. Yes, this state can be hidden from certain components, and only available in some, but its change(s) immediately affect all the components that rely on it, which is “quite global” as the state goes.

You are Bad

You change over time, well I do. We evolve and mutate. Is it bad? Should we be caged into the smallest possible cages so it is easier to reason about us? :) I don’t think so. Ok, so instead should we expose ourselves to anybody who can hurt us? Also, no. We choose how to present ourselves to the Universe, so why can’t we have the same choices in software? We are smart, we can do this!