Marcus Node Bits - Let us talk about yield and generators, shall we?

· April 1, 2014

Node   Javascript   Tools   Koa

Koa Js removes need for callbacks but still have uses non-blocking code. How is that possible?

If you read the code of the last blog post you might have reported a bug or two since I was using a strange asterisk at in the getGreeting-function. Is that really valid Javascript?

And when you looked very closely you might be wondering about the “yield” right there in the middle… What kind of witchcraft is this thing anyway?

These questions and more flew threw my head when I first learned about Koa and the concepts its’s built upon. In this post I will try to explain that a bit and point you to other places where they explain this much better, if you don’t like my tries. This is just how I, a newbie to these concepts, have tried to wrap my head around it. Hey, let’s be completely transparent and say that I have to relearn this just about everyday.

It’s really not that complicated once you understood it the first time… Just like quantum mechanics…

First, let me restate that the things we’re diving into now is bleeding edge stuff. It’s proposed for the EcmaScript 6 and is included in Googles V8 Javascript engine. You will need to tweak your environment in order to get this a version of Node that can execute this. Read this for instructions.

I have learned LOADS from this awesome article by Toby Ho. Here I’m just writing down how I understood it. All credit for any good stuff here goes to Toby. Any crap is probably from me.

What is yield?

Now, on to the war… The first concept that tripped me a bit was yield.  I have not thought much about nor used yield in my mother-tongue, C#, that have had them for quite some time. Quite simple it can be described with the following code:

This will just continue looping forever, so it’s not very useful that way. But look at line 3, we’re not actually returning anything. Instead we are yielding the result. I can’t relate to that expression. It simply doesn’t mean(t) anything to mean. My mind just goes “………”.

But when I understood that:

when the code encounters a yield statement, it suspends execution indefinitely, and doesn’t start again until you tell it to

I was intrigued to say the least. Could this be used instead of callbacks when doing non-blocking, asynchronous code? But how? Let’s press on and see if that is the case.

Aha, but what is generators then?

Ok, but how can I use this then? Or in other words; how can I call this loop? Let’s wrap that loop in a function. Like this:

Please not the little asterisk at the function. This is how Javascript denotes generator functions and I’ll come back to that later. A function becomes a generator function when you have at least one yield in it. And you need to add the asterisk.

“Well, that pretty and everything, but how do we use that function then?” you ask. We have to create a instance of the generator function and then get the next element in the sequence by calling … next(). Like this:

That was the quote about mean when it said: “… until you tell it to”. The code will halt at the yield until we tell it to continue, and we tell it by calling .next() on the function instance. To run this, remember that you have to call Node with the “–harmony” flag, as described in my earlier post. This is the command I used:

node –harmony callingGenerator.js

Two things worth noting here. First, nothing happens until you call .next().The loop in the generator is doing nothing in the meantime. Like a block of code that waits for an asynchronous callback ^^. Also it’s worth noting that what we get back is a object with two properties; value and done (true/false). The done parameter equals true when the sequence is over, as shown in this example:

If you run that you will get done === true for the fourth call to .next() (with the value === undefined). The fifth call throws an exception (“Generator has already finished”).

Tying back to the heading of this section: a generator is simply a function that represents a sequence of values. We saw examples of ending and non ending sequences above. You can use the generator by creating a generator object and call next() to get the next value.  It’s like iterating through the return values of the function.

What use is this? And what’s all this talk about Koa?

This is a mind-twister and interesting and all that… but use can we have from this? Well, yield and generators is in fact the helping hand that will draw us up from callback hell and put our feet back on solid, well-tramped ground.

Did you notice the little passage about the asynchronous wait above? The thing is, when yield is called the code is waiting for you to call .next() to continue. For as long as that take. In the meantime Node can do something else that is more important than waiting. You recognize this? (Saved the fact that I wrote it above ^^). This is what callbacks gives us. Non blocking code. Sadly to the cost of nesting and tightly coupled code, where you have to pass parameters that the inner most code need.

Let’s take a look at an example on how Koa Js uses this. This example is doing some very simple logging and really made the usefulness of generators stand out for me. Start this simple site up and then access it from the terminal with “curl http://localhost:3000 -v” so that you can see the headers returned, and there you see the X-Response-Time reported. As well at the console.

Let’s see if we can break it down a bit. app.use() is how Koa defines middleware. In this case the middleware is just two simple generator functions. When the request comes in the following happens:

First, since it’s defined first, the x-response-time generator function is called. It get’s hold of the current date and time and then yields the passed in next variable. This is a Koa variable that indicates the next middleware in line

So, we end up in the logger middleware. This generator function does the same thing. Get hold of “now” and yields out the next middleware

  • There’s no more middleware defined so we are taken to the response (for any route in this case) that always sets the body to “Hello world!”

When the response is created the execution is turned back to the logging middleware that calculate the response time for this request and blurts it out on the console

The x-response-time middleware does the same thing, but saves the result into the header of the response.

If you want to you could read the middleware code like this: “Get the start time. Do whatever else you need to do. When that is done subtract the current time from the start and save the result in the header”. In a easy to read, sequential fashion. No callbacks and long indention chains.

I like that. A lot. It’s simple, short and easy to read and understand. All the hard stuff about generators… Well we don’t have to think so much about that. Yield means: “do everything else you need to do at this point”.

Here’s another example where we store the data from a post into a Mongo-database using Monk via the coroutine library Co-Monk (which is the simplest way to access MongoDb with generators right now):

Again, using curl (why not?) we can now add a couple of users with the following commands:

And then see the result with this:

curl http://localhost:3000/user -v

Pretty short and sweet for adding user, if you ask me. More interesting, let’s go through the code.

First there’s a couple of requires that bring in the things we are going to use. The “lego” pieces in Koa is pretty small so there’s usually at least a couple of them.

Line 8-11 sees us set up Monk, and use it with “mongodb generator goodness for co” via co-monk. Man I love Monk. It’s really sweet!

On line 14 and 15 we set up our routes and simple direct them to a couple of generator functions that is found below. This is how Koa calls out to other things.

In the create generator function (note the asterisk) we have two yields

  • first (line 21) we parse the body (with co-body) and if that should take time, for big file for example, we yield the response. Koa can let other execute in the meantime. Just like for a callback. But without the callback.
  • secondly, on line 27, we store the object in mongo. And better yield there too since that might take time, or at least is IO.
  • Did you see that try…catch. It’s back baby! Error handling that feels reasonable. Structured exception handling I think it’s called. The alternative is in other words …  :)

In the list generator we just yield the result from our, very simple, Mongo query. See line 38.

Line 43 fires the whole thing up.

Conclusion

Around this point when I’m using Koa I always thing to myself… So what’s the deal with generators, what did that really give me? Its actually pretty easy to forget and to me that is a good thing. Because I just write the code like I would “normally” without callbacks. Feels much more natural to me. And when I’m doing IO or Networking I remember to yield to let Koa do other things.

I have had GREAT use of a couple of articles to explain this to me.

I want to thank all those resources and humbly point to them for more, better and in-depth information. This is just my understanding, written for me to remember. Because I have now forgot how this worked again, and need to start over :)

My code is in those gists listed above or here in a repo.

I like Koa Js. I’ve written a couple of post about it. Here they will be when they are published:

Twitter, Facebook