At the end of November AWS announced that HBase on EMR supported S3 as data store. That’s great news because it means one doesn’t have to keep around an HDFS cluster with 3x replication, which is not only costly but it comes with its own operational burden.

At the same time we had some use cases that could have been addressed with a key-value store and this seemed like a good opportunity to give HBase a try.

What is HBase?

HBase is an open source, non-relational, distributed key-value store which traditionally runs on top of HDFS. It provides a fault-tolerant, efficient way of storing large quantities of sparse data using column-based compression and storage.

In addition, it provides fast lookup of data thanks to indexing and in-memory cache. HBase is optimized for sequential write operations, and is highly efficient for batch inserts, updates, and deletes. HBase also supports cell versioning so one can look up and use several previous versions of a cell or a row.

The system can be imagined as a distributed log-structured merge tree and is ultimately an open source implementation of Google’s BigTable whitepaper. A HBase table is partitioned horizontally in so called regions, which contains all rows between the region’s start and end key. Region Servers are responsible to serve regions while the HBase master handles region assignments and DDL operations.

A region server has:

• a BlockCache which serves as a LRU read cache;
• a BucketCache (EMR version only), which caches reads on local disk;
• a WAL, used to store writes not yet persisted to HDFS/S3 and stored on HDFS;
• a MemStore per column family (a collection of columns); a MemStore is a write cache which, once it accumulated enough data, is written to a store file;
• a store file stores rows as sorted key values on HDFS/S3;

HBase architecture with HDFS storage

This is just a 10000-foot overview of the system and there are many articles out there that go into important details, like store file compaction.

EMR’s HBase architecture with S3 storage and BucketCache

One nice property of HBase is that it guarantees linearizable consistency, i.e. if operation B started after operation A successfully completed, then operation B must see the the system in the same state as it was on completion of operation A, or a newer state. That’s easy to do since each row can only be served by one region server.

Why isn’t Parquet good enough?

Many of our datasets are stored on S3 in Parquet form. Parquet is a great format for typical analytical workloads where one needs all the data for a particular subset of measurements. On the other hand, it isn’t really optimized for finding needles in haystacks; partitioning and sorting can help alleviate this issue only so much.

As some of our analysts have the need to efficiently access the telemetry history for a very small and well-defined sub-population of our user base (think of test-pilot clients before they enrolled), a key-value store like HBase or DynamoDB fits that requirement splendidly.

HBase stores and compresses the data per column-family, unlike Parquet which does the same per column. That means the system will read way more data than it is actually needed if only a small subset of columns is read during a full scan. And no, you can’t just have a column family for each individual column as column families are flushed in concert. Furthermore, HBase doesn’t have a concept of types unlike Parquet. Both the key and the value are just bytes and it’s up to the user to interpret those bytes accordingly.

It turns out that Mozilla’s telemetry data was once stored in HBase! If you knew that then you have been around at Mozilla much longer than I have. That approach was later abandoned as keeping around mostly un-utilized data in HDFS was costly and typical analytical workloads involving large scans were slow.

Wouldn’t it be nice to have the best of both worlds: efficient scans and fast look-ups? It turns out there is one open system out there currently being developed that aims to feel that gap. Apache Kudu provides a combination of fast inserts/updates and efficient columnar scans to enable multiple real-time analytic workloads across a single storage layer. I don’t think it’s ready for prime time just yet though.

What about DyamoDB?

DymanoDB is a managed key value store. Leaving aside operational costs, it’s a fair question to wonder how much it differs in terms of pricing for our example use case.

The data we are planning to store has a compressed size of about 200 GB (~ 1.2 TB uncompressed) per day and it consists of 400 Millions key-value pairs of about 3 KB each uncompressed. As we are planning to keep around the data for 90 days, the total size of the table would amount to 18 TB.

HBase costs

Let’s say the machines we want to use for the HBase cluster are m4.xlarge which have 16 GB of RAM. As suggested in Hortonwork’s HBase guidelines, each machine could ideally serve about 50 regions. By dividing the the table into say 1000 regions, each region would have a size of 18 GB, which is still in the recommended maximum region size. Since each machine can serve about 50 regions, and we have 1000 regions, it means our cluster should ideally have a size of 20.

Using on-demand EMR prices the cluster would have a monthly cost of:

This is an upper bound as reserved or spot instances cost less.

The daily batch job that pushes data to HBase uses 5 c3.4xlarge machines and takes 5 hours, so it would have a monthly cost of:

To keep around about 18 TB of data on S3 we will need 378 $ at 0.021 $ per GB. Note that this doesn’t include the price for the requests which is rather difficult to calculate, albeit low.

In total we have a cost of about 5500 $ for the HBase solution.

DynamoDB costs

DynamoDB’s pricing is based on the desired request throughput the system needs to have. The throughput is measured in capacity units. Let’s assume that one write request per second corresponds to 3 write capacity units as one unit of write capacity is limited to items of up to 1 KB in size and we are dealing with items of about 3 KB in size. Let’s also assume that we want to use a batch job, equivalent to the one used for HBase, to push the data into the store. Which means that we want enough write capacity to shovel 400M pings in 5 hours:

which amounts to about 6510 $ a month. Note that this is just the cost to push the data in and it’s not considering the cost for reads all day around.

The cost of the storage, assuming the compression ratio is the same as with HBase, is:

Finally, if we consider also the cost of the batch job (788 $) we have a total spending of about 11800 $.

In conclusion the HBase solution is cheaper and more flexible. For example, one could keep around historical data on S3 and not have an HBase cluster serving it until it’s really needed. The con is that HBase isn’t automagically managed and as such it requires operational effort.

How do I use this stuff?

We have created a mirrored view in HBase of the main summary dataset which is accessible through a Python API. The API allows one to retrieve the history of main pings for a small subset of client ids sorted by activity date:

view = HBaseMainSummaryView()
history = view.get(sc, ["00000000-0000-0000-0000-000000000000"])

We haven’t yet decided if this is something we want to support and keep around and we will make that decision once we have an understanding of its usefulness it provides to our analysts.

Like most of you, I have no idea what I am doing 98.3% of the time I make a decision. Of that slice of the pie, I am actively getting it not-even-wrong 93% of the time. I bumble through on the strength of the 5.3% of the time I get it right by accident, and the 1.7% of the time a dim sense of direction lasts long enough that I can think more than one step ahead.

Of course, being the geniuses we are, we use almost all of the slim edge we possess over randomness — to the tune of 6.9%  of the available 7% — to convince ourselves we actually know what we’re doing all the time. This evolving, self-congratulatory narrative of determinate agency is what we humans call “culture.” We dislike indeterminacy almost more than non-survival.

This means our survival, at individual and collective levels, rests on our actions during the 0.1% of the time we:

1. Have a dim sense of what the hell we are doing, and
2. Are doing something other than congratulating ourselves about it.

Fortunately, thanks to the miracle of compound interest, over hundreds of thousands of years, we’ve collectively translated this thinnest of thin edges into an altered environment that is highly forgiving to having almost no idea what we’re doing, almost all of the time. This environment is what we humans call “civilization.”

This view is basic calibration concerning the human condition. If you’re lucky, you’ll figure this out at some point. If you’re really lucky, you’ll promptly forget it.

But if you’re really, really lucky, you’ll never figure it out at all and turn into a leader.

***

Dramatizing only a little, my first encounter with leadership was probably around 1993. There I was, screwing around, with no idea what I was doing, when a Tall Handsome Man walked up to me.

THM: “What are you doing?”

Me: “I have no idea.”

THM: “Oh, how about I leader you a bit then?”

Me: “Leader me? Don’t you mean lead me?”

THM (cautiously): “What’s the difference?”

Me: “I suppose if you convince me I am doing the wrong thing or doing the thing wrong, and that I ought to be doing something else or doing it some other way, that’s leading.”

THM: “That sounds like a lot of work. What’s leadering?”

Me: “I have no idea. That’s your word.”

THM: “Fine then, I will leader you. Is what you’re doing working?”

Me: “Well, I’m not dead yet.”

THM: “Carry on then, I’ll be back to leader you some more in a bit.”

While I haven’t yet met anyone capable of actually leading me or anyone else, a lot of people have tried leadering me over the years. Unlike being managered (a story for another day), which is not too intolerable if said manager protects you from some paperwork in exchange for you humoring his or her delusions of psychological insight, being leadered has no benefits at all. It is just plain annoying.

At some point, I got sick of leaders getting in my way. So I wandered off, muttering under my breath and occasionally yelling “STOP LEADERING ME!” over my shoulder to people who were still trying to leader me. Since then, I’ve been mostly wrong, most of the time, all by myself. But as in 1993, I am not dead yet. Contrary to the popular interpretation of such survival, this does not necessarily mean I am doing something right. It just means I am just avoiding doing fatally wrong things.

See, the difference between leading and leadering is that leading is an extraordinarily rare event: one person getting it right for 20 seconds instead of 5 seconds. And in those 20 seconds, getting enough right, and getting it right enough, that the precious, gooey rightness can be shared with others. When some of this precious, gooey shared rightness  gives an entire group a bit of an edge for a while, we call it leading.

Given the default randomness of the human condition, and the extreme power of compound interest, a little bit of leading goes a long way. Many thriving corporations, for instance, live out their entire lifespans fueled by about five minutes of actual leadership. Sometimes those five minutes can even be attributed to the person who later graduates to full-time leadering.

Episodes of actual leading are rare enough that they do not constitute pervasive, persistent and effective behavior patterns. So we do not in fact need a noun like leadership. Most of what passes for leadership is in fact systematic and self-serving misunderstanding of the pervasive, persistent and ineffective (but mostly harmless) behavior patterns corresponding to the verb leadering. 

Leadering is the art of creating a self-serving account of whatever is already happening, and inserting yourself into it in a prominent role. This requires doing things that don’t mess with success (and the baseline for success is continued survival), but allow you to take credit for it. Successful companies might have only about five minutes of actual leading in their stories, but they have hour after endless hour of leadering.

Look around you. Chances are, somebody is doing some useless leadering within stone’s throw of wherever you are. If you have a stone handy, you might want to chuck it at them. You’ll probably miss, given your likely median 93% miss-rate, so you likely won’t get in trouble, but it will feel good.

Leadering is pervasive in nature, and since most people have no idea how evolution works, they tend to assume it must therefore serve some hidden useful function.

It mostly does not.

Most of the time, leadering is  neither adaptive nor maladaptive, but superfluous. It is part of the tolerable burden of non-functional behavior, subsidized by functional behavior, that we carry around as a species. In other words, leadering is the behavioral analog of junk DNA. Sound and fury signifying nothing, which only exists because it takes an order of magnitude more work to eliminate than to tolerate. This is a corollary to Alberto Brandolini’s bullshit asymmetry principle: it takes an order of magnitude more effort to refute bullshit than to create it.

During times of chaotic change, however, leadering goes from being tolerably irrelevant to being actively harmful and FUD-creating. This is because so much attention is focused on the showy leadering that everybody involved misses the opportunity to do the few minutes of actual leading that is necessary during such change.

A lot of what I do is allegedly about helping executives lead. But to be honest, what I really do is try to get them to stop leadering so much.  Hopefully that increases the chances that they’ll spot and exploit the rare opportunities to actually lead when they float by.

Sometimes this is impossible because people can  become so addicted to leadering that they resist any attempt to lower the amount of it they do. When I meet such leadering addicts, I tend to retreat hastily. Even though enabling and amplifying leadering is a far more lucrative business than eliminating it, it is really annoying work that takes a particular kind of extreme patience that I lack.

To understand human leadering and its modern, somewhat less unuseless form, agile leadering, it is useful to understand it in a broader context, with reference to leadering in other species.

***

After humans, lions probably exhibit the most familiar example of leadering in nature. Leadering in lions is such a convincing fiction that it took biologists decades to work out that it did not in fact represent one lion wisely leading others, Mufasa-style.

At first, biologists thought the noble, large alpha-male lions did the leading, guiding the pride through life in the perilous savanna with foresight and courage.

After some research, it turned out that the noble, large alpha-male lions mostly did a lot of roaring, strutting around, screwing and cubs-of-rivals slaughtering. The only somewhat functional part of the theater of sound and fury is that it drives herds of dimwitted wildebeest or zebra into stampeding panics. Then lionesses leap out from strategic hiding places to do most of the thankless work of actual hunting (in human contexts, these behaviors correspond to marketing and sales respectively).

After some more research, it turned out that even this was giving lions way too much credit. The essence of lion leadering is in fact sneaking up on hardworking hyena packs just before dawn, right after a successful hunt, stealing their kill, and posing for wildlife photographers in the dawn light. These pictures are then turned into soul-stirring motivational posters for humans, about the courage and solitude necessary for successful leadering.

There is a reason lions are among the charismatic megafauna, claiming a disproportionate share of nature conservation funds. And it is not entirely a coincidence that charisma is also the primary trait that enables leadering in humans.

Lion leadering then, is at best a lot of noisy roaring to cause panics. More often, it is a case of simple schoolyard bullying to steal lunch from smaller animals. At worst, it is about killing the cubs of outgoing alpha males during a hostile takeover of a pride.

What makes this behavior work out is really the even more depressingly stupid behaviors of wildebeest, zebras and hyenas. They have even less of an idea what they are doing than lions or humans, so they have to do it in larger, faster-reproducing groups to make the odds of continued species survival work out.

The thing is, all social species seem, at first glance, to exhibit wonderful and noble patterns of leadership and cooperation. These turn out, upon more careful examination, to be theaters of showy leadering enabled by indifference or greater cluelessness among the leadered (and slaughtered). Ever since Mark Twain pointed out the first example of leadering and “cooperation” in ants over a century ago, examples have been cropping up all over the place.

Take geese for instance.

What seems to be a case of enlightened and collaborative leadership — the birds take turns being the apex of the V formations we so admire — turns out, upon further examination, to be a case of lazier birds free-riding in the wakes of slightly more energetic ones. As the most energetic bird tires, the next-least-lazy birds ends up at the tip of the V.  The only reason they even head in the same direction is that they all have compasses of sorts, and share the same, but not necessarily correct, genetically coded idea about where to go. Given plate tectonics and other inconvenient truths from earth’s geological and geomagnetic history, the routes of migratory species are not exactly models of optimality.

Humans being suckers for inspirational lies, of course we turn pictures of flocks of geese flying off into the sunset into motivational posters with soul-stirring captions about collaboration and shared leadership. And we mount them right next to similarly captioned pictures of lions, posing with high gravitas over stolen hyena lunches, in the flattering light of dawn.

If you really want to understand how human leadering works though, you have to consider the murmuration of starlings. Once you’ve finished oohing and aahing at the strangely ordered, rapidly shifting patterns of flocking they create, consider this: starlings mostly just murmur (murmurate?) for collective safety from predation. The rest of the seemingly complex behavior is just the mathematical consequences of simple flocking rules. About 93% of the time, the turn-on-a-dime aerobatics serve no higher purpose.

To simplify a bit, all starlings seem to do is head in the same direction as their neighbors. Unlike migrating flocks of geese, they don’t even have a strong shared sense of the wrong absolute direction, so the movements are largely meaningless. It just looks very pretty, because the flocking behavior is at the edge between order and chaos. Small local disturbances can drive the whole flock suddenly in a new direction.

I suspect the behavior evolved to settle in this regime because it allowed enterprising starlings to do some leadering by pretending there is deeper meaning hidden in the cryptic order. Somewhere, right now, there is a starling leadership conference going on, where a starling leadership coach is making reverential, cryptic remarks about emergent tribal leadership and spiritual strange attractors.

As I said, biologists do have theories about how the agile dancing of starling flocks is about avoiding predators like hawks, but let’s face it: they (the starlings that is, though possibly also the biologists) basically have no idea what they’re doing. They just do it in somewhat more visibly beautiful ways than other species. Occasionally, the behavior may pay off in terms of avoiding hawks, but most of the time, the behavior has no significance. The spectacular display of swarm aerobatics is merely what starlings call “culture” and I call “flying bullshit.”

The point of sampling the collective behaviors in the natural world is that it allows us to describe the social contexts in which human leadering practices occur more clearly. There is, after all, no leadering without a base of natural collective behavior to take credit for. Collective human behavior, it turns out, can be reasonably approximated as a mix of lion, goose and starling collective behaviors.

Alpha-male lions, posing for pictures at dawn over stolen lunches, are of course, the epitome of a pride. As the name suggests, the behavior is more charismatic show than substance.

Geese flying in a V, according to a shared sense of direction and a laziness-and-free-riding formation flight model constitute a skein. Depending on the relationship between their current destination and earth’s plate-tectonic and geomagnetic history, their sense of direction is somewhere between good enough and hilariously off.  Thanks to survivorship bias, the paths of surviving species work well enough to perpetuate them, but that does not mean they are either right or capable of surviving further disruption. We just don’t know much about the now-extinct species that got hopelessly lost the last time a continent drifted a bit or the earth’s magnetic pole flipped.

Starlings doing their cryptic strange-attractor dances constitute, as I have noted, a murmuration. Depending on whether or not there are any actual predators around, much of that agile dancing is a waste of energy. Unlike prides, murmurations have no charismatic alpha males taking credit. Unlike skeins, there aren’t even any situational leaders taking turns. I am tempted to call a murmuration of starlings a “leaderless organization”, except that to do so would be to assume that the presence of leadering in lions and geese is more significant than it actually is.

Humans leaders believe that what they do is mostly real leadership rather than leadering. They imagine it is 90% about lion-like wise leadership, 9% setting a goose-like sense of shared and correct direction for all, and 1% cryptic, leaderless, starling-like murmuration.

In practice, human leadership is almost entirely leadering. It is 90% flying starling bullshit that just looks meaningful, 9% poorly calibrated goose-like navigational inertia reflecting ancient realities that have almost certainly shifted, and 1% lion-like roaring and posturing by a charismatic few.

Whatever the actual or claimed ratio, leaders like to  call the three elements agility, mission and vision. Until recently, most leaders only admitted to the presence of mission and vision in the behaviors of the groups they led. That they even acknowledge the presence of agility these days is a huge retreat in the claims made by leadership narratives. So you should not be too upset that they only admit 1% agility rather than 90%.

We need a name for human groups that exhibit this behavioral profile of collective and leaderly behavior. Let’s call it a twittering.

***

Let’s switch terminology to bizspeak. A twittering of humans is some mix of agility, mission and vision.

1. Agility is seemingly significant flocking behavior that involves changing direction frequently and quickly.
2. Mission is navigational inertia, usually with forgotten origins, sacralized into things called values. 
3. Vision actually has nothing to do with the waterfall picture a leader paints, and everything to do with the lion-like posturing that accompanies such painting.

The main job of a human leader is to pretend that the mix of factors driving success is 90% vision, 9% mission and 1% agility, when in fact the mix is 1% vision, 9% mission and 90% agility. The reason they fudge the ratio (aside from the fact that many actually believe it) is that a vision-dominant leadership image gets you lion-like wages. Mission-dominant leadership narratives makes you just one temporary goose among many, paid according to how long you actually stay at the apex of the V. Agility dominant leadership narratives are the worst, since your claim to doing effective leadering might vanish with the next change of direction.

Being a “leader” in the sense of being the starling that is closest to the average direction and geometric center of the flock is so ephemeral a condition that it might get you no more than a pat on the back and fifteen minutes of fame. Lions get big paychecks and bonuses, geese get bonuses when they do play V-apex  so it is smarter to pretend you’re a lion if you can and a goose if you cannot. The starling leadership narrative gets you nothing.

Most of the time, misreporting the actual mix is harmless. Leadering being the junk DNA of organizational behaviors, it merely imposes a tolerable burden. Under benign and slow-changing conditions, it really doesn’t matter what the mix is and whether the CEO pretends to be a lion, first among a bunch of equal geese, or one anonymous starling among thousands. Everybody gets to enjoy a false sense of determinate control over the narrative, and some also get paid more for their more charismatic roles in the theater.

But when the environment is changing, assuming the wrong ratio can be fatal. The false sense of high agency inherited from peaceful periods turns into a kind of blindness to whatever little agency does exist in chaotic, transient times.

During such times, agility is worth more than a mission, and a mission is worth more than a vision.

Or to put it in terms of embodied intelligence, agility contains more intelligence than missions, and missions contain more intelligence than visions. Visions contain so little intelligence that they are practically homeopathic in their ineffectiveness.

Agility is the least-wrong component. Generally, doing random things is better than doing nothing, for the same reason playing the lottery is better than doing nothing when you are down to your last buck. In the best case, you might win. At the very least, you will present fewer exploitable patterns to predators, and won’t get stuck in a game-ending rut doing the same wrong thing over and over again. No matter what the environment is like, the right amount of agility is not zero. There is always a benefit to a certain amount of random direction changing.

Mission is the next-least-wrong component. If you have no idea where to go, it probably makes sense to at least head together in roughly the direction that worked at least once before. A rough sense of direction is much more robust than a detailed plan. It takes something truly rare and literally earth-shaking — such as plate tectonics in the case of really old migration routes — to make a sense of direction maladaptive. The key to successful missions is to only be wrong in ways that don’t really matter to survival, such as belief in a god, and sharply limit the number of things you actually need to be right about. That means there will be fewer ways for you to be catastrophically wrong.

Once set, a sense of rough direction can often last a really long time, but this does not always mean there is any fundamental antifragility that can resist all kinds of change. My favorite example is the case of Kongo Gumi, once the longest-lived corporation. It tamely succumbed to very banal financial problems in 2007, after 1400 years of existence. You’d think such a venerable business would end in a grander way. That’s the sort of assumption too much leadering tempts you into.

The most-wrong component of leadering is vision. Unsurpisingly, visioning is the favored activity within the leadering game, since it justifies the highest pay. It is also the biggest liability during turbulent times. Not only is a clearly painted picture of the future likely to become wildly inaccurate within minutes, it also exposes to rival twitterings a risk surface of exploitable predictability, while simultaneously constraining the creativity of your own twittering.

In turbulent times, agility is essential for keeping the game alive, a mission may be useful depending on how much has not changed, and a vision is serious double jeopardy.

So really, agile leadership is no more than a willingness to sacrifice a false sense of determinate progress. This means fostering the sort of comfort with ambiguity and uncertainty that a 90:9:1 ratio of agility, mission and vision takes, and forgoing some or all of the profiteering that selling a 1:9:90 delusion allows.

Or to distill it to something even a twittering with ADD can understand: stop leadering so much.

This is humanity: a clueless, groping-in-the-dark bumbling-about swarm of 7.2 billion individuals that just barely stays ahead of randomness through the steady workings of compound interest. On the strength of this slimmest of slim edges, we weave the grandest of  grand narratives.

Sarah Perry is a contributing editor of Ribbonfarm.

This essay attempts to place ritual in the context of evolving complex systems, and to offer an explanation for why everything is so ugly and nobody seems to be able to do anything about it.

On Boundaries and Their Permeability

Boundaries are an inherent, universal feature of complex systems. Boundaries arise at all scales, defining the entities that they surround and protecting them from some kinds of outside intrusion. To be functional, boundaries must be permeable, allowing the entities to take energy and information from outside themselves. If we are looking at complex systems, we will find boundaries everywhere.

Boundaries are structures that protect what is within them and allow their contents to solve smaller, more manageable design problems than would be possible in a perfectly interconnected system.

Islands are surrounded by natural boundaries. Strange varieties of life arise on islands that are not seen anywhere else, precisely because they are cut off from the densely interconnected systems present in the ocean and on large land masses. Islands, small systems surrounded by a natural ocean boundary, give life the opportunity to try amazing stunts that would be impossible on a large landmass. Drifting daisies evolve into trees; drifting iguanas learn to swim; fruit flies evolve into fantastic, showy varieties; crabs grow a meter long and figure out how to open coconuts. Tree kangaroos, ground parrots, and giant tree skinks are niches only available on islands. And when the boundary opens (often because of that great destroyer of natural boundaries, humans), the unique species on islands are often out-competed by species that evolved in the diversity-flattening zones of great land masses. Boundaries drive diversity.

Note that the moon, however, despite being separated from the continents of Earth by a significant boundary, is not teeming with unusual life. This boundary is too great, too harsh, and not permeable enough to allow life to adapt to it in the first place.

Life itself has discovered a number of effective boundaries that allow it to diversify and flourish. The cell membrane is the most basic boundary. Trivially, there are no multicellular creatures without cell membranes; less trivially, there are no complex creatures whatsoever without this protecting, organizing, problem-space-limiting innovation. And in evolving a cell membrane, the most important problem to solve was how to get things through it: how to make it permeable enough to be useful.

The diversity explosion of the Cambrian era was aided by the discovery of a different kind of boundary: the exoskeleton. Exoskeletons provided a boundary protecting individual trilobites, which differentiated into many different forms and niches. Cell walls, evolving independently in multiple taxa, protect and shape the forms of plants.

Another kind of boundary that limits the space of problem solving and promotes diversity is Pattern 13 of Christopher Alexander, Sara Ishikawa, and Murray Silverstein’s A Pattern Language: the Subcultural Boundary that occurs between neighborhoods in cities:

The mosaic of subcultures requires that hundreds of different cultures live, in their own way, at full intensity, next door to one another. But subcultures have their own ecology. They can only live at full intensity, unhampered by their neighbors, if they are physically separated by physical boundaries.

Distinct neighborhoods and distinct ways of life can only evolve within permeable boundaries of some kind, so that they can have communication with the outside world, but not be simply absorbed, flattened, and made uniform by forces outside them.

When natural boundaries between human groups are lacking, there has been intense pressure in human history to create effective boundaries of other kinds; military arms races across poorly-defined boundaries may be seen as building a new kind of boundary. This is one interpretation of “good fences make good neighbors;” only good boundaries are capable of organizing people well. (It is important that G. K. Chesterton chooses fences for his famous analogy about reformers.) Kevin Simler considers the functions of boundaries, from cell walls to national borders (amusingly I had written most of this essay, including the cell wall analogy, before I read Kevin’s excellent take). If we think of agency as “the ability to take action,” boundaries are essential for an entity to have any kind of agency.

In the present essay I am mostly interested in boundaries around groups much smaller than a nation state. I suggest that these small group entities – overlapping entities of multiple sizes, from Dunbar-sized “tribes” to neighborhoods of 7,000 or so – have increasingly had their boundaries undermined, and have largely ceased to exist and function. I suggest that ritual, my subject for the past two posts in this series, both functions to draw boundaries and to energize and coordinate human groups within the boundary so defined. I suggest that the loss of these small groups, in favor of nation-level organization of atomized individuals, has had serious consequences for human welfare and human agency. We are missing a layer of organization essential for our happiness.

Individual Imagination, Groups, and Flourishing

Perhaps the most important question of our time is how human beings can flourish and enjoy satisfying, meaningful lives under conditions of material abundance and extreme cultural interconnectedness. In A Dent in the Universe, Venkat proposes that “imagination” is a crucial survival skill at higher levels of Maslow’s hierarchy. Here, I propose that individual “imagination” – even at its best – is woefully underpowered to solve problems of human flourishing by itself.

The lower levels of Maslow’s pyramid reflect material well-being. But material abundance is not itself the cause of anomie and angst. Rather, ancestral, evolved solutions to lower-level problems also tended to contain solutions to higher-level problems as well. As these ancestral solutions are made obsolete by solutions that are more efficient on the material level, the more ineffable, higher-level problems they solved present themselves anew. Simple abundance of food is not the cause of obesity, but rather the loss of carefully evolved ancestral diets. Our ancestors found it easy to get to sleep because they were tired from intense physical activity; we often find it a challenge to get to sleep because modern solutions to material problems do not include physical activity. We are lonely and bored not because of material abundance simpliciter, but because the specific cultural patterns that have reproduced themselves to produce material abundance have whittled away the social and psychological solutions that were built into old solutions to material problems.

Some challenges encountered at supposedly high levels of Maslow’s hierarchy

Here, I hope to motivate a humility toward carefully evolved ancestral patterns, and, especially, to the conditions and forces that allowed such patterns to evolve in the first place. The old patterns, exactly as they existed in the past, simply will not work to solve modern problems. But new, effective patterns do not come in a “flash of insight” to individuals; individuals have relatively little agency and power in shaping the way things are. Individual imagination is weak; evolution is strong.

Boundaries as Constraints on Design Complexity

Evolution, like human designers, faces constraints. Design problems that are too large, complex, and densely interconnected are unlikely to be solved by either process. Christopher Alexander opens his 1964 book Notes on the Synthesis of Form with a quotation from Plato’s Phaedrus:

First, the taking in of scattered particulars under one Idea, so that everyone understands what is being talked about . . . Second, the separation of the Idea into parts, by dividing it at the joints, as nature directs, not breaking any limb in half as a bad carver might.

What does it mean to carve reality at the joints? Alexander provides an explication with an analogy to a hypothetical design problem. He introduces the concept of “fit”, the absence of misfit, which is to say, a design that solves its problem. “Fit” is a song that is beautiful, a chair that is comfortable, a kettle that is not too heavy or expensive and that heats water quickly. “Misfit” is ugliness, discomfort, uselessness, or other failures to solve design problems.

Now imagine an abstraction, a grid of one hundred lightbulbs connected to each other, ten by ten, representing a design problem with many variables. If a light is on, it corresponds to misfit; if a light is off, it corresponds to fit. Each light (variable) is connected to some number of other variables, just as, for instance, the variable of teakettle material is connected to expense and to its capacity to heat quickly.

If all variables are uniformly connected to all other variables, the system has little chance of reaching an equilibrium of “fit” by trial and error; the problem is too complex. However, if there are zones of dense interconnectivity that are not densely connected to each other, then reality may indeed be capable of being “carved at the joints,” and less difficult design problems exist that have a chance of being solved independently of each other, whether by evolution or by human agency:

An interpretation of Alexander’s design space metaphor

The existence of these zones of dense interconnectedness, that are not densely interconnected to each other, is a prerequisite for solvability. They are surrounded by a kind of permeable boundary, as pictured. This is analogous to “information hiding” in object-oriented programming; just as a living cell controls what enters it and acts on it, an electronic “object” limits access to its internal processes, accepting and returning only certain types of information. More ominously, a “black box” with limited interaction that solves a problem effectively is often so useful that it backfires: it becomes a new, required “solution” that limits the space of future design, often ironically resulting in poor fit. (More on this in the later section entitled “Tiling Structures and Monstrosities.”)

Another relevant analogy is the concept of life and death in the game of Go. The game of Go is a game of drawing boundaries. Within a boundary of stones, a structure is “alive” if any action on it can be met with a reply that preserves it, even when surrounded by enemy stones.

A kind of programming is occurring here, a complex tree of if-then statements abstracted into shape. Only a bounded structure can “live,” but within its bounds, it cannot be vulnerable to dangerous inputs. A perfectly impermeable boundary – a solid group of stones, with no holes or “eyes” – is dead, captured when the opponent encircles it. But a structure with enough “eyes” (holes where enemy stones may be placed) lives, and is held as territory.

To summarize, boundaries function to protect their contents from harmful intrusion, to allow for the solution of smaller design problems, and to preserve and encourage diversity. On the other hand, solutions developed at small scales are of limited utility if they cannot be adapted for use on larger scales. There must be communication and connection in order for bounded entities to live. Perfectly impermeable boundaries result in stasis and death, in social and biological life as in the game of Go.

Networks allow bounded entities to communicate and coordinate, and allow solutions developed at small scales to be used more widely. However, densely interconnected systems carry inherent risks not seen on the lower levels of organization. As we have seen, good design solutions cannot emerge from systems that are too densely interconnected. Beyond that, complex networks are vulnerable to (often inscrutable) risk of large-scale collapse. And black box “solutions” often seem to reproduce themselves like a plague, limiting design space and preventing new refactorings. The next section introduces some analogies to help think about networks and the balance between boundaries and interconnectedness.

Networks and Complex Interconnected Systems

On May 30, 2002, three ice climbers died in a fall on Mt. Hood. The incident, described in Laurence Gonzales’ excellent Deep Survival (Chapters 6 and 7), involved several of the most common factors in mountaineering accidents: the climbers were roped together, without fixed protection. That means that they did not anchor themselves to the ice, but attached themselves to each other, so that if one fell, all fell. They chose to form a tightly-coupled system (via the rope), without anchoring that system physically to the ice they clung to. “When a system is tightly coupled,” Gonzales says, “the effects spread. When a system is loosely coupled, effects do not spread to other parts of the system.” Like falling dominos, the mistake of one ice climber spreads to the others he is roped to – especially in the absence of adequate protection. The hope of one or more climbers executing a “self-arrest” – stabbing an ice axe into the ice before it is too late – is hampered by the fact that the force of a free-falling human (or two) is enough to dislocate the shoulders of a human strong enough to hold onto his axe.

There are two important points to this analogy. First, in tightly-coupled (densely interconnected) complex systems, one entity can send force into the system that destabilizes the entire structure. Second, humans are not good at noticing the dangers inherent in such systems. The type of accident reported in Gonzales’ 2003 book continues to occur with clockwork regularity. Gonzales quotes a climber who characterizes climbing roped together without fixed protection as “a suicide pact” – but climbers are apparently not good at noticing when they have entered such an unreasonable pact.

There are many problems with this analogy. In the case of climbers roped together without fixed protection, all the risk comes from individual accident or mistake. In reality, the risks inherent in complex systems often come from the system itself – though individual inputs are often dangerously propagated as well, as with parasite infestation in agriculture.

Another problem with the climbing analogy is that, in complex systems, there are many levels of “falling” other than death. Certainly, the extremely interconnected system humans have built risks complete collapse and extinction. But being “roped together” in a sufficiently interconnected system may also mean that we become stuck at low levels of well-being, unable to evolve better solutions at small scales to problems not quite as dire as death and extinction. Consider healthcare and education systems that require all citizens to participate in them and prevent smaller, better solutions from evolving. Obesity, boredom, and loneliness may not be quite as bad as death, but are levels of “falling” that sufficiently connected and tightly-coupled systems impose on their member human beings, limiting their freedom to attempt smaller-scale solutions.

Consider this substitute analogy, for contrast: instead of being suspended on ice, the people are suspended in air on a commercial airplane. Here, people are not roped together, but entirely dependent on a complex system, any aspect of which may fail.

Who “chooses” when to become part of a complex system, and which of its components to accept or reject? Where is the agency located? Often, poorly-fitting pieces of complex systems seem to be thrust upon us without our consent, with no practical way to refuse it. Network effects can frustrate human agency instead of magnifying it; I call these “tiling structures.”

Tiling Structures and Monstrosities

I use the personal jargon “tiling structure” or “tiling system” to describe a system that causes itself to be replicated, tiling the world with copies of itself. Some tiling structures are biological; humans are a tiling structure, tiling all continents with copies of the same kind of naked primate. Some are technological; agriculture tiled the world with itself not by making humans healthier, taller, or less prone to famine, but by producing sheer numbers and densities of miserable people for thousands of years so effectively (despite all the famines) that other options for subsistence were tiled out of existence.

Tiling structures are one explanation for why urban design looks so uniform (and so soul-crushingly ugly) throughout the United States. Certain forms are ubiquitous because they solve certain delineated problems effectively enough to become effectively mandatory: power lines, big box retail, strip malls, freeways, parking lots, and billboards are such powerful patterns that few locales can refuse them, despite their ugliness and the constraints they impose. Education has tiled the world with itself; it is taken for granted that children are to be locked up in adult-controlled cages for most of the day. Together with the other tiling systems, education has obliterated unique children’s cultures. The democratic government that requires this kind of education (what I call the “free child caging service,” although it is certainly not free) may be regarded as a tiling structure: it tends toward more control and intrusion into the boundaries of smaller entities, constraining what they can do.

Some tiling structures are “top down,” like government education: imposed on sub-entities against their will. Others are “bottom up” – a design problem is solved in such a way that all later actors adopt it, and it gradually becomes just as mandatory and constraining as a top-down imposed pattern. The blogger Viznut examines this latter dynamic with respect to software development; instead of attempting to solve problems by refactoring from scratch and “cutting reality at the joints,” pre-existing chunks are adopted and glued together to form a monstrosity that just barely works and is riddled with misfit:

Tell a bunch of average software developers to design a sailship. They will do a web search for available modules. They will pick a wind power module and an electric engine module, which will be attached to some kind of a floating module. When someone mentions aero- or hydrodynamics, the group will respond by saying that elementary physics is a far too specialized area, and it is cheaper and more straight-forward to just combine pre-existing modules and pray that the combination will work sufficiently well.

Viznut, The Resource Leak Bug of Our Civilization

Whether or not this is a fair description of software developers, I think it is an accurate description of how people build their lives. We select from the available chunks and try to fit them together into a coherent whole – an education here, a job there, a box to live in, entertainment to pass the time. These available “life parts” tend to be black boxes in whose design we have little say. They may not fit together into a satisfying whole at all – the boat they make may not float. Perfectly adequate material solutions fail to provide essential “nutrients” – sometimes literally (as with obesity), sometimes figuratively (sunshine, eye contact, exercise). It is tempting to accuse a person who cannot make a coherent life out of the available parts of having too little imagination; however, I do not think this kind of problem is one that individual imagination is powerful enough to solve. Even the most imaginative among us will tend to build a “monstrosity” instead of a life.

We may ask a very practical question: where lies the agency that accepts or rejects certain “black box” structures or tiling systems? An important myth of our time is that voting is an effective way for individuals to have agency in a democracy. The idea that the aggregate will of millions of people is adequately expressed by voting in elections is a rather outlandish claim, and clearly one with much evidence against it, but the “plausibility structure” of democracy causes us to believe this fiction on faith.

Why democracy seems to work

One pathological boundary that has been imposed top-down by our democratic system is drug prohibition. Total prohibition, in the form of the drug war, drew a boundary that created a very lucrative niche that only the most ruthless, violent actors could fill. The drug war prevented small-scale, non-totalitarian solutions to drug problems from ever being attempted, including the kind of small group rituals that allow people to use drugs in healthy, prosocial ways. The drug war hampers small group agency even more than individual agency; individuals may use drugs underground, supplied by those violent niche-fillers, in isolation or among the dispossessed, but if groups attempt to use drugs in healthy ways, a raid is almost guaranteed. “Prescription power,” limiting the power to prescribe drugs to doctors, is part of the drug war; I do not hold much hope for medicalized drug rituals administered by doctors. Despite wide agreement that the drug war is a failure, humans do not seem to have the agency to end it.

The idea that individual “consumers” express their will effectively by choosing among the options provided in the market is a myth that is related to democratic agency through voting. There is agency in market choice, but it is limited to the options provided, which are in turn limited by what other people are willing to buy. Much of what humans want and need is not possible to supply in markets, and the chunks that are supplied are often not good materials for composing a human life. I think that only small human groups are capable of supplying these benefits that are difficult for the market to capture. Both producers and government find it easiest to tile the world when humans are atomized into individuals, rather than in small groups with appropriate, permeable boundaries; but a nation of individuals makes it difficult for anyone to experience belonging. Only a small group, and not “the nation” or, even worse, “Mankind,” can supply social belonging and even multiply agency. A small group has its own agency, in the sense that a group of 200 or 7,000 is capable of more, when coordinated together with boundaries, than the same number of individuals operating completely independently. (This is why the ideal firm size is not one; transaction costs turn out to be significant.) Firms are efficient at producing material goods and services for the market, but they are not good at providing belonging and happiness for people.

These small-group levels of organization are increasingly missing. Church membership decreases, and no new cults spring up to take their place. Work, education, legal, and residential design patterns make it difficult for local groups to form and express themselves. Rituals increasingly tend toward the spectacle rather than small group participation. And without rituals to set their boundaries and energize them, small groups cannot thrive.

Boundaries, Agency, and Beauty

A story I have heard about the Langley Schools Music Project is that one of the music teacher’s strategies was to supply the children with instruments that were tuned such that they could not make “off” notes; rather than overwhelm children with a piano full of notes, they were given gamelan or other instruments that only made harmonious notes. The children could immediately pick them up and make music, rather than having to wade through years of inharmonious noise. “Toy instruments” (the thumb piano is a beautiful and very functional example) reduce the problem solving space: there are fewer options, but they all sound good and can be combined together to make music that exhibits “fit.”

Maynard Owen Williams, writing in National Geographic in 1921, writes about what happens to folk aesthetics when unfamiliar elements are introduced:

In Merv I saw the havoc modern commerce has wrought with lovely Oriental rugs. The same thing is taking place in the peasant costumes of Czechoslovakia, with the same aniline dyes being substituted for vegetable colors, which were not only much softer when new, but which fade into mellow tones no chemical dye can duplicate.

Factories are calling the women from the farms, where they utilized the winter months in working out the designs traced by the village designer or in evolving their own. Thus, gradually the arts of the past are being lost.

Good solutions to design problems – beauty in all its forms – evolve into being as least as much as it they are created by individual human agency. The solution to design problems in the human realm have had a long time to evolve in ancestral cultures, in which they evolved under more bounded, less interconnected conditions than we experience today. When new variables are added, the old aesthetic cannot instantly absorb them, but must work through many iterations of misfit before good fit is discovered. Individual humans continue to change, elaborate, and shape their aesthetics, but the more elements (choices) are added to the problem space, the smaller the chance of hitting on a good solution.

Christopher Alexander seizes on the same example as Williams in a bit more detail in his 1964 book Notes on the Synthesis of Form to elucidate a model of cultural evolution, which I quote at length:

The Slovakian peasants used to be famous for the shawls they made. These shawls were wonderfully colored and patterned, woven of yarns which had been dipped in homemade dyes. Early in the twentieth century aniline dyes were made available to them. And at once the glory of the shawls was spoiled; they were now no longer delicate and subtle, but crude. This change cannot have come about because the new dyes were somehow inferior. They were as brilliant, and the variety of colors was much greater than fefore. Yet somehow the new shawls turned out vulgar and uninteresting.

Now if, as it is so pleasant to suppose, the shawlmakers had had some innate artistry, had been so gifted that they were simply “able” to make beautiful shawls, it would be almost impossible to explain their later clumsiness. But if we look at the situation differently, it is very easy to explain. The shawlmakers were simply able, as many of us are, to recognize bad shawls, and their own mistakes.

Over the generations, the shawls had doubtless often been made extremely badly. But whenever a bad one was made, it was recognized as such, and therefore not repeated. And though nothing is to say that the change made would be for the better, it would still be a change. When the results of such changes were still bad, further changes would be made. The changes would go on until the shawls were good. And only at this point would the incentive to go on changing the patterns disappear.

So we do not need to pretend that these craftsman had special ability. They made beautiful shawls by standing in a long tradition, and by making minor changes whenever something seemed to need improvement. But once presented with more complicated choices, their apparent mastery and judgement disappeared. Faced with the complex unfamiliar task of actually inventing such forms from scratch, they were unsuccessful.

It is frequently observed that constraints and obstructions are precisely where great art comes from; far from limiting art, they allow it to happen and feed it – the more demanding the constraints, the better. This paradoxical relationship between constraint and expression is the subject of the movie The Five Obstructions (which I highly recommend). An aesthetic is one form of a constraint, and aesthetics tend to be developed, elaborated, and enjoyed in small groups. Certain aspects of reality are excluded in order to focus on the ones within the aesthetic. An aesthetic also provides a context in which forms can exist, fit, and be beautiful (or fail to be).

The work of elaborating an aesthetic together, as a small group, providing context for each other’s selves, is some of the fundamental work of being human, a way for humans to be valuable to each other that markets cannot supply.

It is perhaps a hipster universal, and one I share, to declare that modern fashion is ugly compared to almost every “folk costume” ever devised. Clothing is cheaper and more plentiful than it has ever been in human history, and yet we all look terrible. The “chunks” that individuals may select from the market are difficult to fit together into a coherent aesthetic; most retreat into a nondescript uniform (jeans, printed t-shirts, hoodies) that is more of an apology for existing than an outfit. If people are not good at solving even so simple a problem as fashion, with the help of all those factories, how are they supposed to design excellent lives for themselves? I am not blaming anyone for bad fashion here; I am remarking that we are all deprived of a possible source of beauty and enjoyment by the lack of coherent community aesthetics. We are deprived of both context and excuse for beauty, and individuals are seldom up to the effort and social risk. It is not a problem we are well-designed to solve individually, for it is not an individual problem at all, but one of groups.

The more that economically efficient tiling systems intrusively organize us and our environments, the more beauty is crowded out and eradicated. There is a tendency to imagine that the world is fair and that there must be a trade-off – longer lives and more clothes in exchange for less beauty, perhaps. But it is not clear what entity’s agency is authorizing such a trade. Ugliness is an easy externality to impose because it is difficult for individuals to coordinate against.

Consider the leaf blower. Millions of people are subjected to loud, unpleasant noise every day so that the removal of leaves from certain areas may be performed more efficiently; this is possible because there is no mechanism for the suffering of those affected by blasts of noise to be internalized by the actors who benefit by making it a bit cheaper to blow leaves around. Are we better off because gas-powered leaf blowers exist? I submit that we are much worse off, and we have been “tiled” by the leaf blower tiling structure because there was no coherent agency to refuse this intrusion. Individuals do not have much power to stop the noise; small groups are sometimes successful in excluding them from their environment, but the continued prevalence of leaf blowers indicates the absence of widespread, powerful small-group agency capable of protecting its members’ aesthetic interests.

Noise pollution is rampant and very damaging, but a more subtle form of pollution, equally unremedied, is legible word pollution. Here is Johan Christian Dahl’s 1839 painting View of Dresden by Moonlight:

Johan Christian Dahl, View of Dresden by Moonlight (1839)

Now, here is the same painting given a modern update with word pollution:

Word pollution

Words are processed differently than non-words, and literate people are forced to process words if they are in their visual field. Words on top of beautiful things ruin their beauty. But beauty is a difficult interest to protect in our world, and ugliness externalities flourish. The advertising tiling structure fills the landscape with signs and billboards that create more ugliness than value in the aggregate, but whose damage cannot be recouped on the market. Increasingly, fashion itself is tiled with words and legible symbols – words express social meanings more cheaply and legibly than clothing without words. The result is that everyone’s eyes are constantly being assaulted with unwanted meanings that ruin visual fields that might otherwise be beautiful.

Individuals alone can only retreat from all the noise and ugliness into a walled, private world (if they’re lucky, and tolerate loneliness well). The most sensitive people are the worst off, and of course people vary in sensitivity. Small groups of humans, however, might provide real respite from the tiling structures of the world. The next section considers these entities that have become semi-mythical, described in nostalgic, archaic words like “tribe,” “neighborhood,” or “community”: small groups of humans.

Small Groups of Humans

Small groups, and not masses of disconnected individuals, are the contexts in which most ancestral solutions to human design problems evolved. There is much romanticism these days for “tribes,” a form of organization that has not been economically effective for many centuries. We still instinctively long for this kind of belonging, as our sacrifices to sports teams, fraternities, or churches demonstrate.

Vernor Vinge’s novel A Fire Upon the Deep portrays an alien species of dog-like animals, with each conscious individual made up of multiple animals, forming a group self. In military combat, after most members of a pack are killed, singleton “fragments” wander the countryside, asking other packs, “Where am I? May I be part of you…please?”

I find this depiction to be a poignant analogy for our present state as humans – except that the small group entities mostly do not exist for us to join. They have been tiled out of existence, merged into a mass of individuals who can no longer supply each other what they need. We don’t really know how to make tribes.

Pattern 12 of A Pattern Language, Community of 7,000, declares that “individuals have no effective voice in any community of more than 5,000-10,000 persons.” This is the neighborhood layer of small groups. Small groups ideally (and traditionally) exist in multiple overlapping layers, mirroring the nature of the human self.

Internet communities are becoming more valuable as sources of belonging and meaning, but their geographic boundlessness is a limitation as well as a benefit. Only local groups can protect their members’ interests in local physical space; quiet internet communities of intense connoisseurship cannot protect us from noise or ugliness in the meat world. Internet communities have less power than local communities to give us eye contact, sunshine, and exercise, or even to allow us to wear beautiful clothing in public. While I have much hope for the internet as a ritual domain, the groups that will be most powerful in solving the problems set out at the beginning of this essay will be local, on-the-ground, “colocated” groups.

It is difficult to say how to bring these groups into existence, but I think that doing rituals together is an unavoidable component. Rituals function as permeable boundaries, network people together, smooth conflict, provide beneficial mental states, and allow the group to practice the expression of its agency as a corporate entity. Whether the rituals exist for the benefit of the group, or the group exists as an excuse to do the rituals, is a matter of perspective.

In 2015 there is no shortage of options when it comes to JavaScript frameworks. Between Angular, Ember, React, Backbone, and their numerous competitors, there's plenty to choose from.

One can compare these frameworks in various ways, but I think one of the most interesting differences between them is the way they manage state. In particular, it is useful to think about what these frameworks do when state changes over time. What tools do they give you to reflect that change in your user interface?

Managing the synchronization of app state and the user interface has long been a major source of complexity in UI development, and by now we have several different approaches to dealing with it. This article explores a few of them: Ember's data binding, Angular's dirty checking, React's virtual DOM, and its relationship to immutable data structures.

Projecting Data

The basic task we're talking about is taking the internal state of the program and mapping it to something visible on the screen. You take an assortment of objects, arrays, strings, and numbers and end up with a tree of text paragraphs, forms, links, buttons, and images. On the web, the former is usually represented as JavaScript data structures, and the latter as the Document Object Model

We often call this process rendering, and you can think of it as a projection of your data model to a visible user interface. When you render a template with your data, you get a DOM (or HTML) representation of that data.

This by itself is simple enough: While the mapping from data model to UI may not always be trivial, it's basically still just a function with straightforward inputs and outputs.

Where things start to get more challenging is when we start talking about data changing over time. This can happen when the user interacts with the UI, or when something else happens in the world that updates the data. The UI needs to reflect this change. Furthermore, because rebuilding DOM trees is expensive, we'd like to do as little work as possible to get that updated data on the screen.

This is a much more difficult problem than merely rendering a UI once, since it involves state changes. This is also where the solutions out there begin to show their differences.

Server-Side Rendering: Reset The Universe

"There is no change. The universe is immutable."

Before the era of Big JavaScript, every interaction you had with a web application used to trigger a server roundtrip. Each click and each form submission meant that the webpage unloaded, a request was sent to the server, the server responded with a new page that the browser then rendered.

With this approach, there wasn't really any state to manage in the front-end. Each time something happened, that was the end of the universe, as far as the browser was concerned. Whatever state there was, it was a backend concern. The frontend was just some generated HTML and CSS, with perhaps a bit of JavaScript sprinkled on top.

While this was a very simple approach from a front-end perspective, it was also very slow. Not only did each interaction mean a full re-rendering of the UI, it was also a remote re-rendering, with a full roundtrip to a faraway data center.

Most of us don't do this in our apps anymore. We may render the initial state of our app server-side but then switch to managing things in the front-end (which is what isomorphic JavaScript is largely about). There are people that are still succeeding with more sophisticated versions of this pattern though.

First-gen JS: Manual Re-rendering

"I have no idea what I should re-render. You figure it out."

The first generations of JavaScript frameworks, like Backbone.js, Ext JS, and Dojo, introduced for the first time an actual data model in the browser, instead of just having some light-weight scripting on top of the DOM. That also meant that for the first time you had changing state in the browser. The contents of the data model could change, and then you had to get those changes reflected in the user interface.

While these frameworks gave you the architecture for separating your UI code from your models, they still left the synchronization between the two up to you. You can get some sort of events when changes occur, but it is your responsibility to figure out what to re-render and how to go about it.

The performance considerations of this model are also left largely to you as an application developer. Since you control what gets updated and when, you can tweak it pretty much as you'd like. It often becomes a balancing act between the simplicity of re-rendering large sections of the page, and the performance of re-rendering just the bits that need updating.

Ember.js: Data Binding

"I know exactly what changed and what should be re-rendered because I control your models and views."

Having to manually figure out re-rendering when app state changes is one of the major sources of incidental complexity in first-gen JavaScript apps. A number of frameworks aim to eliminate that particular problem. Ember.js is one of them.

Ember, just like Backbone, sends out events from the data model when changes occur. The difference is that with Ember, there's also something the framework provides for the receiving end of the event. You can bind the UI to the data model, which means that there is a listener for update events attached to the UI. That listener knows what updates to make when it receives an event.

This makes for a pretty efficient update mechanism: Though setting all the bindings up takes a bit of work initially, after that the effort needed to keep things in sync is minimal. When something changes, only the parts of the app that actually need to do something will activate.

The big tradeoff of this approach is that Ember must always be aware of changes that occur in the data model. That means you need to have your data in special objects that inherit from Ember's APIs, and that you need to change your data using special setter methods. You can't say foo.x = 42. You have to say foo.set('x', 42), and so on.

In the future this may be helped somewhat by the arrival of Proxies in ECMAScript 6. It lets Ember decorate regular objects with its binding code, so that all the code that interacts with those objects doesn't necessarily need to adhere to the setter conventions.

AngularJS: Dirty Checking

"I have no idea what changed, so I'll just check everything that may need updating."

Like Ember, Angular also aims to solve the problem of having to manually re-render when things have changed. However, it approaches the problem from a different angle.

When you refer to your data in an Angular template, for example in an expression like {{foo.x}}, Angular not only renders that data but also creates a watcher for that particular value. After that, whenever anything happens in your app, Angular checks if the value in that watcher has changed from the last time. If it has, it re-renders the value in the UI. The process of running these watchers is called dirty checking.

The great benefit of this style of change detection is that now you can use anything in your data models. Angular puts no restrictions on that - it doesn't care. There are no base objects to extend and no APIs to implement.

The downside is that as the data model now doesn't have any built-in probes that would tell the framework about changes, the framework doesn't have any insight into if and where it may have occurred. That means the model needs to be inspected for changes externally, and that is exactly what Angular does: All watchers are run every time anything happens. Click handlers, HTTP response processors, and timeouts all trigger a digest, which is the process responsible for running watchers.

Running all watchers all the time sounds like a performance nightmare, but it can be surprisingly fast. This is mostly because there's usually no DOM access happening until a change is actually detected, and pure JavaScript reference equality checks are cheap. But when you get into very large UIs, or need to render very often, additional performance optimization trickery may be required.

Like Ember, Angular will also benefit from upcoming standards: In particular, the Object.observe feature in ECMAScript 7 will be a good fit for Angular, as it gives you a native API for watching an object's properties for changes. It will not cover all the cases that Angular needs to support though, as Angular's watchers can do much more than just observe simple object attributes.

The upcoming Angular 2 will also bring some interesting updates on the change detection front, as has been described recently in an article by Victor Savkin.

React: Virtual DOM

"I have no idea what changed so I'll just re-render everything and see what's different now."

React has many interesting features, but the most interesting of them in terms of our discussion is the virtual DOM.

React, like Angular, does not enforce a data model API on you and lets you use whatever objects and data structures you see fit. How does it, then, solve the problem of keeping the UI up to date in the presence of change?

What React does is effectively take us back to the good old server-side rendering days, when we simply didn't care about state changes: It renders the whole UI from scratch every time there may have been a change somewhere in it. This can simplify UI code significantly. You (mostly) don't care about maintaining state in React components. Just like with server-side rendering, you render once and you're done. When a changed component is needed, it's just rendered again. There's no difference between the initial rendering of a component and updating it for changed data.

This sounds immensely inefficient, and it would be if that was the end of the story. However, the re-rendering React does is of a special kind.

When a React UI is rendered, it is first rendered into a virtual DOM, which is not an actual DOM object graph, but a light-weight, pure JavaScript data structure of plain objects and arrays that represents a real DOM object graph. A separate process then takes that virtual DOM structure and creates the corresponding real DOM elements that are shown on the screen.

Then, when a change occurs, a new virtual DOM is created from scratch. That new virtual DOM will reflect the new state of the data model. React now has two virtual DOM data structures at hand: The new one and the old one. It then runs a diffing algorithm on the two virtual DOMs, to get the set of changes between them. Those changes, and only those changes, are applied to the real DOM: This element was added, this attribute's value changed, etc.

So the big benefit of React, or at least one of them, is that you don't need to track change. You just re-render the whole UI every time and whatever changed will be in the new result. The virtual DOM diffing approach lets you do that while still minimizing the amount of expensive DOM operations.

Om: Immutable Data Structures

"I know exactly what didn't change."

While React's virtual DOM approach is pretty fast, it can still become a bottleneck when your UI is big or when you want to render very often (say, up to 60 times per second).

The problem is that there's really no way around rendering the whole (virtual) DOM each time, unless you reintroduce some control into the way you let changes happen in the data model, like Ember does.

One approach to controlling changes is to favor immutable, persistent data structures. They seem to be a particularly good fit with React's virtual DOM approach, as has been shown by David Nolen's work on the Om library, built on React and ClojureScript.

The thing about immutable data structures is that, as the name implies, you can never mutate one, but only produce new versions of it. If you want to change an object's attribute, you'll need to make a new object with the new attribute, since you can't change the existing one. Because of the way persistent data structures work, this is actually much more efficient than it sounds.

What this means in terms of change detection is that when a React component's state consists of immutable data only, there's an escape hatch: When you're re-rendering a component, and the component's state still points to the same data structure as the last time you rendered it, you can skip re-rendering. You can just use the previous virtual DOM for that component and the whole component tree stemming from it. There's no need to dig in further, since nothing could possibly have changed in the state.

Just like Ember, libraries like Om do not let you use any old JavaScript object graphs in your data. You have to build your model from immutable data structures from the ground up to reap the benefits. I would argue that the difference is this time you don't do it just to make the framework happy. You do it because it's simply a nicer approach to managing application state. The main benefit of using immutable data structures is not to gain rendering performance, but to simplify your application architecture.

While Om and ClojureScript have been instrumental in combining React and immutable data structures, they're not the only game in town. It is perfectly possible to just use plain React and a library like Facebook's Immutable-js. Lee Byron, the author of that library, gave a wonderful introduction to the topic in the recent React.js Conf:

I would also recommend watching Rich Hickey's Persistent Data Structures And Managed References for an introduction to this approach for state management.

I myself have been waxing poetic about immutable data for a while now, though I definitely didn't foresee its arrival in frontend UI architectures. It seems to be happening in full force though, and people in the Angular team are also working on adding support for them.

Summary

Change detection is a central problem in UI development, and JavaScript frameworks attempt to solve it in various ways.

EmberJS detects changes when they occur, because it controls your data model API and can emit events when you call it.

Angular.js detects changes after the fact, by re-running all the data bindings you've registered in your UI to see if their values have changed.

Plain React detects changes by re-rendering your whole UI into a virtual DOM and then comparing it to the old version. Whatever changed, gets patched to the real DOM.

React with immutable data structures enhances plain React by allowing a component tree to be quickly marked as unchanged, because mutations within component state are not allowed. This isn't done primarily for performance reasons, however, but because of the positive impact it has on your app architecture in general.