holy crap, my world is flat!

This realization has crept upon me. Even though I’ve talked about the flatness of information structures before, I’m seeing a strong pattern emerge which is, well, kind of blowing my mind. Keep in mind, I am a serious structuralist. I spent endless hours as a kid constructing vast cities out of pillows and furniture. I’ve always been drawn to the 3 dimensional representation of how we think, knowing full well that I could never be satisfied with a simple hierarchy of knowledge. Later in life, when I heard about neural networks, I was very turned on. Everything could connect to anything. Everything does connect to everything.

Point clouds

The flattening of my world finally hit an interesting point as I began to work with Google Refine (soon to be relaunched as Open Refine). I was trying to rework an Excel Spreadsheet for Ying Chan’s graduate seminar on Food Security. We found this very interesting dataset which maps food issues based upon news reports. The spreadsheet was in Chinese, but I wasn’t so worried. I was probably more taken back by the fact that I didn’t know all the names of China’s provinces in English, so what’s the big deal if I don’t know the Chinese. Besides, it’s all just a pattern. Anyway, the data was weird. I could see it immediately. The column which listed the location from the report, was a multi-valued set (comma separated), which listed the various regions in which the food issue occurred. My goal with this data, was to produce a map, using Google Fusion Tables. So before I did anything, I needed to sort out this weird column.

Now, I’ve always hated working with spreadsheets. This may have something to do with my lack of financial acumen, or might be a product of my early introduction to relational tables, but I’m pretty sure it is grounded in my deep need to create a structure – a multi-dimensional structure from the information. Simply put, I wanted this data in a database. I wanted to create at least a 2nd normal form relational database, so that regions could be linked to incidents. But because of how I was planning on working (using fusion tables), I needed to keep this data all flat. I resigned myself to this substandard position and began to work with Google Refine.

This problem was rather trival, once I could get past the non-trival – namely my own built-in world-view. I first has to fill all empty fields, so that wouldn’t be affected later, but then I simply had to split on multi-valued cells and fill down the empty cells. I now had a flat (and redundant, I kept muttering) structure. But it was ready to be worked with.

This whole time I am amazed at the power of Google Refine, as well as what is possible with both Drive and Fusion Tables. Because they all so neatly link together, they make manipulating and moving the data so easy. And even though I still was struggling with the idea of work with spreadsheets, I began to understand how important this simple structure is.

Although I believe all humans like structure, and even like me, like to form deep understanding of the multitude of relationships that can exist, when it comes to simply doing, we love taking the straightforward path. That’s where we line up everything we know and we duplicate our data, so we can audit and see it for what it is. We don’t want to have to juggle relationships in our minds when it comes to the affairs of the work in front of us. Sure, there is a deep satisfaction from being able to handle such complex relationships, but all of us, aren’t always able to deal with that, day to day.

This flattening of our world – the one I write (and think) about with respect to content is no different. But for some reason, I’ve been hanging on to the notion that it didn’t apply to data, to more pure datasets. The reality is, it is the same.

Republic, Lost and #Lesterland

I watched Lawrence Lessig’s recent TED talk, in which he makes the argument about the corruption in the United States. I enjoyed it, because I love the ideas in the book. I’m not quite finished, but close. But the TED talk I found a little odd. Specifically, his introduction of this analogous place called Lesterland. So I quickly tweeted my thoughts (intelligent and coherent like all tweets:

Well, Lessig responded asking why I thought this and although I quickly tried to respond  (tripping over my embarrassment, since I have been a tremendous fan and have read all his books), I don’t think I did it justice.

What I didn’t like about Lesterland is that it is was a very thin analogy. It used some Simpson characters to portrait the Lesters, but from the first slide, we were looking at a map of the United States. Clearly Lesterland is the United States. So, I didn’t know why he was going to the effort to use this thinly draped analogy. In the Republic, Lost, Lessig spends most of the book countering the responses that people will have to the idea that Congress is corrupt. He is set on using this word, which is obviously loaded. He knows people will object to it, because corruption implies corrupt people. So he emphasizes another type of corruption: systematic corruption. And this is, ultimately, want he wants us to focus on, because it is what is preventing all of us from having a functioning government. Great ideas. Yet, somehow this word, corruption, which is so loaded, forces Lessig in his TED talk to try to find a gentler way to introduce the idea that money is unduly influencing governance.

The Lesterland analogy doesn’t do what Lessig had hoped. No one is fooled, so it just distracts. The substance of his lecture is there, but the whole part about Lesterland doesn’t advance the idea. He would be better off finding or creating an analogy which doesn’t so easily map to the United States, so that people will understand it and then make the connection. This is the power of analogies. You agree to something and then once you make the connection, you find yourself agreeing to an idea you might not easily have agreed to, given certain internal biases.

I wish I had a suggestion for him. Not that he really needs my help. Lawrence Lessig is far smarter than I will ever be. Yet, somehow I was disappointed that he used a trite and simply not very effective analogy to introduce such an important idea. Money, itself, plays this kind of systematic influence the world over. Many of us know this and believe it, but have trouble seeing it in ourselves. Admitting it would paralyze us and prevent us from ever believing we could make a change. We like the traditional idea of corruption because it points to bad actors. This has always been our issue with Good and Evil. It’s difficult to see such things as black and white when it touches our own behaviour.