I’m back

Well, at least I am going to use this platform to describe a few things I have been thinking about and working while I look for work in Sydney.

A few years ago, I was busy doing work for the Journalism & Media Studies Centre at the University of Hong Kong, including teaching the Digital Journalism, principles and tools course. For a while this was a good fit for me. I am still quite focused on content management, especially relating to media. I was also taken by the rapidly evolving landscape that journalism is facing. Eventually though, I found that academia was not for me. Perhaps one day I will write about that.

At the beginning of 2016, I decided to leave HKU and start a small web development and consulting firm, Aporia Limited. I knew a lot of people who needed help with their small to mid-sized sites, so it was an easy direction to head in.

I spent the last year and a half developing various projects. I wrote about a few on Aporia’s site. I may add a few posts for the last few projects, which had some interesting aspects to them. But throughout this time, I have decided that doing small projects is quite hard and not very satisfying. I like working with clients over a longer period, helping them get to somewhere completely different. Small development projects have their challenges and some are very interesting. But unfortunately many of the projects I was working on did not have the resources to think long-term. In the end, I found them not very satisfying.

So now, onto new things.

The News will follow us

I woke up this morning, feeling a little more groggy than usual. I could have slept more, but a voice from the other room started to fill my apartment. It got louder and finally entered my room, announcing that things were changing on Twitter. My eyes opened a little wider; I wanted to hear what they were saying.

As I climbed out of bed and moved in automatic mode to the coffee machine, I caught the general idea from the announcer: Twitter was going to open source their interface and stop developing it. They had realised that their value was in being a utility, not a publisher. This was amazing news!

I switched on the screen and as the voice faded, a picture appeared showing a press conference. The sound matching the video came on and Jack Dorsey, the CEO, was at a podium addressing the group of interested citizen-journalists.

“When I created Twitter, I was fascinated by emergency dispatch systems. They were built upon lots of small information updates. I saw the potential for all of us to take part in this kind of system, sharing updates on whatever we were doing. There was no business idea behind it in the beginning. It was just a useful system.”

A small message appeared below the video point to a 60 minutes interview from 2013 with Dorsey. I touched that and the press conference paused, while I caught up on what Dorsey meant about dispatch systems. After it was finished, I returned to the press conference and watched again. I was 10 minutes behind the live event, but that didn’t matter. After all, the difference between live and rebroadcast was in the eye of the beholder.

I ate some breakfast as Dorsey continued to speak:

“A lot of people have challenged Twitter’s value, because it wasn’t growing like Facebook and making the kind of money that Silicon Valley companies should make. But many people who rely on Twitter, especially Journalists and Activists, cried out to keep it the way it is. After Robinson Meyer’s The Decay of Twitter article appeared, I knew we had a deep divide between the company and the network.”

I paused the broadcast here and jumped into the shower. I needed to get going or I would be late.

After getting dressed and running out the door, I began to message my network, seeking feedback on this news. A few in my interest group around this area were sending out opinions and discussing what would happen next. I hadn’t fully caught up on the announcement yet, so I only ‘voiced out’ my excitement over the network that something good might be happening.

When I got on the MTR, I opened my mobile and the conference was ready to resume. But first I wanted to read Meyer’s article. I had read it before, but I needed a refresher. Fortunately, the Atlantic had front-ended the article with an explainer, reminding me of what I had read. I was able to see a few key points from the article, as well as highlights I had made and a few comments from my network. In about five minutes, I was refreshed. Back to the broadcast…

“What this means is that Twitter the company is becoming the utility company that many have suggested it was and should continue to be. Our focus will be on customers, paying customers. The content and the use of the network will be left to the end-points. The Net Neutrality debate has been settled. You can’t be in both the network and the content business. Although this line is still being blurred by many, especially Facebook, we are now admitting that the interface, the application that provides the content is too deeply tied with that content. If we are to continue to support and help the network grow, we must excuse ourselves from the business of developing that interface. We’ve seen third-party developers prove that they understand the hearts of the users better than us. Yes, that was a reference to our failed UI change last year, introducing hearts to replace stars.”

Several people from my network began to comment on this. The live feedback mechanism developed by Periscope had really been embraced in broadcasts.

“We are certain that this shift will open up a lot of new possiblies for communication throughout the world, and above all else, Twitter, the company, wants to be involved in shaping this future. Note that I say ‘involved’. This is what is primarily changing. We are admitting that Twitter the network is larger than Twitter the company. This is ok though. Our investors should not fear, because this step we are taking proves that we are agile and can adapt to innovation outside of our company. We will not let disruption destroy our company and our good work. Instead, we are ready to streamline our focus and grow our company along different lines.”

At this point, both the transcript of the conference and a company report of Twitter’s reorganisation were available for viewing. I wanted to see the report first. This was not your parent’s Company reports. It was an interactive story about the future changes, including earnings projects and calculators to figure out how your investments in the company might be affected. I did have a small investment in Twitter, so I quickly calculated that I could either try to sell immediately, or wait at least one year to see the effects of this shift. I would need to explore this deeper later, but for now, I was comfortable enough leaving the report.

I switched to the major opinion pieces being written right now. I had my favourite tech journalists’ channels right in front of me. A few were live streamed. Dan Gilmore preferred the stream of conscience open writing approach. I watched his text ‘write itself’ in the live editor, including corrections and comments, footnotes and links–each a potential path to go down. After returning from a footnote which led to a case study about issues in establishing a global utility, I found the article was now presented as a video intro from Gilmore with an accompanying card stack. I loved the way that stories were constructed now. Dan Gilmore was not the only author in this news package, but he was the leader of a team – the Gilmore team. This model has started to take root over a year ago, after the success of small data journalism teams proving their worth with top quality, comprehensive and live news reporting.

When I arrived at work, my screen came alive to a few preferred channels to begin to gage the general public reaction. Reddit made up a lot of this content, but I was seeing it broken down and organised by many of the people from my network. We were all collaborating now on making sense of this announcement. Changes in Twitter’s valuation and the opinions were beginning to influence the news, but what surprised me the most was that my Tweetbot desktop client was finishing updating itself. There were 3 new features available, all built upon the news of this developing story. The feature that most caught my eye, was a little button that allowed me to tell the system and my network, that I was ‘caught up’ on this news. I clicked it and moved on to other work.

Facebook does not understand friendship

As we further hand over the management of our social relationships to Facebook, perhaps we need to pause for a moment and consider whether Facebook understands friendship.

To begin with, when you meet someone for the first time, the only thing you know about them is what you know publicly or perhaps what you have been told by mutual friends. Why cannot Facebook implement such a filter? My new friends should not be able to dive into my past until I share with them those moments: photos, status updates and even friendships.

Putting friends into lists is not how we handle friendships. Each relationship is unique. Aside from the fact that we shouldn’t have to manage such a list from an user experience point of view, it is unreasonable to think of friendship in this way. Each friendship, even the most cursory one, is unique. It should be an evolving development. As a friendship grows, so should access to certain more sensitive details. Rather than putting content into lists, perhaps we can develop over time a way of organising the type of content and how we like to share. For some, pictures remain private. For others, it is our rants. I’m a firm believer in a place to write without scrutiny, either privately or anonymously. Why does this not exist in Facebook? A nice thing to consider is that Facebook supports more private musings, they may encourage more public sharing (which is obviously their goal).

Finally, we are offered the opportunity to mark certain friends as ‘good friends’, which means we receive all of their status updates. We don’t need to open the flood gates (unless we abhor FB’s filtering to begin with). Even good friends need secrets. I removed my girlfriend from this list, because I didn’t think I need to see all of her interactions on Facebook. She’s still one on my best friends. I have a few siblings still in this list, but I am very close to removing one brother because he is prolifically ridiculous!

These are just some recent musings on our online social lives. We are all still figuring out how this space works. It is important that we provide feedback to the companies that play an important role in our social space. I don’t believe these features are impossible or extreme. Facebook has an amazing group of programmers and designers. I am certain they can find a way to develop a more realistic version of our social networks.

That is, if they want to be our friend.

holy crap, my world is flat!

This realization has crept upon me. Even though I’ve talked about the flatness of information structures before, I’m seeing a strong pattern emerge which is, well, kind of blowing my mind. Keep in mind, I am a serious structuralist. I spent endless hours as a kid constructing vast cities out of pillows and furniture. I’ve always been drawn to the 3 dimensional representation of how we think, knowing full well that I could never be satisfied with a simple hierarchy of knowledge. Later in life, when I heard about neural networks, I was very turned on. Everything could connect to anything. Everything does connect to everything.

Point clouds

The flattening of my world finally hit an interesting point as I began to work with Google Refine (soon to be relaunched as Open Refine). I was trying to rework an Excel Spreadsheet for Ying Chan’s graduate seminar on Food Security. We found this very interesting dataset which maps food issues based upon news reports. The spreadsheet was in Chinese, but I wasn’t so worried. I was probably more taken back by the fact that I didn’t know all the names of China’s provinces in English, so what’s the big deal if I don’t know the Chinese. Besides, it’s all just a pattern. Anyway, the data was weird. I could see it immediately. The column which listed the location from the report, was a multi-valued set (comma separated), which listed the various regions in which the food issue occurred. My goal with this data, was to produce a map, using Google Fusion Tables. So before I did anything, I needed to sort out this weird column.

Now, I’ve always hated working with spreadsheets. This may have something to do with my lack of financial acumen, or might be a product of my early introduction to relational tables, but I’m pretty sure it is grounded in my deep need to create a structure – a multi-dimensional structure from the information. Simply put, I wanted this data in a database. I wanted to create at least a 2nd normal form relational database, so that regions could be linked to incidents. But because of how I was planning on working (using fusion tables), I needed to keep this data all flat. I resigned myself to this substandard position and began to work with Google Refine.

This problem was rather trival, once I could get past the non-trival – namely my own built-in world-view. I first has to fill all empty fields, so that wouldn’t be affected later, but then I simply had to split on multi-valued cells and fill down the empty cells. I now had a flat (and redundant, I kept muttering) structure. But it was ready to be worked with.

This whole time I am amazed at the power of Google Refine, as well as what is possible with both Drive and Fusion Tables. Because they all so neatly link together, they make manipulating and moving the data so easy. And even though I still was struggling with the idea of work with spreadsheets, I began to understand how important this simple structure is.

Although I believe all humans like structure, and even like me, like to form deep understanding of the multitude of relationships that can exist, when it comes to simply doing, we love taking the straightforward path. That’s where we line up everything we know and we duplicate our data, so we can audit and see it for what it is. We don’t want to have to juggle relationships in our minds when it comes to the affairs of the work in front of us. Sure, there is a deep satisfaction from being able to handle such complex relationships, but all of us, aren’t always able to deal with that, day to day.

This flattening of our world – the one I write (and think) about with respect to content is no different. But for some reason, I’ve been hanging on to the notion that it didn’t apply to data, to more pure datasets. The reality is, it is the same.

Eventual inconsistency

A friend just sent me a link on ‘Eventual consistency’, a term used to describe a problem of syncing data between cloud servers. This was in response to some bizarre behaviour I have been noticing in Facebook.

The term itself, eventual consistency, reminded me of Italo Calvino, for many reasons. Check out pretty much any of his books (If on a winter’s night a traveler) or stories (t zero) and you’ll know how important time was to him.

IMG_5259.JPGHis final work was a lecture he was unable to give (due to a problem he had with time), entitled “Six Memos for the Next Millennium”. In the lecture, he outlined the six important ideas (or values) that needed to be addressed in literature in the future. He did not finish the sixth memo.

“In the front of the collection his wife later published is a list of the six memos in Calvino’s handwriting, though the sixth and final is faint, as if someone had attempted to erase it.” *

“The five memos Calvino wrote cover lightness, quickness, exactitude, visibility and multiplicity. The final lecture would have been on consistency.”

For me, this is one of the more sublime ideas in the world, and only Calvino could have pulled off so brilliant a literary stunt. Perhaps the term should be eventual inconsistency, as we are all prone to problems with time eventually.

 

 

Open Data – 2014 Digital 21 Strategy – Public Consultation

Dear Gregory So and the OGCIO,

I wished to add some perspective to the role that open data can play in the Digital 21 Strategy. The passage below from the public consultation provides a very simplistic portrayal of how every government should proceed when it comes to open data. This statement does not go far enough. The success of any open data strategy is dependant upon whether it gets embedded in the culture of how all Government Departments function. It is not sufficient to say that data will be released in machine-readable formats. Instead, there needs to be a greater emphasis on the idea that much of the information that is produced by the Government should be publicly released immediately (produced to be made public). The position that Data.One has taken is exploratory at best. They seek to find ‘high-value’ datasets and help release them. This endeavour misses the point. Who is to say what value particular information may have? As part of the Journalism and Media Studies Centre at The University of Hong Kong, I know that the value of information is sometimes only seen over time and after much examination. The true benefits of open data come when a government believes in the importance of exposing how they function. This leads to greater scrutiny, but also greater understanding. The well-functioning government works with all its stakeholders, and this includes its citizens and public and private organisations.

The Code on Access to Information sits at the heart of the role of open data in Hong Kong. Many have been calling for a Freedom of Information Act to provide a legal framework in which to reliably get information. Even setting this issue aside and focusing on the relationship between a government and its information, it is safe to say that the Government is in the information business. It is essential to its health to produce as much public information as possible The Hong Kong Government has a robust press release system, and yet it suffers from two main issues. The first is that press releases are in a non-machine-readable format and there are not sufficient mechanisms to programmatically search the archives. This is a small technical issue compared to the second issue. The real problem is that HKG (and they are not alone in this) still believe their role is to explain the meaning and value of the information released. This process is an archaic view of government information. Just like Data.One’s view of ‘high-valued’ datasets, the value of information should be open to the experts in those fields. One of the many jobs that Journalists provide is to make sense of information. When we to work with raw data, we are able to connect the dots in a much more meaningful way. Instead, we are faced with decrypting a typical Government press release. There is truth in data. The Government of Hong Kong does good work, and it behoves itself to illustrate that with its data.

As a more practical suggestion for the passage below (#34), I would encourage the OGCIO to incorporate a few ideas:

The first being that HKG needs to create a ‘culture of open’. This is perhaps a slightly strange term, but it truly embodies the intent that governments the world over should have. And since the 2014 Digital Strategy is meant to be a roadmap for the next many years, it will be essential that it articulates the principles of using technology to improve its information architecture and exchange. And this is, it should be noted, not specifically a technological issue. Although technology appears to drive much of what we do, it is nevertheless grounded in deeper principles. If the section on open data does not include reference to opening and improving the access to information, by establishing a long term goal of releasing information ‘early and often’, this passage will do nothing more than position the government as a non-agile organisation.

The final point is that government departments will look to this strategy as a way to interpret their relationship to information. The more open the language is, the more encouraging the OGCIO can be. We have found in many cases that some departments are simply unsure whether they can release information when it is requested. It is somewhat telling that the Guide on the Code of Access to Information is almost 5 times as long as the actual Code. Hong Kong should learn from the United Kingdom which has set in motion a well-established FOIA, which includes a very meaningful archives law. They have also gone on to guide their employees with things like the need to define a public task. This kind of guidance will help to establish a culture of open and enable quicker innovation and advancement in technology, in a way that a prescription for more machine-readable data (and a few APIs) could ever do.

Thank you for the opportunity to provide some feedback. I look forward to seeing Hong Kong embrace its role as a 21st century global city.

Regards,
Darcy

—————————–

2014 DIGITAL 21 STRATEGY - PUBLIC CONSULTATIONReference: from 2014 DIGITAL 21 STRATEGY – PUBLIC CONSULTATION

34. Through different channels (like press releases, publications, websites, etc.), the Government releases a lot of information in different areas. However, most of such information can only be read but cannot be used. In view of the immense benefits of widening access to PSI for free and easy re-use, we propose to make all Government information released for public consumption machine-readable by default. Where appropriate, datasets will be released with application programming interfaces (APIs), providing predefined functions to make their retrieval easier.

I’m tired of looking at websites. I want to use them.

The world isn’t getting any simpler. All of us have woken up this century to find ourselves tied down to computers. Our work and life depend upon them. And now, even in light of our privacy being continuously violated, we’re (for the most part), unable to disconnect. So in this uncomfortable space, our relationship with technology, computers and ultimately the Internet, is pushing us to find better ways to ease our new found digital lives. And yet, each time we visit a website, even a new website, we bang our heads against the sheer idiocy of the experience.

“Why are they making me do this? Why so many clicks? Can’t they just give me what I need?”

simple tram

Who they are in this equation is lost on me, and I develop and build websites. For a long time, I’ve advocated principles of content first and a general de-emphasis on design. We don’t need more amazingly stunning websites. We need far simpler web experiences, that allow us to do what we want. And rather than turn everything over to some company to trust our private data, our address books, our personal surfing experience, we need to just learn how to build websites that allow a clean path of choice.

Don’t do everything for me. I can figure out what I need, if you can make clear what it is you offer.

The greatest insight I have had recently as a developer is that we don’t need to do much. We just need to get out of the way. We don’t need to provide deep hierarchical menus to allow someone to traverse our site, we need to produce our content well and allow the people who want to find our information half the chance.

So, although I can appreciate beautiful things, I cannot appreciate them when they come at a cost.

Corresponding in the social net

mind your headI must be getting old. I’ve been corresponding with a few people through Facebook’s messages and I’m finding the experience a little disheartening. Perhaps it is because I get these messages, which can be quite long, on my mobile while I am walking around. I want to read them (but can’t because I’ll run into someone). I admit I sometimes park myself next to the wall along the sidewalk for everyone’s safety. But even as I read, what I imagine would have been great material for letters sent by boat across the ocean,  I am simply unable or unwilling to respond from my little corner of the urban flow. So I make a mental note to respond later when I am home, sitting down, and able to give the proper attention that I believe my friends deserve.

This has led to think a little bit about long form communication. Years ago, while working on a job, I use to write, what I am certain my coworkers believed were small treatises regarding certain aspects of the project. Often I would get quick response like: “can you please use bullets and make your emails smaller.” How dare they? Do they think I just like to write, to see my words on the page (don’t answer that – it’s meant to be rhetorical). Sometimes, I think I would have been better off printing them out and nailing them to their door. Most of the time, I would love to send in reply a link to my latest favourite comment on lists, which includes a great insight on bullet points. But I don’t, because maybe I understand the environment of communication has changed and is continuing to change. So as I get older, I guess I just lament that a little.

But let’s think about what communication is. At its root, it is about being able to express something and have the other person understand. In deep intimate relationships, trusting and giving relationships, a little can be said, maybe even something can be expressed without words and there is no misunderstanding. I recently came across a wonderful term called exformation, which refers to the discarded information that still leaves a message in a given context understood. This reminds me of Claude E. Shannon’s work on Information Theory and signal noise. We have such different ways to communicate as human beings, some of it using technology, some of it using less technical means, like a hand-shake or a kiss. And whatever we do, we are communicating beings. The interesting part is thinking about when we use which form of communication. And that, regardless of what is trendy or what is used by our friends, remains a decision based in what works. Are we able to express ourselves in a given medium? That is the only way to choose the right way to communicate.

As a writer, I sometimes think I would rather write a short story for a friend, than explain how I feel. I also love links to odd ideas and I share them often on Twitter and Facebook. I send emails to some friends and family, because I know they will read those. I leave post-its for others. Sometimes, I chose to be very calm in how I talk. Other times, I need to stand and I find my hand takes on a life of its own. Communicating is so wonderfully diverse. And believe me when I say, I think about it a lot – I love communicating. I would give up all sorts of things in my life to find more and more people to communicate with, because in the end, all I’ve got is what I think and how I choose to express it. When I write you, when I text you, I am telling you a story. It may be the most intimate thing in my life and I may choose to post it somewhere publicly, but I do so, with only one idea in mind, that I understand who I am talking to and how my words or lack of words (my exformation) will make sure the communication does what I want it to. And the back and forth of a correspondence, whether it is real-time or over many many months, is an expression of my understanding of what you have communicated to me (please don’t write long emails, send me bullet points, because that’s all I will read). So it is the appropriate response to the given correspondant. That is all that matters.

As I lament the death of long form writing, I saw a great article about the winner of social media. Can you guess? Email, of course. Certainly you can argue against this, but I’ve always considered email part of the social network, and perhaps the best for a given context. Before that term arrived and we were all just talking about social theory online, email was clearly the most important. It was the most used. But somehow, as Facebook and others arrived and began to produce outstanding numbers about their users, we all just kind of forgot that we use email all the time. And yet, even though this is true and our jobs depend on it, in my day to day networking, when I met someone, I’m trying to get their Twitter handle or become friends with them on Facebook or LinkedIn. In other words, I’m looking to follow them. I don’t want to communicate (yet), but I do want know more about them. I’m looking for them to share with me, and to do so in a way in which they do not yet need to care who I am (which is why I prefer Twitter). But at some point, the communication flips around, and I find I’m trying to respond to their tweets, whether they follow me or not; and I’m trying to get their attention, whether they notice me or not. Occasionally I strike up a little conversation. Usually public, but eventually it can progress to the private kind. It’s after corresponding over a longer period of time, that I find I miss email. And I miss the willingness to stop, take time (not on the street, as I said) to respond intelligently and with a care, crafted for this person, so that they really do understand me. This is my goal. To make my communication good and meaningful. So when when I lament the long form, it’s probably nothing more than an existential lack in myself, that the particular person I’m communicating with, simply isn’t the person I have made that deeper connection to, so that I would take out a fountain pen, and under candle light, write for hours and hours and then carefully place it in an envelope, seal it in wax and then set it on its way across land and sea to arrive months later in the hands of someone who truly understands me.

Why is Open Data difficult for Governments?

I’ve been asked this question a lot lately. And recently, a journalist asked me this during an interview for a story on his problem getting data on noise pollution from the Hong Kong Government. Christopher DeWolfe quoted me in the article, Hong Kong’s Silence on Noise Pollution (which was also printed in the SCMP and is online behind their paywall):

Without open-source data, there is no way to see if a similar situation exists in Hong Kong. “There’s potential for the government to be embarrassed – it’s like having an auditor, and that’s why they drag their feet on releasing information,” said Darcy Wade Christ, a researcher for the University of Hong Kong’s Journalism and Media Studies Centre’s Open Government Data initiative, which is pushing for more transparency in the government.

This question comes to me a lot because of my work at the JMSC on the OpenGov project, but also because I am very involved with the Open Data Hong Kong group. I am more than happy to discuss this issue because I think it is at the heart of how we begin to form a new relationship with our governments all over the world.

As I said in my quotation, opening data has the potential to cause a lot of trouble for the government. This explains, in part, why they are reluctant to open up certain datasets. There are obviously other reasons, including costs required to convert or manage ongoing public data. But by far, the most important is that Governments do not know what opening certain data will do. All governments will be conservative when it comes to things they do not understand. Technology (especially the Internet) has always presented this problem, because it is new, always changing and its effects are unknown. And since there is no government without politics, all decisions pay some heed to the political ramifications of doing anything.

When I was talking to Christopher DeWolf, I wanted to underscore the idea that Open Data does pose the potential for embarrassment because it enables citizens (and politically motivated groups) to audit and present displeasing results. But it also has the potential to improve how the government functions for the very same reason, in that it provides additional auditing. It can also improve the relationship between citizens and their government by engendering more trust. For these two reasons alone, it is in all of our interests to continue to find more ways to open up public data. Some people like to stress that any data the government has generated should be public because it was funded by taxpayers. I don’t usually use this argument, because it can end up dead-ending around the issue that not everything should be public – that the government is given the responsibility to protect all citizens, and that some data may negatively affect a certain group of people. I believe the Hong Kong Government takes this issue quite seriously. They are always conservative when it comes to embarrassment. This may have been why I chose to use that word (wisely or not). The HKSAR has a history of defamation lawsuits, but it is also part of a culture that believes strongly in not publicly embarrassing others. There is a time and place for showing your power, I have come to understand.

So when we talk about the reasons why any government should open more datasets, let’s keep in mind that this is a complex issue and that all arguments have many sides to them. The recent global appeal to Government Transparency and Open Data is good, but it will take time. We have to be certain of discussing both the good and the bad. We have to appreciate that governments will drag their feet and given their very jobs (not as public servants, but as politicians), we have to at least acknowledge why they may not embrace the ideals we believe in. But we can still make a little noise here and there. We can still improve how we speak and use powerful words like audit (are you afraid of an audit?). And we can also continue to explain why we want certain data. The Open Data Hong Kong group is often discussing Open Data as a general mandate. In my work at the OpenGov project, we are usually going after lesser understood and known datasets (land use and corporation information) without fully understanding the goal of what we want. We can extol the importance of having more information. We (especially journalists) can begin to articulate a premise we might have – that having certain data will help to prove or disprove our idea. But no matter what any of us do, this process will take a long time, because politically, there is very little incentive to open oneself up to critique.

We could all afford a little auditing in our lives, but who invites an auditor into their house?