Tonight the Apps for Amsterdam awards ceremony takes place and stage one of the Dutch open data trajectory will be completed.
Last year at the end of summer I helped Thijs Kleinpaste and Stefan de Bruijn co-author a proposal to sponsor open data within the municipality of Amsterdam. This proposal was accepted near unanimously by the commission in November (full write-up) and it started a roller coaster ride for open data in Amsterdam that is now starting to have far wider effects throughout the Netherlands.
Hack de Overheid (Hack the Government), the soon-to-be foundation I’m in the board of, partnered with the City of Amsterdam and Waag Society to realize the competition and a series of events. This series culminated for us in Hack de Overheid #3 an inspiring day and hackathon for over a hundred developers who built civic apps.
But as I said this completes just the first stage of what is bound to be a long and tortuous road. As we speak there are local initiatives being formed to open up data in at least Enschede, Rotterdam, Utrecht, Eindhoven and the Hague. It will be interesting to see what comes out of that and if some of the smaller cities may in fact outpace us here in the capital.
But we need to do more. Recent questions about privacy violations in data releases make it more than a little obvious that there is a massive issue in data literacy. I wholeheartedly agree with Adam Greenfield if he says that data and its affordances need to be a core subject starting from school onwards. We need to explore materials, interventions and processes that allow us to teach data literacy and that allow others to teach it for us if we ever want to spread this knowledge at scale.
Literacy is required not only in school children but also in decision makers in business and government right now if we want to keep the momentum we have right now. Future developments run the risk of being hamstrung by backlashes against the malignant consequences of data or open data being unused because the ecosystem is not in tune. There are still lots of issues to be resolved around ownership, privacy, responsibility, licensing and business models.
From a commercial point of view, the sustainability of many of the applications in the contest is doubtful. Creating proof of concept apps for the data is a more than a good start, but it is by no means enough. The real need is for open but comprehensive systems where open data is a given. That data needs to be technically excellent and fully engrained in the fabric of our information society so that everybody can use it to enrich their app/site/discourse. Data owners and producers need to participate and be accountable for their data to accept feedback from the public both in the specific and in the general case. Such a system cannot be built or be static, but needs to be grown and evolve continuously. The only thing we can do is plant, nurture and weed.
So tonight will be fun, but let that not distract us from the massive amount of work still ahead. We are ready for it. Will you join us?
The agenda is filling up again just before the summer break. Alper will speak at:
May 24th – Technical review of city dashboard concepts at HvA
A brief bit of teaching with design and technical critique of city visualization dashboards developed by students.
May 25th – Apps for Amsterdam Awards Night
Judging and attending the awards for the Amsterdam open data application contest.
May 27th – What Design Can Do
Presenting an engaged data-centric approach for designers’ benefit (blurb).
Here are the slides for a talk I gave at /dev/haaglast Friday ambitiously titled “Fixing Reality with Data Visualization” which was well received. I promised to write it up here, so here it is.
Starting off with some introductions. We are Monster Swell, this equation is the central challenge of our practice.
To start with the title inspiration for this talk. I recently finished this book by Jane McGonigal.
“Reality is Broken” by Jane McGonigal recently came out and it’s not really true, but it’s quite opportune. Reality isn’t broken, but there is —as always— lots that can be improved. Slapping a gamification label on that is a false exit because it implies that such improvement can be done easily by the magic of games.
The core idea of the book is that:
1. Reality can be fixed by game mechanics (voluntary participation, epic stories, social collaboration, fitting rewards), and
2. That reality should be fixed by game mechanics.
Both of these points: the possibility and the desirability of such are the subject of fierce debate both within game design circles and without.
We are now seeing a superficial trend of gamification, badge-ification and pointification where everybody is rushing forward to add as many ‘game-like’ features to their application/concept to look tuned into the fun paradigm.
Fortunately this does not work. Checking in for points and badges is fun at first, but is hardly a sustainable engagement vector. Foursquare mostly did a bait and switch with their game until they got enough critical mass to be useful along other vectors.
Things that are difficult remain difficult even if they are gamified. ‘An obstacle remains an obstacle even with a cherry on top.’
Ian Bogost terms this exploitationware. Our own discussions concluded with that if you are not the one playing, you are being played.
In our practice we look for deeper ways to engage people and affect them. There are hardly any one-to-one mappings to be found and the effects that are most worthwhile are the higher order ones. As Kars Alfrink says:
“We don’t tell them to coordinate, we create a situation within which the way to win is to coordinate.”
Corollary: A game about violence does not immediately make people violent.
But another way of looking at it might render it as a map. The metaphor of men and liberties and territory to occupy already points towards that comparison.
Looking at it in another way it could also be a Cartesian grid with binary data values plotted onto it. A data visualization of a phenomenon we don’t know (yet).
Coming back to the map parallel, this picture of center pivot irrigation systems (by NASA) in Garden City, Kansas looks awfully similar to the goban and this is just an aerial photograph with some processing applied to it.
So to come to this point:
‘Any sufficiently abstract game is indistinguishable from a data visualization.’
The difference just is that a game is a visualization of a game model and its rules. The whole point of playing a game is learning those rules and uncovering the model of the game is essence ‘breaking’ a game. After this point it usually ceases to be fun.
And its complementary point:
‘Any sufficiently interactive data visualization is indistinguishable from a game.’
And indeed the best ones are highly interactive and offer various controls, abstraction levels and displays of data deep enough to engage users/players for a long time. It is also the reason that in our practice we don’t occupy ourselves much with visualizations in print media.
To continue the point about games: many games are either quite concrete or very abstract simulations. This is most obvious with sim games such as Sim City pictured below.
Simulations are subjective projections of reality both because of the choices that the designer of the simulator has embedded in their choices for the projection and because of the interpretation of the player of the simulation and how their ingrained notions allow them to interpret the simulation.
Ian Bogost (picture) in his book Unit Operations coins a state of being called ‘Simulation Fever’.
Bogost says that all games in some way are simulations, and that any simulation is subjective. The response people have to this subjectivity is one of either resignation (uncritically subjecting oneself to the rules of the simulation, taking it at face value) or of denial (rejecting simulations wholesale since their subjectivity makes them useless). Taken together, Bogost calls these reactions simulation fever. A discomfort created by the friction between our idea of how reality functions and how it is presented by a game system. The way to shake this fever, says Bogost, is to work through it, that is to say, to play in a critical way and to become aware of what it includes and excludes.
I think we could use the correspondence between games and visualizations to coin a corresponding term called Visualization Fever.
Those are my most important points, that good and interesting games and good and interesting data visualizations share many of the same characteristics. We can use data and its correspondence with reality (or lack thereof) to create a similar fever.
(This graphic is somewhat rudimentary but it was made within Keynote in five minutes and I hope it gets the point across.)
The visualization process shares a lot of similarities with the open data process that we are involved in. It is a perpetual conversation and the visual part is only one place where it can be improved. Data collection, discussion on results and errors, sharing of data and the resulting products, controllability of the outputs and being able to remix and reuse them and incorporating this process as feedback back into atoms are all areas that need active participation.
There is nothing easy about this. It is a ton of hard work and long tedious conversations. Fortunately most of it is worth it.
Some examples of visualization fever in action.
Verbeter de Buurt is the Dutch version of See Click Fix and it works really admirably. It creates a subjective map of an area with the issues that a group of people have signalled in their neighborhood. Nothing really is said about who these people are and if these issues are indeed the ones that are the most pressing (we all know the annoying neighbour who complains about dog poo to whomever will hear it). By making issues visible, this map imposes its view of the city onto the councils and exerts change.
Planning systems at an urban scale is a very difficult process. These planning stages are being opened up to the general public using consultation and other means but it remains to be seen if and how citizens can comprehend the complex issues that underlie city planning.
One step to help both experts and laypeople to better come to grips with the city that they are inhabiting is to create macroscopes that in one view show the entire scale and all the things that are in a system in such a way that we can make (some) sense of it. These Flowprints by Anil Bawa-Cavia are a great example of doing such for public transportation.
And done right these visualizations can reveal the systems of the world or in this case the order flow of trains in the Netherlands. Everybody knows how crowded Dutch rail is, which trains go where along which routes, but actually seeing it happening in front of your eyes in a real-time visualization gives you an insight and a tangible grip on the system that you did not have before.
So what do we fix?
We use visualizations and their compressed interactive views to expose system design choices and errors. They can also be used to give depth to a specific point, something which journalists are increasingly finding necessary. People consuming data heavy news want to be able to poke that data themselves.
A lot of visualizations I have seen thusfar serve not much more than to reinforce pre-existing judgements almost as if the person creating the visualization sought to build that which they wanted to see. Visualizations will need to be better, more flexible and draw upon more data if we want to break out of these throughs of shallow insight.
The brief as stated by the nice people at Bloom as well is that having a visualization serve solely as a visual output is too limited a use of the interactions created. You should be able to use the same interactions in the visualization to also influence the underlying model either directly or indirectly. That is to say the model and the representation should be bidirectionally influencing.
Planetary, the latest app by Bloom is a great example of that. It shows you a beautifully crafted astromusical view, but it also allows you to play your music library from within that very same visualization.
We need to bring visualization and deep data literacy to the web and infuse any relevant site and system (that is to say all) with them. Many people asking for data visualization think that they are some magical fairy dust that will make a site awesome by its very touch. This is of course not true.
Data and interactive visuals can generate value and insight for any site that employs them properly.
In the presentation Data Visualization for Web Designers by Tom Carden he remarks that web developers already know how to do all this. These are exactly the tools we have been employing over the last years to create interactive experiences (and we plan to use them more and more).
Internet Explorer is still the cripple old man of the web, but given understanding clients (and users) and some compatibility layers, you may be able to get away with using a lot of this stuff as long as the result is awesome enough.
The other trend is the idea that there need to be bridges built between web people and GIS people. Preferably how to create GIS-like experiences using the affordances that the web necessitates. A trend we were thinking about neatly summarized (blog) at a #NoGIS meetup by Mike Migurski.
GIS people have tremendous tools and knowledge but they are not accustomed to work in a very web way: quick, usable, beautiful. Web people can build nice sites pretty quickly, but they tend to fall flat when they need to work with geographical tools that are more complex than the Google Maps API.
If we can combine these two powers, the gains will be immense.
We can create subjective views to exert power upon reality and try to fix things for the better. The subjectivity is not a problem, as often the values embedded in the views are the very point. Subjectivity creates debate and debate moves things forward.
The tools we have to create these views are getting ever more powerful, but there is also a lot of work to be done.
As a wise man said: “The best way to complain is to make things.” (picture)
For the Amsterdam UIT Bureau and I Amsterdam we created this Foursquare map designed to display nightlife activity around the Leidseplein (entertainment) area with recent checkins, specials and current mayor and photographs of a selected group of venues. We strongly believe in creating autonomous displays that take cues from the environment —in this case using Foursquare— and deliver clear actions to the audience as well as a sense that the area they are in is alive and all they have to do is go out and connect to it.
Technically we used Foursquare’s OAuth2 API which is outstanding. To be able to share one token across all requests we employ a file based PHP cache that relays the necessary requests for us. Main technology was created in collaboration with Panman Productions.
We ran a major update to the previous concept we did for the Dutch Labour Party using their canvassing results for the previous elections. The previous version crammed all the interaction into a tabbed balloon on a Google Map. This update turns that inside out and creates a full blown site called: “PvdA – Altijd in de buurt”.
The site shows canvas results tallied per city to show the biggest positive and negative issues according to constituants and their perception of politics.
The potential for a data driven approach to politics is tremendous. A site like this in effect gauges the sentiment in any given locality and in an ideal scenario it would also give people and politicians ways to collaborate to improve the situation. Any improvement realized can then be recorded and used to rally voters at subsequent elections.
Alper is speaking at /dev/haag this Friday giving a presentation with the title: “Fixing reality with data visualizations” tying together a bunch of strands.
It promises to be a fun event and you can still register at meetup.
An exploratory project for the Dutch weekly de Groene Amsterdammer (yes: the Green Amsterdamer) concerning a survey posed to a large number of social scientists asking their assessment of the most important problems troubling the Netherlands currently.
As an end result 75 submissions were returned with answers in essay form detailing the biggest problem of the Netherlands, the most overblown issues and the most unnoticed issues according to the scientists. This made for a very large amount of textual content which would have been difficult to quickly get into.
We chose to see how quickly we could hook up Protovis to visualize the key issues according to each scientist. All of the essay style answers were clustered to a set of themes (by the people preparing the story) and this was input to Protovis’s bubble chart to give a tag cloud like representation of the issues. See the interactive chart on Groene.nl or the screenshot below:
The quick visual summary and the filters help drill down to a specific issue in a specific problem category quickly. Clicking a bubble displays links to the full text contribution of the relevant scientists.
This was mostly a process exploration to see how a default library such as Protovis could be employed in a journalistic context and to see where the bottlenecks fall. We found that Protovis’s explanatory power really shines if you have a good dataset. However it took some time to get the data machine-ready. The result was produced efficiently and adds a much needed visual summary to the slew of textual content. Most time was spent on wrangling the dataset and finalizing the interaction details of the chart.
Our Alper has joined the board of Hack de Overheid a Dutch think tank that creates software and events to advance thinking about transparent government and open data in the Netherlands. Actually more of a do tank in that respect.
Each year Hack de Overheid holds a developer day where civically inclined programmers gather to exchange knowledge and create new open data projects either with government’s consent or without.
This year the devcamp is part of a broader program along with an application contest for local data and local applications in the city of Amsterdam called Apps for Amsterdam. There is a lot of momentum and it looks like open data is finally being taken seriously.
Until the event, updates here may be a bit sparse, but do register for the March 12th event if you have any interest in data and let’s create something great together.
The past weeks Alper has been giving lectures at the Willem de Kooning design academy on the subject of data visualization. The students should be busy creating their projects these coming weeks and we eagerly anticipate their results.
We will be represented at the Cognitive Cities conference in Berlin this weekend to talk about city data visualization. And next week we’ll be at the Infographics conference trying to talk some sense into those that think print is the end all of data.
The documentary deals with the flash crash of May 6th, 2010 when the black box trading operations on Wall Street went haywire and dropped the index 900 points to recover just minutes later. I’d already read about the possibility of such events from Kevin Slavin’s January 2010 Social Computing Summit presentation which has been noted down and blogged about by Michal Migurski. Both are recommended reading.
We have seen various troubles with the stock exchanges in the past year and this event especially seems one worth investigating because it exemplifies the complexity in todays exchanges and the total lack of control humans have over the process.
Another reference which I thought was important is the concept of the macrospcope by John Thackara which I first heard used and expanded upon in Matt Webb’s Reboot keynote. The related reading on the BERG blog is interesting but primarily the definition by Thackara: “A macroscope is something that helps us see what the aggregation of many small actions looks like when added together.”
So let’s see.
Documentary
For an international audience the concept of Tegenlicht may need some explanation. Tegenlicht is a documentary that examines world events by interviewing experts interspersing the interviews with visuals and a voice-over to create a dramatic storyline. The app contains the entire show in high quality which is in part why it is so heavy.
The shelf life of Tegenlicht documentaries is quite high. For another concept we recently rewatched their 5 year old documentary ‘De dag dat de dollar valt’ (Eng. The Day the Dollar Fell) because it was still relevant and interesting. The 45 minute length with drawn out shots can be a bit taxing for today’s YouTube attention spans, but byte-sized information is not their game. TED is much better at that. They concern themselves with the documentary as a dramatic art form that needs to engross its audience.
Given that concept —highly traditional television, cinema almost— it is interesting how you would interject/overlay/add interactive features into the narrative whole. This was touched upon briefly in the presentation, but that is not what this app concerns itself with. You can view the documentary and jump back and forth through the various segments while additional content is presented for your perusal.
It is clear though that traditional broadcasters are still very much struggling not only with the internet but also with the spectrum of television, cell phone, laptop, iPad and the locus of interactivity (if there is any interactivity). Tweede scherm is one such recently award-winning concept that displays supplementary information to add context to the main experience on the large screen.
Infographics & Visualization
The app is decked out with a nice cadre of infographics and visualizations and those are indeed its most important selling point. There is a list of them on the home screen. Several time series, an animated display, a multi-layered map overlay and a world map with live stock updates:
The visualization that gets the most emphasis and also is used often in the documentary is a time series display of the stock price around the crash:
It starts out nice and flat with a display of trading velocity (not quantity) and pricing information along with the time. There is a global display with scrubber that you can use to navigate over the entire run of the data and the crash is nicely colour coded. The vertical scale is a bit confusing as the one above does stop at zero (and then goes on for a bit more) but the other ones don’t. So confusing in fact that in the documentary one woman remarks ‘Apple is going to zero.’ which it is in fact not.
And then it goes South:
You can see the drop in prices and the variability. It becomes even more clear if you also enable the Bid & Ask information which is available as an overlay and shows you the differences between the prices asked and bid for the stock at that moment:
So that is a one-dimensional time series (with two extra dimensions available on request) with a beautiful presentation and animation. Another interesting piece of information is the potential locations for data centers around New York and what factors they need to take into account to carry as little risk as possible. You can see the map and the various factors involved and slice it yourself:
Another visualization is a map of the world (in catalogtree’s signature geographical bubble display) with semi-real time updates of the world’s exchanges and how they are doing right now:
The visualizations add a lot of panache to the documentary and are aesthetically very pleasing to behold which gives them a high show and tell value.
From an information design point of view however they are underwhelming. The information density is low, it is difficult to compare several datasets and the visualizations do not offer different types of information at different zoom levels. Also: the interaction is nearly trivial.
The issue of game design and game-like experiences was touched upon during the launch event to conclude that none of the makers had a lot of expertise (or even any affinity) with games. That is unfortunate because game design with its experience in dealing with highly interactive experiences of high density information spaces can add a lot to a data visualization.
Result
The whole issue of interactive television and how to combine a long dramatic form with visualizations (and what kind) seems to be a difficult one to solve and not the one being tackled here. As Erwin mentions in his review in Bright, added value is a highly pressing issue when it comes to traditional media trying to produce content for the iPad. That is exactly what this is: a nice packaging of a traditional television program with interactive features in a combination that will most probably remain interesting and relevant in the future.
The documentary is very attractively presented on the iPad. The extra content especially is more prominent than normal when it would have been put on a back page somewhere on the website. The video playback is also one of the first cases in Dutch broadcasting where the presentation is native to the device.
Another benefit is that this experiment can probably live on as a packaging format for other documentaries. The richness of the experience combined with the quality of the video, the pairing of additional content and the clear payment model make a lot of sense for something which already has high production costs. With magazines and newspapers you are adding a lot of extra weight to something that has a low margin and is ephemeral (daily, weekly). A well produced documentary such as Tegenlicht can live on for a long time and this seems to be a more suitable incarnation for that than most.
