Open Data

We did most of our data work within Hack de Overheid / Open State where we believe our efforts produced the largest possible impact. During the past two years as part of my program direction, curation, advice and advocacy along with the help of my extremely capable fellows has enabled a series of events and a network to evolve in the Netherlands, the likes of which the world has not yet seen.

A summary would already be too long to post here (just skim the weeknotes to get a glimpse), but to name some highlights: several app competitions, numerous hackathons (the next of which is in Rotterdam) with apps developed in the hundreds and a range of theme networks around certain subjects. We have turned the Netherlands from an open data laggard into a country with one of the most vibrant ecosystems in Europe. Doing this together taught me one of the most important lessons in doing business: people working together can achieve more than they can alone.

To add to that, the foundation we built is going to be even stronger in 2013 which will alleviate my personal involvement in the operational day-to-day and enable me to focus on my own data business again.

Services

On the services part, we have diversified from a pure visualization offering to a set of services that emphasize product development in a big data context as well as taking on teaching to develop data capabilities within organizations.

We focus on Analysis, Creation and Teaching because we have found that a pure focus on visualization is not pertinent to solve significant problems for most organizations. The issues they deal with are far more complex and capable visualizations are only one small part of the solution.

Added to that we found that data literacy at all levels in society from decision makers to journalists and designers to the general public is severely lacking. This creates a challenging environment for those busy in the field of data. Explaining the work they do to the general public is not only in a practitioner’s self-interest, it is also a moral responsibility.

Location

Also we moved our base of operations to the European startup capital Berlin while still maintaining offices in the Netherlands.

Or you can view it on Blip.tv.

We’ve been pleased with the feedback on this blend of algorithms, new aesthetic and object-oriented ontology. More to follow.

I pre-rolled a screencapture of Aaron Straup Cope’s Wanderdrone to add ominous foreboding to the mix of design/advertising enthusiasm permeating the room. Crosslab called the night a night about Dynamic Design, which I didn’t really get, but I retook an old talk about algorithmic design but now heavily updated to incorporate current thinking about algorithms, the new aesthetic and object oriented ontology.

Our offices in Berlin-Kreuzberg

Given the fact that we as Monster Swell are a company that does a lot of stuff with maps we are affected by the fact that mapping is being turned on its head. And it’s not because there aren’t enough interesting maps, there are now more than ever before. Just to show a couple.

Eric Fischer’s Twitter Traffic Maps of various cities, here New York:

Google Hotel Finder with an isochrone projection in London:

Timemaps, a map of the Netherlands distorted by the amount of travel time required during various times of day:

But right now there is a projection inversion going on where a lot of the time we no longer project the real world onto flat surfaces and call that maps, but where we overlay maps themselves back onto reality. And what we call maps does not need to have any relation with physical reality anymore, we can map anything onto anything using any (non-)geometric form we choose.

This is mainly a consequence of us putting the internet into maps. But if you think again, the internet is not the only place where we put maps. We put the internet into pretty much everything by now.

So maps are creeping back into the real world and we get odd clashes when we try to overlay a map back onto the territory or when we try to perfectly capture a capricious world, as you can see in these Google Maps and Street View examples: 1, 2, 3, 4. I don’t know how long they will be online over at the New Aesthetic Tumblr since that has been closed by James Bridle right now.

We got QR codes to enable the machine readable world. These hardly have any real world use (just go over to the WTF QR Codes Tumblr) but they function more as cultural icons, precursors of a strange and inscrutable future.

And even more interestingly they are being used for instance by the Chinese to calibrate spy satellites. So these are maps on the earth that are being used to create better maps of the earth.

The New Aesthetic is when this kind of projection inversion happens more widely, not just in the realm of maps, but in all of the places in the world that the internet touches. By now that is nearly everything. The examples that were being collected over at the New Aesthetic Tumblr showed how the arts were picking up on this trend.

All of these things have been created by algorithms which are not as mysterious as many people make them out to be. Algorithms are how computers work and increasingly how the world works. They codify behaviour and quoting Robert Fabricant, as designers ‘behaviour is our medium’. Being a designer should entail more than a passing knowledge of and proficiency with algorithms. We are moving into a world where creative work is becoming procedural. The most important media are prescriptive and set rules for the world more than they are descriptive and depict the world.

The real problem with algorithms is that they often involve us but they are completely alien to us (in the Bogostian sense). They are operationally closed. Operational Closure means that things may work in ways that are not at all obvious to us, neither at first nor after we poke into them because any kind of sense we make of it is either partial or does not translate into our frame of reference. Algorithms get inputs and perform outputs but the way they operate on these has nothing to do with how we as humans think about the world. Think is not even the right word, but we try to relate to them from our human cognition. The machines see us, but they do not ‘see’ us in any way we would recognize as seeing and we have no idea what it is that they see.

The Machine Pareidolia experiment over at Urban Honking is a good example.

The ability to see faces in things is a basic aspect of our visual pattern recognition. When we teach that same skill to computers we get unexpected consequences. It is the same with the flash crash on the stock market that happens in the blink of an eye without anybody really knowing what caused it. The rationales of the algorithms are opaque to us and their emergent behaviour unpredictable.

As Kevin Slavin mentioned in an interview: the more autonomous the algorithms are and the more effects they have on our daily lives, the more we may be accommodating them without realizing it.

There was this story recently that scientists have created a robot fish that is so good at mimicking the behaviour of regular fish that it can become their leader. This is what worries me. Who says we are not all following robot fish most of the time?

So that is what I think is the biggest challenge right now for designers. Try to create systems that harness the open and generative power of the internet while on the other hand remaining human and aligned with human interest. One way would be to make the internals of algorithms transparent so people can enter into an informed relationship with them.

Unfortunately there are no magic bullets for this whatever your local design visionary has been telling you. There never have been. Everything is made up of withdrawn objects that are mediated towards one another with unexpected consequences. To quote Graham Harman from the Prince of Networks:

“the engineer must negotiate with the mountain at every stage of the project, testing to see where the rock resists and where it yields, and is quite often surprised by the behaviour of the rock.”

There are no ideas that will solve all problems, there are no products that will do everything. There is only the work through which we may gain more understanding and make better things. So with that, I hope we all can do good work.

It is possible for residents of the EU to request from Twitter all of the data it has stored about them in accordance with European data protection laws (just follow the steps). Some Twitter users have requested their data and filled in the necessary paperwork. After a while they have gotten all of their records including a file with all of their tweets in it.

I had seen Martin Weber’s post about this before but when I saw Anne Helmond post about her experiences as well, I was prompted to carry out the idea I’d had before: to import an entire Twitter archive into Thinkup to complement the partial archive it contains of my longtime Twitter use (since September 2006).

I use Thinkup myself enthusiastically to supplement existing archival, statistics and API functionality around the web and more importantly to have it under my own control. These services serve as my social memory and it is nice to have a copy of them that can’t disappear because of some M&A mishap. It has proven useful more than once to be able to search through either all of my tweets or all of my @replies. But as noted, Thinkup can only go back 3200 tweets from when first you install it because of Twitter API limits. For people like me (35k tweets) or Anne (50k tweets), that’s just not enough.

I installed a new Thinkup on a test domain and asked for (sample) files from Anne and Martin and went at it. Command-line being the easiest, I took the upgrade.php script, ripped out most of its innards and spent an afternoon scouring the Thinkup source code to see how it does a Twitter crawl itself and mirrored the functionality. PHP is not my language of choice (by a long shot), but I have dabbled in it occasionally and with a bit of a refresher it is pretty easy to get going.

I finally managed to insert everything into the right table using the Thinkup DAO but it still wasn’t showing anything. Gina Trapani —Thinkup’s creator— told me which tables I had to supplement for the website to show something and after that it worked! A fully searchable archive of all your tweets in Thinkup.

The code is a gist on Github right now and not usable (!) without programming knowledge. It is hackish and needs to be cleaned up, but it works ((It should scan available instances and only import tweets if they match an instance in your install among many many other things.)). Ideally this would eventually become a plugin for Thinkup but that is still a bit off.

What’s the point of all this? There are a couple:

First it shows that data protection laws such as the ones we have in Europe do have an effect (see also for instance: Europe v. Facebook). Even on the internet laws have teeth and practical applications. Data protection laws can be useful if they are drafted on general principles and applied judiciously.

But the result you get: a massive text file in your inbox is not the most usable way to use or explore half a decade’s worth of social media history. That’s where Thinkup comes in. It’s brilliant functionality serves as a way to make this data live again and magnifies for each person the effect of their data request.

Secondly, for any active user of Thinkup, supplementing their archive with a full history is a definitive WANT feature. Twitter has been very lax in providing access to more than the last 3200 tweets. If a lot of users used their analog API to demand their tweets, Twitter may be forced to create a general solution sooner.

Lastly, Thinkup has applied for funds with the Knight Foundation to turn itself into a federated social network piggy-backed on top of the existing ones. Thinkup would draw in all of the data that is already out there into its private store and then build functionality on top of that (sort of an inverse Privatesquare). Having access to all of your data would be a first step for any plan that involves data ownership and federation.

I presented this hack yesterday at the Berlin Hack and Tell. Your ideas and comments and help are very welcome.

There are two upcoming events at which I will be speaking that bear mentioning here.

There will be an evening in Pakhuis de Zwijger to celebrate the Nederland van Boven television series that the VPRO produced in the Netherlands ((Borrowing conceptually from Britain from Above among others.)). I will be joining the esteemed panel there as a board member of Hack de Overheid to talk about issues of democracy, participation and truth in cartography.

The week after that there’s “Social Cities of Tomorrow”. I will be speaking in a brief timeslot about Apps for Amsterdam about how you can create a data commons for your government of organization and where to take it from there.

Never having used D3 before this was an ideal excuse to learn it and a near ideal dataset to employ. The result is as follows (click through for the interactive version):

This was featured on Sargasso the next day.

The graphic is not directly clear, but the data is deep and interesting enough to afford some exploration and it yields insight into the behaviours of various political parties during the reign of this cabinet. And what seems to matter a lot to people: it looks quite pretty.

With regard to D3, I think I will use it more often. It works quite similar to Protovis with which we have done some stuff before, but it feels much more current. Protovis itself is discontinued in favor of D3 according to a notice on the site and D3 seems a very worth successor.

The NS API and its limitations

The NS, the dutch railway system provider, provides an API which allows a developer to build upon it. While it is a nice effort and opens up a lot of possible applications, we found out that for the TIMEMAPS project it was not an ideal API to work with.

But, our requirements were pretty ambitous to begin with, and, from a practical point of view, not what an API designer would call “typical”. In TIMEMAPS, given any point T in time, we need to know, for every train station, how long it takes to travel at moment T to any other train station in the netherlands. Even for a small country like the Netherlands, this becomes a pretty big matrix of travel possibilities, given that there are 379 train stations in the country.

Ideally, for every element in this matrix an API call has to be done to get the actual planning.

Given that the NS API only allows an app to do up to 50.000 requests per day and we did not want to hammer the already stressed API servers too much, we needed to come up with a solution, while not sacrificing the real time aspect too much.

An open source travel planner..?

Another API call that the NS offers are the “Actuele vertrektijden”: given a station, return the 10 first trains that depart from it. It returns also the train numbers: a “unique” number which is assigned to a train on a single trajectory for the day (it might be re-used though in time). By linking the departure times from different stations through this train number, it should possible to see when a train that departs from A passes through B, if it is on the same trajectory.

However, some drawbacks popped up while implementing this approach.

• For long trajectories (>1h) this approach did not work since the arrival station did not yet list the departure of the train you departed on since it was too far in the future
• There was no API call for arrival times for trains on stations: this made it impossible to take the stopover-time into account and it was not possible to use this planning mechanism for destinations on the very end of the trajectory (e.g., no departure listed for the arriving train)
• Doing a “naive” planning this way takes a considerable amount of database processing power as each stopover adds 2 self-joins to the database query, thus increasing exponentially in complexity.

Scraping of the departure times gives a increasingly complete graph of the railway system, and this graph, combined with the geographical location of stations might be used in a search algorithm to make an offline planner. For me however this aproach was too far of a longshot for the already pretty complex project so I decided to put this approach in the fridge for now.

However, this effort has brought me in contact with the OpenOV guys who are dedicated to liberate all public transportation data in the Netherlands. In the future, I hope I can contribute something to their wonderful initiative.

Doing consessions

Luckily, the TIMEMAPS project had one “business rule” with respect to its visualization: only stations that are near the border of the map are allowed to modify the map. That made the list of stations considerably smaller: after selection there were 60 stations left.

However this limited the practical application of the map in that some of the displayed travel times are not accurate: for the remaining, smaller / non-border stations we chose to interpolate the travel times between the “main” stations: an inaccuracy, given the fact that it often takes longer to travel from a minor station (e.g. Eindhoven Beukenlaan) to any other city. But for the sake for the clarity of the visualization, we agreed on this concession.

Data model & worker processes

There are two worker processees running in the background.

One process constantly (approximately 1 request per 1.5 second) queries the NS API for any A → B trip that has no planning in the future. This process favors distance: it tries first to find plannings for longest A → B trajectories, since the NS API also returns every timing information for intermediate stops, allowing to get more than one planning per API request. This planning information is stored in the database and kept for at least a week.

Table "public.static_planning"

 Column             |            Type             | Modifiers
--------------------+-----------------------------+------------------------
 id                 | integer                     | not null
 station_from       | character varying(32)       | not null
 station_to         | character varying(32)       | not null
 time               | timestamp without time zone | not null
 duration           | integer                     | not null
 ns                 | boolean                     | not null default true
 fetchtime          | timestamp without time zone |
 spoor              | character varying(32)       |
 aankomstvertraging | integer                     |
 vertrekvertraging  | integer                     |

Another process constantly queries the Actuele Vertrektijden API for every station (not only border stations). This information is used for the “fallback” scenario of step 3), in which no real planning is found for the station combination and we fall back on a fixed travel time, but do include the scraped departure time.

Table "public.vertrektijd"
     Column     |            Type             | Modifiers
----------------+-----------------------------+-----------
 station        | character varying(32)       | not null
 time           | timestamp without time zone | not null
 vertraging     | integer                     | not null
 ritnummer      | integer                     | not null
 eindbestemming | character varying(32)       |
 fetchtime      | timestamp without time zone |

Building the travel time matrix

The current map exposes an API to the N^2 matrix of the current time at the URL /api/reisplanner/actueel. It is a JSON long list where each entry looks like this:

["std",
 "amf",
 "2011-10-30 13:13:00",
 7440,
 "2b"]

This particular entry shows that the next train from Sittard (std) to Amersfoort (amf) leaves on 13:13h, from track 2B and takes 7440 seconds (2 hours and 4 minutes). For every station to another station (for the “border stations”) there is an entry in this list.

A second URL, /api/reisplanner/history?date=2011-10-29T22:00:00Z, gives this list for a certain date in the past.

Given the fact that we were unable to query every planning in real time, these results are build up in a three-step phase:

1. Given each station A, B, check if there has been a planning retrieved for A → B for which the start time is in the future. Return the planning that is closest to the current time.
2. Failing condition 1), check if there has been a planning retrieved for A → B last week. Return the planning that is closest to the current time minus 7 days. We assume that for every day of the week, the planning is the same. Note that this does not hold for holidays / festive days.
3. Failing condition 1) and 2), return the planning tuple in which we assume a constant, pre-fetched travel time (a static matrix for times between A and B without time information). We assume that the first train leaving for A is the right train for getting to B.

A combiner algorithm retrieves for every station-to-station combination the results from step 1, otherwise those from step 2 and as final fallback step 3 (which always has a result, although it might not be accurate).

mod_reisplanner – the module making all this happen

Above processes have all been implemented in Erlang as a module for Zotonic. It will be open-sourced soon, so that it hopefully can serve as a basis and/or inspiration for other applications using Erlang and the NS API.

This article is the second in a series about the TIMEMAPS project.  TIMEMAPS’ concept and design are by Vincent Meertens, the implementation is by Arjan Scherpenisse.

Arjan is that rare breed of artist né programmer formally trained in both but picking neither side. He is active on the most innovative edge of software as well as building physical interaction projects and schooling others in programming be it in Erlang or some other language.

The TIMEMAPS project written up just before this post is the first of we hope many forays into data visualization for Arjan and we look forward to collaborate on many such projects in the future.

The map is a real-time interactive map. Clicking a city allows one to set the perspective to a city of his choice. Hovering the map shows a pop-up which highlights the time it takes to travel to the city the mouse is currently over. Every coloured “ring” on the map denotes 30 minutes of travel time, at the current time.

Drawing the map with Canvas

The visualization is done using the HTML5 canvas. Why canvas, and not just SVG, one would ask? Good question: I wanted to learn more about the canvas and thus was a bit biased. I think the project could have been done with SVG as well.

The map consists of a set of polygons: the outline of the Netherlands and its various islands. All the cities are located on those shapes with all 379 train stations. Furthermore, there are several bridges between the islands, like the big “Afsluitduik”, which each connect 2 vertices of the polygons.

The initial, un-transformed shape of the country and the station positions is the same as that on the famous yellow overview map that the NS uses in the stations: it is a schematic view of the Netherlands, constrained in a grid of 0, 45 and 90-degree lines.

The drawing algorithm first draws all the polygons and bridges, and subsequently fills those areas with a pattern of colored concentric circles. This is done in canvas by blitting the previous shape with a pre-rendered image of the circles using the compositeOperation method. The distances between the circles are scaled to represent 30 minutes of travel time. Then, the cities are drawn as big/small dots (main stations are bigger) and connected to the current city by a thin white line.

The information hovers (a plain HTML div) are done by using the “mousemove” event on the canvas and calculating which city is the closest to the current mouse location. Clicking a city causes the current perspective to shift to the clicked city in an animated fashion, using a simple (cosine) transition.

Map deformation

The angles at which cities view each other are kept constant. So, for example, viewed from Rotterdam, Utrecht centraal is always at a 45-degree angle, regardless of the time it takes to travel from Rotterdam to Utrecht. The actual city location is scaled proportionally along these angles: if it takes less time to travel, the city is pulled closer; if it takes more time it is pushed further away. But the angle remains constant.

The polygons (that make the actual shape of the map) are “magnetic” and each vertex “sticks” to the cities it is initially closest to, in a weighted fashion. This algorithm is loosely based on the article “Feature point based mesh animation applied to MPEG-4 facial animation”.

For the islands, this mesh-stretching was mixed 60%/40% with a simple vertex displacement to prevent the islands from becoming unrecognizable: since there are no stations on islands, they are prone to more deformation since the feature points (cities) lie further away.

Problems in the visualization

The shape of the map sometimes is deformed beyond recognition because in certain cases cities which are normally close are being pushed away beyond cities that are normally far away: thus causing the polygon to turn “inside out” and cause cities to appear to be located in the sea.

Another issue is the 45-degree grid constraint: the mesh stretching algorithm does not take this into account because this constraint is applied in a later calculation stage: this sometimes causes cities to be located in the sea as well. A temporary solution for this was to add more vertices to the polygons so the map had more flexibility while stretching.

Application to other maps

The Netherlands is a pretty ideal country in the way the transportation system is organized: viewed from the center, “de Randstad”, or Utrecht or Amersfoort, it is indeed so that travel times do increase almost linearly with geographical distance. I do not think this holds for every country: especially with the advance of faster railways (the fast Fyra train was not taken into account in our implementation!), the map might deform in ways that are beyond recognition and beyond representation in the 2D domain.However it might be an interesting experiment to apply the same techniques to a different country.

This article is the second in a series about the TIMEMAPS project. TIMEMAPS’ concept and design are by Vincent Meertens, the implementation is by Arjan Scherpenisse.

A video of the screen:

The screen in context:

It is a welcome refresher from the static posters and the static videos that usually litter these high profile locations. The foursquare coloured view of the area is always fresh and shows a view on the local flavour and the people that visit the venues around.

From an urban development point of view it may be odd to draw more attention to the already highly crowded Leidse Square area. But it comes to reason that new developments such as these will be tested on high density locations first. We would be very interested to create augmentations in public space to make locations in Amsterdam’s periphery more appealing.

First: I do not agree with the premise that most apps created in government challenges are quickly abandoned. I have not done a tally of our Apps for Amsterdam contest, but the completeness and polish of most apps submitted was impressive. I still use several of the apps from that contest regularly. Snelstepontje.nl for finding out which ferry to take is a godsend just to name one.

Maintenance is indeed an issue. It is my personal experience that if the app is deployed to a suitably robust platform (Google App Engine is a notable one), it may continue to run unsupervised for many years.

But yes, I do have my own doubts when it comes to the sustainability of apps from app contests as I have stated in my review of Apps for Amsterdam.

Data quality is the largest issue on all levels and it needs to be addressed. From gathering data, to publishing it, to responding adequately to issues. Most datasets that are released for contests are not of the highest quality due to time constraints. And after the contest is over they are seldom kept up to date by the publishing office. When it comes to sustainability, government should first turn to itself and start releasing their data in a way that is sustainable.

Besides releasing the data in a proper format, a very important consideration is the licensing. Re-using data should happen under conditions as liberal as possible (CC0 preferred) as not to deter companies from investing in using that data.

But even then creating apps that are successful and sustainable at scale may be too lofty a goal. Productizing apps in a professional way implies conceiving, building and expanding a startup company. If one or more such initiatives come out of a hackathon that may be called a resounding succes. But what of the rest?

Well, communities of practice are built on exactly that: practice. Data does not overnight become readily at hand and usable. It takes a lot of hard work from all of us.

Having organized several hackdays we are seeing an increase in number of people attending and their proficiencies as well as a wider awareness of the possibilities of data in journalism, government and politics. Those are exactly the things we need if we want to make open data (and not just applications) the foundational fabric of our information society.

In the summer lull over at Hack de Overheid we are organizing a day of programming at a fortress which in itself already is a unique event: Apps for Noord Holland. But during the day the people from ESRI will give a workshop about geo data which we think is very worthwhile for any programmer who wants to get started in this field.

So if you want to spend a day on a fortress learning about GIS and programming, go right ahead and register. It promises to be a terrific day.

In the animation, if you look at the still at August of 2009, you see the US gaining some traction and this flare across the pond. That is Amsterdam where at the time Foursquare was being adopted hand over fist.

The story behind that is somewhat interesting and has been told, but this graphic does make it poignant again. Having visited SxSWi that last March, Robert Gaal and myself saw the launch of Foursquare and quickly got hooked. That was the year that location had not been played out yet at all, Latitude was fresh, Fire Eagle was still relevant and Brightkite was being used. Location was on the cusp.

Back in the Netherlands we quickly got in touch with the guys to get the service launched here. We thought waiting would probably result in the Netherlands being served last (as usual). After some back and forth we got everything up and running and Amsterdam was the first international city on Foursquare. The rest is history as can be seen in the graph.

There are several similar tools out there that help you create your own map but we feel that they are not as easy as they should be and most all of them are created in Flash. Statlas is built on Polymaps and therefore fully compatible with the open web. Creating a map is a simple as painting by numbers.

Our initial explorations set us on our way to create the easiest and most generative atlas tool we could imagine. Statlas is setup to allow you to choose a group of regions and for each of those regions enter a value (numerical, color or other) to create a map coloring. That map can then be shared, printed, embeded wherever you want . But anybody can also take a public map and edit it to improve upon existing data or to express their differences with them. It is also possible to export data to CSV, use other tools to collect statistics and re-import them back into Statlas.

Feedback

This initial release is geared towards the Dutch context as we have been developing it with the Netherlands in mind first. We are going to quickly add more regions and we are solliciting requests for regions you may want to add. If you have ideas, requests and or Shapefiles, please send them our way so we may add them.

This is a most preliminary beta release of a functional piece of software. We are envisioning much more data heavy and live updating views in the near future, but a project of this scope can balloon too easily. We’ve heard no end of people who wanted to use it for one cause or another and we wanted to show something first. After this release we’ll see which direction is most in demand of pursuing.

Last year at the end of summer I helped Thijs Kleinpaste and Stefan de Bruijn co-author a proposal to sponsor open data within the municipality of Amsterdam. This proposal was accepted near unanimously by the commission in November (full write-up) and it started a roller coaster ride for open data in Amsterdam that is now starting to have far wider effects throughout the Netherlands.

Hack de Overheid (Hack the Government), the soon-to-be foundation I’m in the board of, partnered with the City of Amsterdam and Waag Society to realize the competition and a series of events. This series culminated for us in Hack de Overheid #3 an inspiring day and hackathon for over a hundred developers who built civic apps.

The completion of the contest tonight and the sometimes stunning applications —many of which display excellence in cartography and visualization— submitted to it mark another high point I am proud to be a part of.

What’s next?

But as I said this completes just the first stage of what is bound to be a long and tortuous road. As we speak there are local initiatives being formed to open up data in at least Enschede, Rotterdam, Utrecht, Eindhoven and the Hague. It will be interesting to see what comes out of that and if some of the smaller cities may in fact outpace us here in the capital.

But we need to do more. Recent questions about privacy violations in data releases make it more than a little obvious that there is a massive issue in data literacy. I wholeheartedly agree with Adam Greenfield if he says that data and its affordances need to be a core subject starting from school onwards. We need to explore materials, interventions and processes that allow us to teach data literacy and that allow others to teach it for us if we ever want to spread this knowledge at scale.

Literacy is required not only in school children but also in decision makers in business and government right now if we want to keep the momentum we have right now. Future developments run the risk of being hamstrung by backlashes against the malignant consequences of data or open data being unused because the ecosystem is not in tune. There are still lots of issues to be resolved around ownership, privacy, responsibility, licensing and business models.

From a commercial point of view, the sustainability of many of the applications in the contest is doubtful. Creating proof of concept apps for the data is a more than a good start, but it is by no means enough. The real need is for open but comprehensive systems where open data is a given. That data needs to be technically excellent and fully engrained in the fabric of our information society so that everybody can use it to enrich their app/site/discourse. Data owners and producers need to participate and be accountable for their data to accept feedback from the public both in the specific and in the general case. Such a system cannot be built or be static, but needs to be grown and evolve continuously. The only thing we can do is plant, nurture and weed.

So tonight will be fun, but let that not distract us from the massive amount of work still ahead. We are ready for it. Will you join us?

1. May 24th – Technical review of city dashboard concepts at HvA
   A brief bit of teaching with design and technical critique of city visualization dashboards developed by students.
2. May 25th – Apps for Amsterdam Awards Night
   Judging and attending the awards for the Amsterdam open data application contest.
3. May 27th – What Design Can Do
   Presenting an engaged data-centric approach for designers’ benefit (blurb).
4. Update: May 27th – Participating in an expert meeting on the topic of Cities, Games and Data organized by Virtueel Platform (our entry).
5. June 6th – Spring Break Stimuleringsfonds voor de Pers
   Going to demo Statlas live for the first time at the congres of the people who funded it.
6. June 20th – Expertisedag Journalistiek
   Giving an introduction to data journalism to a large amount of journalists.

And Kars & Alper will be attending the Alice Taylor lunch lecture this Thursday in Hilversum. Talk to us at any of these events if you are present.

Starting off with some introductions. We are Monster Swell, this equation is the central challenge of our practice.

To start with the title inspiration for this talk. I recently finished this book by Jane McGonigal.

“Reality is Broken” by Jane McGonigal recently came out and it’s not really true, but it’s quite opportune. Reality isn’t broken, but there is —as always— lots that can be improved. Slapping a gamification label on that is a false exit because it implies that such improvement can be done easily by the magic of games.

The core idea of the book is that:

1. Reality can be fixed by game mechanics (voluntary participation, epic stories, social collaboration, fitting rewards), and
2. That reality should be fixed by game mechanics.

Both of these points: the possibility and the desirability of such are the subject of fierce debate both within game design circles and without.

We are now seeing a superficial trend of gamification, badge-ification and pointification where everybody is rushing forward to add as many ‘game-like’ features to their application/concept to look tuned into the fun paradigm.

Fortunately this does not work. Checking in for points and badges is fun at first, but is hardly a sustainable engagement vector. Foursquare mostly did a bait and switch with their game until they got enough critical mass to be useful along other vectors.

Things that are difficult remain difficult even if they are gamified. ‘An obstacle remains an obstacle even with a cherry on top.’

Ian Bogost terms this exploitationware. Our own discussions concluded with that if you are not the one playing, you are being played.

In our practice we look for deeper ways to engage people and affect them. There are hardly any one-to-one mappings to be found and the effects that are most worthwhile are the higher order ones. As Kars Alfrink says:

“We don’t tell them to coordinate, we create a situation within which the way to win is to coordinate.”

Corollary: A game about violence does not immediately make people violent.

So what do games have to do with visualizations?

Most people will recognize this as being a goban.

But another way of looking at it might render it as a map. The metaphor of men and liberties and territory to occupy already points towards that comparison.

Looking at it in another way it could also be a Cartesian grid with binary data values plotted onto it. A data visualization of a phenomenon we don’t know (yet).

Frank Lantz’s GDC presentation was very opportune (and very worth watching) for this particular point: “Life and Death and the Middle Pair: Go, Poker and the Sublime”

Coming back to the map parallel, this picture of center pivot irrigation systems (by NASA) in Garden City, Kansas looks awfully similar to the goban and this is just an aerial photograph with some processing applied to it.

So to come to this point:

‘Any sufficiently abstract game is indistinguishable from a data visualization.’

The difference just is that a game is a visualization of a game model and its rules. The whole point of playing a game is learning those rules and uncovering the model of the game is essence ‘breaking’ a game. After this point it usually ceases to be fun.

And its complementary point:

‘Any sufficiently interactive data visualization is indistinguishable from a game.’

And indeed the best ones are highly interactive and offer various controls, abstraction levels and displays of data deep enough to engage users/players for a long time. It is also the reason that in our practice we don’t occupy ourselves much with visualizations in print media.

To continue the point about games: many games are either quite concrete or very abstract simulations. This is most obvious with sim games such as Sim City pictured below.

Simulations are subjective projections of reality both because of the choices that the designer of the simulator has embedded in their choices for the projection and because of the interpretation of the player of the simulation and how their ingrained notions allow them to interpret the simulation.

Ian Bogost (picture) in his book Unit Operations coins a state of being called ‘Simulation Fever’.

Kars described this in a recent presentation at Mobfest:

Bogost says that all games in some way are simulations, and that any simulation is subjective. The response people have to this subjectivity is one of either resignation (uncritically subjecting oneself to the rules of the simulation, taking it at face value) or of denial (rejecting simulations wholesale since their subjectivity makes them useless). Taken together, Bogost calls these reactions simulation fever. A discomfort created by the friction between our idea of how reality functions and how it is presented by a game system. The way to shake this fever, says Bogost, is to work through it, that is to say, to play in a critical way and to become aware of what it includes and excludes.

I think we could use the correspondence between games and visualizations to coin a corresponding term called Visualization Fever.

Those are my most important points, that good and interesting games and good and interesting data visualizations share many of the same characteristics. We can use data and its correspondence with reality (or lack thereof) to create a similar fever.

(This graphic is somewhat rudimentary but it was made within Keynote in five minutes and I hope it gets the point across.)

The visualization process shares a lot of similarities with the open data process that we are involved in. It is a perpetual conversation and the visual part is only one place where it can be improved. Data collection, discussion on results and errors, sharing of data and the resulting products, controllability of the outputs and being able to remix and reuse them and incorporating this process as feedback back into atoms are all areas that need active participation.

There is nothing easy about this. It is a ton of hard work and long tedious conversations. Fortunately most of it is worth it.

Some examples of visualization fever in action.

Verbeter de Buurt is the Dutch version of See Click Fix and it works really admirably. It creates a subjective map of an area with the issues that a group of people have signalled in their neighborhood. Nothing really is said about who these people are and if these issues are indeed the ones that are the most pressing (we all know the annoying neighbour who complains about dog poo to whomever will hear it). By making issues visible, this map imposes its view of the city onto the councils and exerts change.

Oyster Flowprint from Anil Bawa-Cavia on Vimeo.

Planning systems at an urban scale is a very difficult process. These planning stages are being opened up to the general public using consultation and other means but it remains to be seen if and how citizens can comprehend the complex issues that underlie city planning.

One step to help both experts and laypeople to better come to grips with the city that they are inhabiting is to create macroscopes that in one view show the entire scale and all the things that are in a system in such a way that we can make (some) sense of it. These Flowprints by Anil Bawa-Cavia are a great example of doing such for public transportation.

And done right these visualizations can reveal the systems of the world or in this case the order flow of trains in the Netherlands. Everybody knows how crowded Dutch rail is, which trains go where along which routes, but actually seeing it happening in front of your eyes in a real-time visualization gives you an insight and a tangible grip on the system that you did not have before.

So what do we fix?

We use visualizations and their compressed interactive views to expose system design choices and errors. They can also be used to give depth to a specific point, something which journalists are increasingly finding necessary. People consuming data heavy news want to be able to poke that data themselves.

A lot of visualizations I have seen thusfar serve not much more than to reinforce pre-existing judgements almost as if the person creating the visualization sought to build that which they wanted to see. Visualizations will need to be better, more flexible and draw upon more data if we want to break out of these throughs of shallow insight.

The brief as stated by the nice people at Bloom as well is that having a visualization serve solely as a visual output is too limited a use of the interactions created. You should be able to use the same interactions in the visualization to also influence the underlying model either directly or indirectly. That is to say the model and the representation should be bidirectionally influencing.

Planetary, the latest app by Bloom is a great example of that. It shows you a beautifully crafted astromusical view, but it also allows you to play your music library from within that very same visualization.

We need to bring visualization and deep data literacy to the web and infuse any relevant site and system (that is to say all) with them. Many people asking for data visualization think that they are some magical fairy dust that will make a site awesome by its very touch. This is of course not true.

Data and interactive visuals can generate value and insight for any site that employs them properly.

In the presentation Data Visualization for Web Designers by Tom Carden he remarks that web developers already know how to do all this. These are exactly the tools we have been employing over the last years to create interactive experiences (and we plan to use them more and more).

Internet Explorer is still the cripple old man of the web, but given understanding clients (and users) and some compatibility layers, you may be able to get away with using a lot of this stuff as long as the result is awesome enough.

The other trend is the idea that there need to be bridges built between web people and GIS people. Preferably how to create GIS-like experiences using the affordances that the web necessitates. A trend we were thinking about neatly summarized (blog) at a #NoGIS meetup by Mike Migurski.

GIS people have tremendous tools and knowledge but they are not accustomed to work in a very web way: quick, usable, beautiful. Web people can build nice sites pretty quickly, but they tend to fall flat when they need to work with geographical tools that are more complex than the Google Maps API.

If we can combine these two powers, the gains will be immense.

We can create subjective views to exert power upon reality and try to fix things for the better. The subjectivity is not a problem, as often the values embedded in the views are the very point. Subjectivity creates debate and debate moves things forward.

The tools we have to create these views are getting ever more powerful, but there is also a lot of work to be done.

As a wise man said: “The best way to complain is to make things.” (picture)

So better start complaining.

The project is live at its own URL and in an iframe on the IAmsterdam site.

Technically we used Foursquare’s OAuth2 API which is outstanding. To be able to share one token across all requests we employ a file based PHP cache that relays the necessary requests for us. Main technology was created in collaboration with Panman Productions.

The site shows canvas results tallied per city to show the biggest positive and negative issues according to constituants and their perception of politics.

It got some attention on various weblogs: Arnhem Direct, Sargasso, PvdA.nl, Johnny Wonder

The potential for a data driven approach to politics is tremendous. A site like this in effect gauges the sentiment in any given locality and in an ideal scenario it would also give people and politicians ways to collaborate to improve the situation. Any improvement realized can then be recorded and used to rally voters at subsequent elections.

It promises to be a fun event and you can still register at meetup.

As an end result 75 submissions were returned with answers in essay form detailing the biggest problem of the Netherlands, the most overblown issues and the most unnoticed issues according to the scientists. This made for a very large amount of textual content which would have been difficult to quickly get into.

We chose to see how quickly we could hook up Protovis to visualize the key issues according to each scientist. All of the essay style answers were clustered to a set of themes (by the people preparing the story) and this was input to Protovis’s bubble chart to give a tag cloud like representation of the issues. See the interactive chart on Groene.nl or the screenshot below:

The quick visual summary and the filters help drill down to a specific issue in a specific problem category quickly. Clicking a bubble displays links to the full text contribution of the relevant scientists.

This was mostly a process exploration to see how a default library such as Protovis could be employed in a journalistic context and to see where the bottlenecks fall. We found that Protovis’s explanatory power really shines if you have a good dataset. However it took some time to get the data machine-ready. The result was produced efficiently and adds a much needed visual summary to the slew of textual content. Most time was spent on wrangling the dataset and finalizing the interaction details of the chart.

The project got a fair amount of attention in national media (and links to the chart) e.g.: ‘Integratie meest overschatte probleem van deze tijd’ , ‘Wat zijn de 10 grootste sociale problemen van Nederland?’

Each year Hack de Overheid holds a developer day where civically inclined programmers gather to exchange knowledge and create new open data projects either with government’s consent or without.

This year the devcamp is part of a broader program along with an application contest for local data and local applications in the city of Amsterdam called Apps for Amsterdam. There is a lot of momentum and it looks like open data is finally being taken seriously.

Until the event, updates here may be a bit sparse, but do register for the March 12th event if you have any interest in data and let’s create something great together.

Statlas is going into full production.

The past weeks Alper has been giving lectures at the Willem de Kooning design academy on the subject of data visualization. The students should be busy creating their projects these coming weeks and we eagerly anticipate their results.

Hack de Overheid which we are co-organizing is going into full swing with the annual developer event on March 12th in Amsterdam (more on which in a separate post).

We will be represented at the Cognitive Cities conference in Berlin this weekend to talk about city data visualization. And next week we’ll be at the Infographics conference trying to talk some sense into those that think print is the end all of data.

The documentary deals with the flash crash of May 6th, 2010 when the black box trading operations on Wall Street went haywire and dropped the index 900 points to recover just minutes later. I’d already read about the possibility of such events from Kevin Slavin’s January 2010 Social Computing Summit presentation which has been noted down and blogged about by Michal Migurski. Both are recommended reading.

We have seen various troubles with the stock exchanges in the past year and this event especially seems one worth investigating because it exemplifies the complexity in todays exchanges and the total lack of control humans have over the process.

Another reference which I thought was important is the concept of the macrospcope by John Thackara which I first heard used and expanded upon in Matt Webb’s Reboot keynote. The related reading on the BERG blog is interesting but primarily the definition by Thackara: “A macroscope is something that helps us see what the aggregation of many small actions looks like when added together.”

So let’s see.

Documentary

For an international audience the concept of Tegenlicht may need some explanation. Tegenlicht is a documentary that examines world events by interviewing experts interspersing the interviews with visuals and a voice-over to create a dramatic storyline. The app contains the entire show in high quality which is in part why it is so heavy.

The shelf life of Tegenlicht documentaries is quite high. For another concept we recently rewatched their 5 year old documentary ‘De dag dat de dollar valt’ (Eng. The Day the Dollar Fell) because it was still relevant and interesting. The 45 minute length with drawn out shots can be a bit taxing for today’s YouTube attention spans, but byte-sized information is not their game. TED is much better at that. They concern themselves with the documentary as a dramatic art form that needs to engross its audience.

Given that concept —highly traditional television, cinema almost— it is interesting how you would interject/overlay/add interactive features into the narrative whole. This was touched upon briefly in the presentation, but that is not what this app concerns itself with. You can view the documentary and jump back and forth through the various segments while additional content is presented for your perusal.

It is clear though that traditional broadcasters are still very much struggling not only with the internet but also with the spectrum of television, cell phone, laptop, iPad and the locus of interactivity (if there is any interactivity). Tweede scherm is one such recently award-winning concept that displays supplementary information to add context to the main experience on the large screen.

Infographics & Visualization

The app is decked out with a nice cadre of infographics and visualizations and those are indeed its most important selling point. There is a list of them on the home screen. Several time series, an animated display, a multi-layered map overlay and a world map with live stock updates:

The visualization that gets the most emphasis and also is used often in the documentary is a time series display of the stock price around the crash:

It starts out nice and flat with a display of trading velocity (not quantity) and pricing information along with the time. There is a global display with scrubber that you can use to navigate over the entire run of the data and the crash is nicely colour coded. The vertical scale is a bit confusing as the one above does stop at zero (and then goes on for a bit more) but the other ones don’t. So confusing in fact that in the documentary one woman remarks ‘Apple is going to zero.’ which it is in fact not.

And then it goes South:

You can see the drop in prices and the variability. It becomes even more clear if you also enable the Bid & Ask information which is available as an overlay and shows you the differences between the prices asked and bid for the stock at that moment:

So that is a one-dimensional time series (with two extra dimensions available on request) with a beautiful presentation and animation. Another interesting piece of information is the potential locations for data centers around New York and what factors they need to take into account to carry as little risk as possible. You can see the map and the various factors involved and slice it yourself:

Another visualization is a map of the world (in catalogtree’s signature geographical bubble display) with semi-real time updates of the world’s exchanges and how they are doing right now:

The visualizations add a lot of panache to the documentary and are aesthetically very pleasing to behold which gives them a high show and tell value.

From an information design point of view however they are underwhelming. The information density is low, it is difficult to compare several datasets and the visualizations do not offer different types of information at different zoom levels. Also: the interaction is nearly trivial.

The issue of game design and game-like experiences was touched upon during the launch event to conclude that none of the makers had a lot of expertise (or even any affinity) with games. That is unfortunate because game design with its experience in dealing with highly interactive experiences of high density information spaces can add a lot to a data visualization.

Result

The whole issue of interactive television and how to combine a long dramatic form with visualizations (and what kind) seems to be a difficult one to solve and not the one being tackled here. As Erwin mentions in his review in Bright, added value is a highly pressing issue when it comes to traditional media trying to produce content for the iPad. That is exactly what this is: a nice packaging of a traditional television program with interactive features in a combination that will most probably remain interesting and relevant in the future.

The documentary is very attractively presented on the iPad. The extra content especially is more prominent than normal when it would have been put on a back page somewhere on the website. The video playback is also one of the first cases in Dutch broadcasting where the presentation is native to the device.

Another benefit is that this experiment can probably live on as a packaging format for other documentaries. The richness of the experience combined with the quality of the video, the pairing of additional content and the clear payment model make a lot of sense for something which already has high production costs. With magazines and newspapers you are adding a lot of extra weight to something that has a low margin and is ephemeral (daily, weekly). A well produced documentary such as Tegenlicht can live on for a long time and this seems to be a more suitable incarnation for that than most.

The VPRO has brought forth a great release of a beautiful looking product. Tegenlicht Money & Speed is out in the App Store in a Dutch version now with an internationalized release forthcoming.

It’s good to sea that there is a wider trend in consumer mapping platforms right when we are underway with ours. Here’s a brief survey of the ones we found during a cursory examination. There are bound to be more. If you know them, please let us know in the comments.

ArcGIS

ArcGIS has a mapping platform based probably on the ArcGIS server, a paid for cloud mapping platform.

Looks nice, like a web based version of Google Maps combined with Google Earth with all the different overlays you can put on there. I tried to create a map and share it on Facebook which oddly enough did not work. The sharing, embedding and standalone map versions do look well thought out but if they don’t work they’re probably not tested well.

View Larger Map

ESRI

ESRI the company behind ArcGIS has another ‘Make a map’ tool which is a lot more restricted but because of that provides a clearer experience.

This doesn’t offer a ridiculous amount of options, but it is very clear and nicely done and the sharing options are also very straight forward. An embed of that map is below:

Dotspotting

Dotspotting is Stamen‘s platform for putting dots on a map currently in its ‘SUPER ALPHA-BETA-DISCO-BALL VERSION’. As they describe it, it’s intended to make the process of visualizing city data easier, more open and more robust.

That is pretty much the same reasons we started on this road in the first place. Mapping and data literacy are necessary in web development as well as the other way around: web literacy is necessary for those that make the heavy-duty maps. The two need to meet to create the applications and ease of use we are looking for.

A script to export my Foursquare checkins in an easy way and create a sheet with those is forthcoming. Anyway, Statlas is best described as that: a way to project values onto regions and enable people to play with that dynamic.

Weet meer

Weet Meer got launched very recently in a beta release and is limitedly available up until next month. It does a decent job in displaying the statistics offered by the CBS and offers some statistical relations and tools to compare things with.

That is a brief overview of what is already out there. We’re glad that we have hit a nice timing to be able to develop ours and fulfill an actual need out there: to be easily able to make maps of a set of values to a group of regions.

Then they launched open.nos.nl to a limited amount of people at first to play around with and see the initial beginnings. It is quite modest right now with three end-points and not a lot of functionality or content to play with, but it’s a solid beginning and it is clearly meant to grow.

Busy though we are right now, I got a bit of an itch on Monday and whipped together this simple example: Ons Nieuws which is a glanceable news display of the latest news on NOS.nl meant to replace the tt101 screens that run at pretty much every news/publicity department.

Ons Nieuws is heavily influenced among others by: DEXTR. Glanceability and ambiance seems to be getting back in vogue also with apps such as Trickle. This thing we whipped up is really a very simple proof of concept version mainly meant to exercise the API, but it may lead to more serious applications.

Some projects we have been involved on in the past (Vervuilingsalarm, Schoolvinder) are even name checked by Ernst-Jan which is very nice except for one thing: these are prototypes we slapped together some years ago, in less than a day with minimal support AND these are still the best examples we have in the Netherlands of proper open data projects? It’s a testament to Google App Engine that these are still running and it’s a testament to the climate towards these kind of projects in the Netherlands that they have found little to no backing.

We are running into this problem when we are promoting open data in the Netherlands. It’s difficult to find sustainable, complete, thriving examples of applications built on open data like there are ample of in the USA and in the UK. So they ask: “Do you have any good examples of apps that will convince us why we should open up the data?” “Well,” we reply, “obviously if you haven’t opened much up yet, there won’t be too many good Dutch examples yet. But look at these from abroad!”

It’s a rather annoying Catch-22 that is used to justify institutional inertia. So yes, the growing mindshare around the subject is good and it’s flattering that the stuff we built is being used as the leading examples, but we will have to do better still. Stay tuned. We’re setting a bunch of stuff in motion that will nudge the status quo forward by a fair amount.

We got the announcement a couple of weeks ago that our proposal for subsidy had been accepted. We spent December gauging interest, taking in the project and building a team that can execute this in Q1 2011. Team introductions forthcoming after we’ve kicked it off.

The Brief

The assignment is still the same one that prompted the original Dutchstats:

Given a set of values for a set of geographical regions visualize the mapping from the values to the regions in a way that is interesting, useful and pleasant.

Simple enough to be doable. Broad enough to be generally applicable.

The original Dutchstats was mainly concerned with Dutch municipalities as geographical regions and election results as values and we will be continuing along that line, but we will be looking into opening up both the values and also the geographical regions for anybody who has something to contribute to either. The idea is to create a generative atlas.

Atlas

A generative atlas mostly to see if we can give the concept of an atlas new currency in the online world.

In the Netherlands there is an atlas called the Grote Bosatlas which still is the standard atlas for everybody in and out of school. But asking people around the question: when is the last time you have even thought of an atlas, let alone got and leaved through a Bosatlas, everybody draws a complete blank. Google Maps has supplanted most of the topographical and wayfinding functionality of paper maps and atlases to the extent that it has wiped out the original concept out of people’s heads.

The social geographical function of the atlas has been replaced by a ton of projects working either with or on Google Maps/Earth using GIS or placing points on the map (using location or geocoded data), Stamen’s Dotspotting is a good example of that. Besides those web centric approach there’s also a slew of closed/semi-closed mapping tools from statistical offices, government bodies etc. that are built on poor and closed technology and are limited to the task at hand (which they usually do poorly at that).

Technology

We’re going to determine as we go the technology that we’re going to use, but the project needs to be webcentric and is allowed to be bleeding edge (though perhaps not as bleeding as the original prototype) so I hope we can avoid using Flash completely.

Depending on how much of the base components are already available (data stores, tile servers, rendering engines), we will be focusing more on the application part. But if such components are not yet available or up to par, we will be investing in building them ourselves.

Process

In our practice we believe in standing on the shoulders of giants, sharing alike and giving credit where credit is due. We will be doing this project completely in the open not because we don’t have a customer for it but because everybody is a potential customer and they should be able to see and participate from the earliest stages on.

Any software that we produce will be released under a very liberal open source license. So that anybody can use our stuff and we hope to advance the state of mapping online in our own modest amount. Also all our design research and progress will be posted to this blog in chunks of a week or a bit more (depending on our sprints).

Fully open is the only way we can imagine doing this. We hope you will join us.

Spotted by Kars Alfrink at the Dutch Game Garden:

The front of the card bears a logo of which there are two currently consisting of the text Monster Swell in all caps in Akzidenz-Grotesk —a choice font— occluded by a waveform image. That is conceptually consistent and pretty straight forward.

The backside of the card is a pastiche of a map/infographic to get a handle on the regular clutter you see on business cards. We’ve tried to make clear choices, compress the content and make it readable for those familiar with these kind of graphics and those who are not:

It was a lot of fun and productive working with the Ponies and expect a collaboration on their side of the Randstad in 2011. We’re pretty pleased with how the cards turned out and we hope you like them too.

Luckily we can make it coincide nicely with the Open Data Day that takes place on December 4th. We understand that it probably is Pakjesavond that evening for some, but I have opened up @ouroffice in the Volkskrantgebouw for hacking during the daytime, see the City Events page on the Open Data Day wiki.

So that constitutes a two part event:

1. Hacking — @ouroffice (2nd floor, Wibautstraat 150, ring Alper +31-6-24553306 to get in) will be open from 12:30 to 16:00 for anybody interested to drink our coffee and sit down and hack or discuss technical issues. A heads up as a comment, tweet or text that you will be there would be nice and will help us plan. The office seats around 8 people and has WiFi but that can be expanded.

2. Drinks — We will then adjourn at 16:00 for Saturday afternoon drinks to the nearby Ysbreeker (follow @MonsterSwell for updates on that) which should give everybody enough time to drink a couple of beers and then either enjoy Sinterklaas or go somewhere for dinner.

Feel free to join either: Facebook event page.

Dutch Data Drinks is an informal social event meant to get everybody interested in data together and talking to exchange information and form collaborations. Topics include: open data, transparent government, data visualization, cartography, statistics, data mining, journalism and pretty much anything generating, processing, analyzing or displaying data.

Update: It looks that there’s a bunch of interest from higher up and that during the hacking part some announcements will be made and some datasets revealed etc. More news to follow.

Alper will also join a Hack de Overheid team of experts to provide technical support at the Conference for Investigative Journalism in Ghent. We will aid journalists with their data issues and questions and we will also develop an application during the conference based on the demand we see.

Update: the slides of the presentation at SMC have been posted to their Slideshare.

Sound is not completely in sync, but this should serve as a nice narrative accompaniment to the slides posted here before:

Civic Duty in a Hyper-Connected World from Alper Çugun on Vimeo.

Alper used the opportunity to take three minutes to address the council before the meeting and posted a call to action for better and more effective digital public services using open data and asked the city to open up more of its data.

When the proposal was finally treated it was adopted near unanimously (tweet) by the entire council with also a positive recommendation by the alderman. The alderman commented that because he used to be an open source developer, an open data project had been on his list of things to do for a while now and he welcomed this proposal. His idea was to spend the allocated €10’000 on projects in the form of bounties to maximize the effectiveness and first grab the low-hanging fruit.

At the same time in San Francisco it seems an open data law was voted into effect (link to legislation). The ordinance is interesting to read and lists:

Findings:

An open data policy will provide benefits to the City, which include:

1. enhanced government transparency and accountability
2. development of new analyses or applications based on the unique data the City provides
3. mobilization of San Francisco’s high-tech workforce to use City data to create useful civic tools at no cost to the city
4. creation of social and economic benefits based on innovation in how residents interact with government stemming from increased accessibility to City data sets

City departments should take further steps to make their data sets available to the public in a more timely and efficient manner.

It would seem that the time is now ripe to push this agenda through local legislative bodies. Given the current trend towards better digital services and transparency a suitably drafted proposal for open data with a realistic goal can scarcely have any opponents.

We’re going to look into passing more proposals towards open data like this following the lead of Amsterdam.

Update: the minutes for the commission meeting have been posted: Dutch PDF

Esther Gons (also pictured below) does visual representations of complicated concepts and she did this sketch of my presentation:

Click for a bigger view and her take on the event.

Annotated slides and video will be available shortly.

The commission will have a consulting round on November 8th where I will be addressing the council to clarify the benefits of open data and exhort the city to open up.

Working on releasing it in more formats.

The situation is that Dutch transit information is controlled by a single private entity who are not sharing any of their data. The Dutch government plans to write a tender for a public data warehouse (to be called: NDOV) to be built with all transit information in it. Now this is both a threat for the private institution currently exploiting the data and an opportunity for the Netherlands to make a —much needed— jump forward in the realm of transit information. Which makes this an interesting turning point.

After making the altogether clear and well-known argument for open transit data, I propose three hard requirements such a data warehouse should fulfill. Here in English:

1. All public transit data: journey plans, live locations, departure times and (un)planned mutations in the NDOV have to be readable both for humans on a website and for computers via an API.
2. All this data has to be freely accessible to everybody without limitations.
3. There always has to be a high quality journey planner but others have to be allowed to build their own or build on top of it.

Doesn’t sound too unreasonable, does it?

Full text in Dutch on Alper’s personal weblog for those interested.

Their 18th edition on November 8th is on our topic: Data. So instead of organizing an event in November, I would like to point all of you towards there. It is a very nice event and like I said there will be data and there will be drinks, ergo: Dutch Data Drinks.

Their RSVP process is somewhat notorious due to the popularity of the event. So stay tuned for their first batch of tickets released on: Monday, October 25th – 12:00. After that there will be another batch and if anybody from this group hasn’t managed to get in by then, we’ll see if we can make other arrangements. But let me emphasize: you should be able to get in via the normal process if you just go to the Meetup page at the given time.

So see you all on the 8th of November!

I am intendant at an Amsterdam based technology and design firm called Monster Swell. We’re occupied with creating visualizations and other systems out of data. Some examples of our work:

We landscape silence.

We map discontent.

We create atlasses.

The issue I wanted to bring forward on this event about the future of hacking is about how we can hack our ways to a better future, and how we in fact MUST do that, because if we won’t change our future for the better, somebody else may do it for the worse. Hacking is in fact a civic duty.

First I will start with a background of the context we’re going to do this all in.

A Hyper-Connected World. Fantastic illustration (CC-by-nc license) from Tom Coates’s dConstruct talk.

Insert all your old web 2.0 stapels here: social networking, the web of creation. Be it on facebook, twitter, last.fm or wikipedia.

And we could use these to talk with each other about the things that occupy us, though most (not all) of this was very virtual.

Now the world starts to talk back to us using sensors, fields, smarts, wires and wireless

“That which primarily conditions choice and action in the city is no longer physical, but has become a matter of the invisible and intangible overlay of networked information.” —Adam Greenfield at LIFT

On the one hand everything’s going to be crammed full with sensors. The dutch word: ‘volgesensord’. A terrific sensorification.

Types of sensors: audio, movement, orientation, video, RFID, IR, etc.

Personal: everything you can cram into a cell phone

De-personalized: cc-tv, car/cycle-loops, transit gates

photo license: Creative Commons by-nc

Many of these sensors are personal and embedded in our most personal devices. The iPhone alone has a great many of them.

Wireless in the world from timo on Vimeo.

But many others are all around us with their own sensor fields and connections to each other and the network, permeating everywhere through our lives without us even knowing it.

“Wireless in the world” video by Timo Arnall

Pachube by Usman Haque is a network backbone for these kind of sensors. It is one of many visible switching stations of the data in/out points that you can use to link up with some of the sensors in the world.

So the world can sense data, but it can also show data back to us. Displays themselves by feedback and conditioning already change people’s behaviour.

Each and every representation we make is subjective, but it is a connection between the real life territory around us and the map we deduce from it (Tom Coates).

Reality is made up of the built environment, but more and more layers of information are getting added on top just like on this Pretty Map (by Stamen), which is pretty pretty.

Flickr Shapefiles for cities and neighborhoods
Urban areas from Natural Earth
Road, highway and path data from OpenStreetMap

GPS sensors in every train, added to the display, knowing where a transit object is at any given moment will influence your behaviour.

This got built by a couple of people on ScienceHackday in London in an afternoon. In the Netherlands we are prohibited by our transit agencies from building something similar.

This is Nuage Vert.

We can map these variables away from the computer screen and onto reality itself.

This is a particularly inspired example of that: “the vapour emissions of he Salmisaari power plant in Helsinki will be illuminated to show the current levels of electricity consumption by local residents.”

Remember Guid Raid 11/20/09 in Orgrimar from Daniel Petersen on Vimeo.

This is another kind of territory in World of Warcraft with a highly specialized and optimized display, made to help you to keep certain parameters within certain bounds and win the day for your team.

LayAR is an Amsterdam based company making great strides in adding an additional layer on top of reality and turning the above World of Warcraft video into reality quicker than I am really comfortable with.

Augmented (hyper)Reality: Domestic Robocop from Keiichi Matsuda on Vimeo.

The latter half of the 20th century saw the built environment merged with media space, and architecture taking on new roles related to branding, image and consumerism. Augmented reality may recontextualise the functions of consumerism and architecture, and change in the way in which we operate within it.

A film produced for my final year Masters in Architecture, part of a larger project about the social and architectural consequences of new media and augmented reality.

Just like the shipping container revolutionized transportation, we get to interact with the world because the roads have been cleared. The network and its basic transport has been standardized. Everybody speaks HTTP, APIs are roughly figured out, devices are talking to each other.

Fucking shipping container cliché (but, hey it works!).

photo license: Creative Commons by-nc-nd

We are trying to work this equation. Getting things to act in an interesting and useful way is one of the biggest design challenges of our time.

Wherein I try to illuminate some stuff about our time.

We are inbetween something, an interregnum of sorts. I can’t really prove this, but I guess more of you may feel that we are at an inflection point at times where great changes are afoot and nobody can say with any amount of certainty whether the future will look better or worse or make any meaningful predictions about it.

Most science fiction writers can’t write a future anymore because the present is too shaky, or if they do it’s usually rather dismal like this Douglas Coupland piece.

Gramsci said: “The old is dying and the new cannot be born.  In this interregnum there arises a great diversity of morbid symptoms.” “now is the time of monsters”

Referenced via Jon Taplin’s state of America.

Combine that with the Shirky principle: These old dying institutions are with their last breaths trying to preserve the problems they are solving while they may not be relevant anymore in a networked world.

photo license: Creative Commons by

Public space built around physical infrastructures was the meeting ground for people. This is Dam Square, this used to be a center of not only trading, but of knowledge exchange and power. How very much unlike it is today. Smart people would go here, now it’s just tourists and other people without taste that convene on this square.

When the city becomes an increasingly virtual entity composed of networked services, these publics will form around shared issues of concern, along non-traditional lines, and different places.

See the presentation on Digital Infrastructures by Martijn de Waal.

photo by Shane Gavin, license: Creative Commons by

The information we have at our disposal and the speed of media have lead to a partial crumbling of traditional authority not only in politics but also in law, medicine, media and any field really.

photo license: Creative Commons by-nc

The world is getting increasingly complex and interconnected

So with this hyperconnected world and the changes in society, what can we do?

Video: New York Talk Exchange

These issues mean that this is the time for hacking. We cannot lean back and trust the future will be better than the past. We need to make it so.

Hacking is using things for unintended consequences to improve the current situation. A better word for ‘innovation’ if ever there was one.

A bottom up approach each person has their preferences, their needs and their agency and they try to create better choices and reduced opportunity costs for themselves.

This is not me, but some fat dude on Flickr (though of course you only have my word for that). An example of people who are measuring and hacking their lifes are the Quantified Self group. A bunch of grownups who are sensorizing their lives and measuring EVERYTHING to be able to spend their time and energy better.

photo by Michael Verhoef license: Creative Commons by-nc-nd

Street Hacks, The street finds it own use for things (William Gibson). Creative adaptation. People modify the offerings of companies and governments for real life use already. How can they do the same for the information offerings provided?

How will information and interaction become a hackable commodity? What tools do we need to build for people to be able to create their own solutions?

photo by Jan Chipchase, license: Creative Commons by-nc-sa

Privacy does not equal secrecy. Privacy is the function where somebody is in control of their personal information and can see what happens to it. It is very much more a perceptual quality than it is a hard one.

Facebook saying that people can /control/ their information does not help anybody if the GUI and its side effects are too complex (still) for people to do so in a meaningful way.

Information works for those that are able to use it. Enabling literacy and offering information in a palatable way to people is a necessary step for everybody to be able to become a part of information society. Too much of technology as it is right now is magic for too many people.

A top down approach where we try to design a society that bypasses the tragedies of the commons and the lowest common denominators?

We need to create an increase in engagement and participation, judiciously using transparency and design to create better living conditions.

There are loads of initiatives and policy makers are desperate to include more people into the process but most efforts are not much more than lip service for a variety of reasons. We need to figure out better how to include more people.

Obligatory Jane Jacobs quote

This is a Maneki Neko, which is a cat and the eponymous short story by Bruce Sterling.

It’s a great read and a rather ideal solution to what a digital government should/could look like if everything worked perfectly.

Our government shows itself to be more than capable in wasting hundreds of millions of €s on failing IT projects without showing much transparency or innovation for it. Especially in this period of crunch, this is simply not good enough anymore.

Systems need to work, they need to be resilient and user friendly up to much higher standards than we have had thusfar. How we will do this is an important but open question.

Lessig has a point here and that faith is bearing awfully thin as it is in our part of the world.

Right wing parties in Amsterdam are right now interested in opening up budget data. Normally they aren’t this in favor of open data initiatives but their motivation here is simple and clear: they want to cut budgets, and to campaign to cut budgets, you must first show that there are budgets. Transparency helps them and if there’s no informed response, they will get away with it.

photo by Joi Ito, license: Creative Commons by

A bunch of examples how we are using data and representation to change our environments both from the top down as from the bottom up.

This is my personal map of Amsterdam on Weeplaces. I can see where I’ve been regularly and what my friends’ preferences are. This uses Foursquare for the location data, but that could be Facebook or Hyves or any other social checkin service.

The act of checking in creates a personal, local and global taste map of the area. All these checkins in aggregate give you views of the area, enable you to create personalized maps.

MIMAQ: Mobile Individual Measurements of Air Quality
project by Digitale Pioniers + WAAG

Democratizes the measurement tool so that everybody can measure themselves instead of relying on the measurements provided by some institution (RIVM). Also the feedback and presentation of measurement values is a lot better (a lot more gamelike) than those normally provided to us by government institutions.

Verbeter de buurt, the Dutch version of Fix My Street

Verbeter de buurt is a way of linking your mental model of what is wrong in your area with your local council and using the web to both campaign for your issue and publicly shame your council to fix stuff. A great way of how a subjective map can be used to change reality.

Chromaroma Visualisations from Mudlark on Vimeo.

A montage of visualisations generated from the travel data on the live alpha version of the game Chromaroma

Chromaroma, your ambient actions in London transit are a game for you and your team. Top down game mechanics can influence overall transit behaviour.

Get excited and make things. It is now easier and cheaper than ever before to organize, act and build/create stuff. So why don’t we go ahead and do that? Society is malleable.

I’m going to show some examples just how malleable things are:

Thanks Matt Jones for a visual and a Zeitgeist (image under Creative Commons by-nc-sa license).

A project by a friend trying to convert Times Square into art: Times Square to Art Square (new website and a party on Monday!). Justus Bruns had this idea at some point and because he was surrounded by the right people, they told him: “Why don’t you go do it.” Which he is now doing by sheer conviction and the help of a large group of people.

Another piece by Adam Greenfield on community development and the Network Revolution blog by Ernst-Jan Pfauth.

This is Newspaper Club (they just started shipping internationally). Russell Davies says: 
“There are massive bits of infrastructure lying around unused.”

The old is not going away any time soon, but we can use it in new ways. Repurpose existing systems in interesting ways by leveraging the network or similar effects.

This is a reprap. It is a special kind of 3D printer, one that has the plans and the facility to print out itself.

There are more fablabs and repraps in the Netherlands per capita than anywhere else in the world. A rare example of innovation subsidies gone to the right spot.

We have the tools and the means, the only thing remaining is to actually do it. Hacking is our civic duty. We can’t wait for hulks of institutions. They’ll only believe us if we show them it’s possible anyway.

And we have to be the ones who do it to be able to determine what the results are.
“What it is… is up to us.” —Howard Rheingold

Thanks to Adam Greenfield for opening up and stimulating much of the (ethical) crossover thinking between the real world and the online world.

Also credits to this deck by Chris Heathcote.

Presentation under a Creative Commons Attribution Non-Commercial ShareAlike license.

The news has been picked up by Emerce and Dutchcowboys among other publications.

The project was a proof of concept protoytype whipped together in a very brief time presciently using Google App Engine and Pachube both of which have grown leaps and bounds since then. Vervuilingsalarm is a concept that shows how you can use standard web technology and scraping to hook sensors into the wider web of data to modernize them and to help others build upon them. That it is still running nicely without any maintenance is a testament to the solidity of Google App Engine.

We would love for attention and funding to expand the project into a wider sensor network with more data patches, interaction and sharing built in to become a hub for Dutch environmental measurements while continuing to use the strength of external services such as Pachube and others.

Brief:

The Internet of Things is an abstract term for something that is quickly becoming real. The world is being filled with sensors and actuators, all of which are linked into systems and being fed back to us with real and virtual displays. This is changing the fabric of society and the definition of what society is. It is our responsibility as citizens of this hyper-connected world to hack our environments to work better however we can.

Hope to see you there and have a lively discussion on this topic.

The abstractness of the direction was deliberatly kept abstract and not explanatory, because only few people really understand the way Quants work and think.

So purely for visual effect (I would argue wholly against the quote above), but still nice at that.

Curious about the stuff catalogtree have been working on for what I believe is the same series but I haven’t kept track of television for ages now. (via @gertjan)

We had 30+ people cross-cutting disciplines from academia, business, policy, design and technology from Amsterdam and outside. Discussions were lively and ranged from the most technical to the more ideological.

What is definitive is that there will be a follow-up. Interest for the event was plentiful, but tonight was mired with scheduling conflicts. The next one will probably have a brief program of lightning talks, but will then also be largely focused on getting people interacting with each other. Ideas, projects, venue suggestions and sponsorship welcome!

Thanks to all those that attended, and to James Burke for encouraging the organization.

We imported over 10’000 records collected by volunteers into a map and aggregated results by city and by postal code. This makes it easy to browse through and drill down into the surveyed areas and see what the local issues are.

PvdA Survey Website

Attention: NOS.nl

Process: “Online campagnes — Canvassen voor de PvdA”

Creator: Alper Cugun