mugshotI am an assistant professor (UD) at the Web & Media group at the Computer Science department of the Vrije Universiteit Amsterdam (VU). I am also a senior research fellow at Netherlands Institute for Sound and Vision. In my research, I combine (Semantic) Web technologies with Human-Computer Interaction, Knowledge Representation and Information Extraction to tackle research challenges in various domains. These include Cultural Heritage, Digital Humanities and ICT for Development (ICT4D). More information on these projects can be found on this site or through my CV .

Speech technology and colorization for audiovisual archives

[This post describes and is based on Rudy Marsman‘s MSc thesis and is partly based on a Dutch blog post by him]

The Netherlands Institute for Sound and Vision (NISV) archives Dutch broadcast TV and makes it available to researchers, professionals and the general public. One subset are the Polygoonjournaals (Public News broadcasts) that are published under open licenses as part of the OpenImages platform. NISV is also interested in exploring new ways and technologies to make interaction with the material easier and to increase exposure to their archives. In this context, Rudy explored two options.

Two stills from the film ‘Steegjes‘, with the right frame colorized. Source: Polygoon-Profilti (producent) / Nederlands Instituut voor Beeld en Geluid  / colorized by Rudy Marsman, CC BY-SA

One part of the research was the autonomous colorization of old black-and-white video footage using Neural Networks. Rudy used a pre-trained NN (Zhang et al 2016) that is able to colorize black and white images. Rudy developed a program to split videos into frames, colorize the individual frames using the NN and then ‘stitch’ them back together into colorized videos. The stunning results were very well received by NISV employees. Examples are shown below.

Tour de France 1954 (colorized by Rudy Marsman in 2016), Polygoon-Profilti (producent) / Nederlands Instituut voor Beeld en Geluid (beheerder), CC-BY SA

Results from the comparison of the different variants of the method on different corpora
Results from the comparison of the different variants of the method on different corpora

In the other part of his research, Rudy investigated to what extent the existing news broadcast corpus, with a voice-overs from the famous Philip Bloemendal  can be used to develop a modern text-to-speech engine with his voice. To do so he have mainly focused on natural language processing and the determination to what extent the language used by Bloemendal in the 1970s is still comparable enough to contemporary Dutch.

Rudy used precompiled automatic speech recognition (ASR) results to match words to sounds and developed a slot-and-filler text-to-speech system based on this. To increase the limited vocabulary, he implemented a number of strategies, including term-expansion through the use of Open Dutch Wordnet and smart decompounding (this mostly works for Dutch, mapping ‘sinterklaasoptocht’ to ‘sinterklaas’ and ‘optocht’. The different strategies were compared to a baseline. Rudy found that a combination of the two resulted in the best performance (see figure). For more information:

Share This:

ArchiMediaL proposal granted by Volkswagen Stiftung

Volkswagen stiftung letterI received a good news letter from Volkswagen Stiftung who decided to award us a research grant for a 3-year Digital Humanities project named “ArchiMediaL” around architectural history. This project will be a collaboration between  architecture historians from TU Delft,  computer scientists from TU Delft and VU-Web and Media. A number of German scholars will also be involved as domain experts. The project will combine image analysis software with crowdsourcing and semantic linking to create networks of visual resources which will foster understanding of understudied areas in architectural history.
From the proposal:In the mind of the expert or everyday user, the project detaches the digital images from its existence as a single artifact and includes it into a global network of visual sources, without disconnecting it from its provenance. The project that expands the framework of hermeneutic analysis through a quantitative reference system, in which discipline-specific canons and limitations are questions. For the dialogue between the history of architecture and urban form this means a careful balancing of qualitative and quantitative information and of negotiating new methodological approaches for future investigation.

Share This:

A Look Back at the 2nd BDE Workshop on Big Data in Health, Demographic Change and Wellbeing

[reblogged from]

On 9 December 2016, the second workshop for the Big Data Europe Health, Demographic Change and Wellbeing societal challenge was held in Brussels. The aim of this workshop was to highlight progress from the BigDataEurope project in building the foundations of a generically applicable big data platform which can be applied across all Horizon 2020 societal challenges. This workshop specifically focused on health, and showcased our first pilot’s application to early bioscience research data.

The workshop in full effect

The workshop had 15 participants, from within the health domain and outside it, including many participants from the European Commission. Together we discussed different perspectives on how we may use appropriate H2020 instruments and work programmes to better integrate the ecosystem of linked data repositories, data management services and virtual collaboration environments to increase the pace of knowledge sharing in health.

The workshop featured presentations from BDE’s Simon Scerri and Aad Versteden on the general goals and progress of the BigDataEurope project and the BDE infrastructure respectively. After lunch, Ronald Siebes (BDE / VU Amsterdam) presented the first pilot in this specific domain. More information on that pilot can be found here. An extensive round-table discussion followed, in which possible options for new applications and connections were considered.

Snapshot of the SC1 pilot interface, as presented by Ronald Siebes

One question raised was whether the generic BDE infrastructure can be used by European SMEs. The fact that the BDE infrastructure is completely Open Source, very easy to install and features intuitive interface components makes re-use relatively simple even for smaller institutions and companies.

A significant part of the discussion focussed on possible new use cases for expanding the scope of the pilot. One suggestion was to look at post-hoc integration of clinical data, which represents a typical problem of data ‘variance’. This would require integrating information from different versions of medical questionnaires, which may be recorded or stored in different ways. Data provenance is also a key concern, as keeping a trail of what has happened to clinical data is crucial to tracking patients’ histories. Once integrated, this data could then be mined to identify biases or data patterns.

Finally, the workshop participants discussed potential connections to other European projects. Here many projects were mentioned including the MIDAS project, the Big-O project on childhood obesity, the PULSE projects and IMI / IMI2 projects including EMIF. We will be seeking collaborations with these projects and will continue to develop new and interesting Big Data use cases in this domain in the coming year.

More images can be found below: BDE Health Workshop SC 1.2

Share This:

Web of Voices and W4RA video at the Webscience@10 TV Channel

For its 10th anniversary, the Web Science Trust organized an event Webscience@10. For this event, a Webscience@10 TV channel was launched to showcase different research and education initatives around the world. On behalf of the VU Network Institute and W4RA, we submitted our Web of Voices video as well as a short introduction to the W4RA team.

You can watch the ~10 hours of video content at You can find us (listed under Netwerk Institute Amsterdam) at 2h31mins:

Share This:

Niels’ paper awarded first Bob Wielinga award at EKAW

Niels Ockeloen’s paper on Data2Documents was awarded the first Bob Wielinga memorial award for best research paper at the 20th International Conference on Knowledge Engineering and Knowledge Management (EKAW2016). “Data 2 Documents: Modular and distributive content management in RDF” was authored by Niels Ockeloen, Victor de Boer, Tobias Kuhn and Guus Schreiber from the Web and Media group.. The paper describes Niels’ PhD. work on a method for creating human readable web documents out of machine readable Linked Data, focussing on modularity and re-use. You can view the slides for Niels’ presentation slides here

Niels wins Best Paper Award

The award is named after Prof. Bob Wielinga, one of the most prominent European scientists in the area of knowledge-based systems, best known for his work on the KADS methodology, who has been one of the key influences on the development of the area in the past three decades. Bob was both my own and Guus Schreiber’s promotor so this makes it extra-special for us. In 2009 he was also appointed at our department, where he continued supervising PhD students until he passed away earlier this year. It is especially nice that the award, which was named after Bob Wielinga goes to work that is not only authored by people from Amsterdam but also work that Bob at some point discussed with Niels in the Basket, before his passing.

Share This:

W4RA research displayed in Museon

In modern day research, dissemination is key and it is therefore always nice to see research results being shared with the public in new and unforseen ways. Our work within the Web for Regreening in Africa (W4RA) is now part of a exhibition in the Museon museum in the Hague. The exhibition focuses on the United Nations’ 17 Sustainable Development Goals (SDGs) for 2030. As content partner of the Museon, the W4RA programme of VU has contributed ideas, visuals and texts for the exposition related to SDG No. 15, entitled “Plant in het Zand” (Plant in the sand).museon exhibition images

From the press release: Land degradation and desertification are increasing due to both natural and human causes, including climate change and population pressures. Areas can no longer meet the needs of their populations, with famine and poverty as a result. There are various solutions, but regreening – the natural (re)generation and protection of trees by local farmers themselves – is a highly successful one. Belts of trees act as windbreaks, helping to stop soil blowing away, keeping it moist for longer, and providing a micro-climate that is better for people, animals and plants. Trees also provide food and many other economically useful products.

Within the W4RA programme, we integrate local ICT web and mobile app innovations to support local knowledge sharing around regreening efforts.


Share This:

VU looking further in Mali

[This post by Anna Bon is cross-posted from See also the VU press release: VU looking further in Mali]

On 13 October 2016, the W4RA team organized and co-chaired, a Green Climate Funds workshop together with Malian farmer organization AOPP (l’Association des Organisations professionnelles paysannes). The objective of the meeting was to form a consortium and prepare a project plan, which will be submitted in the framework of this United Nations program.

The workshop was attended by representatives from the Dutch Embassy, the Swedish and Norwegian embassies, and by development (donor) agencies from the EU, Germany, the United Nations Capital Development Fund, the Global Environment Facility (GEF) and a range of Malian and Dutch development organizations.

The workshop in full effect (photo Anna Bon/
The workshop in full effect (photo Anna Bon/


Mali is one of the poorest countries in the world, plagued by the effects of climate change and a civil war in the northern regions. The effects of land degradation and desertification are a serious threat to the food security of millions of people, especially those living in rural regions.

Recently, the United Nations prioritized its support to Mali in the framework of the Green Climate Funds, a new programme to fight the effects of climate change on global scale. In response to a call for proposals, organizations in Mali are forming consortia, to prepare project proposals for funding by the Green Climate Funds.

Through ongoing interdisciplinary research collaboration, W4RA has obtained extensive experience in socio-technical field-based action research in West Africa. Building on partnerships with local partners (AOPP, Sahel Eco and Radio Rurale – Mali, Réseau MARP -Burkina Faso, University for Development Studies – Ghana) VU’s research programme W4RA wants to contribute to regreening, local knowledge sharing, local innovation and emerging rural agro-forestry value chains.

Meanwhile the W4RA is training students, through community service education, in rural Africa. This is done through the ICT4D master course (artificial intelligence, information science, computer science,) and various master research projects (Network Institute Academy assistants, various master research projects).


Share This:

The Role of Narratives in DIVE

[This post is based on Maartje Kruijt‘s Media Studies Bachelor thesis: “Supporting exploratory search with features, visualizations, and interface design: a theoretical framework“.]

In today’s network society there is a growing need to share, integrate and search in collections of various libraries, archives and museums. For researchers interpreting these interconnected media collections, tools need to be developed.  In the exploratory phase of research the media researcher has no clear focus and is uncertain what to look for in an integrated collection. Data Visualization technology can be used to support strategies and tactics of interest in doing exploratory research

Dive screenshotThe DIVE tool is an event-based linked media browser that allows researchers to explore interconnected events, media objects, people, places and concepts (see screenshot). Maartje Kruijt’s research project involved investigating to what extent and in what way the construction of narratives can be made possible in DIVE, in such a way that it contributes to the interpretation process of researchers. Such narratives can be either automatically generated on the basis of existing event-event relationships, or be constructed  manually by researchers.

The research proposes an extension of the DIVE tool where selections made during the exploratory phase can be presented in narrative form. This allows researchers to publish the narrative, but also share narratives or reuse other people’s narratives. The interactive presentation of a narrative is complementary to the presentation in a text, but it can serve as a starting point for further exploration of other researchers who make use of the DIVE browser.

Within DIVE and Clariah, we are currently extending the user interface based on the recommendations made in the context of this thesis. You can read more about it in Maartje Kruijt’s thesis (Dutch). The user stories that describe the needs of media researchers are descibed in English and found in Appendix I.

Share This:

CLARIAH Linked Data Workshop

[This blog post is co-written with Marieke van Erp and Rinke Hoekstra and is cross-posted from the Clariah website]

Linked Data, RDF and Semantic Web are popular buzzwords in tech-land and within CLARIAH. But they may not be familiar to everyone within CLARIAH. On 12 september, CLARIAH therefore organized a workshop at the Vrije Universiteit Amsterdam to discuss the use of Linked Data as technology for connecting data across the different CLARIAH work packages (WP3 linguistics, WP4 structured data and WP5 multimedia).

Great turnout at Clariah LOD workshop

The goal of the workshop was twofold. First of all, to give an overview from the ‘tech’ side of these concepts and show how they are currently employed in the different work packages. At the same time we wanted to hear from Arts and Humanities researchers how these technologies would best suit their research and how CLARIAH can support them in familiarising themselves with Semantic Web tools and data.

The workshop
Monday afternoon, at 13:00 sharp, around 40 people showed up for the workshop at the Boelelaan in Amsterdam. The workshop included plenary presentations that laid the groundwork for discussions in smaller groups centred around the different types of data from the different WPs (raw collective notes can be found on this piratepad).

Rinke Hoekstra presented an Introduction Linked Data: What is it, how does it compare to other technologies and what is its potential for CLARIAH. [Slides]
In the discussion that followed, some concerns about the potential for Linked Data to deal with data provenance and data quality were discussed.
After this, three humanities researchers from each of the work packages discussed experiences, opportunities, and challenges around Linked Data. Our “Linked Data Champions” of this day were:

  • WP3: Piek Vossen (Vrije Universiteit Amsterdam) [Slides]
  • WP4: Richard Zijdeman (International Institute of Social History)
  • WP5: Kaspar Beelen and Liliana Melgar (University of Amsterdam) [Slides]

Marieke van Erp, Rinke Hoekstra and Victor de Boer then discussed how Linked Data is currently being produced in the different work packages and showed an example of how these could be integrated (see image). [Slides]. If you want to try these out yourself, here are some example SPARQL queries to play with.hisco integrated data example

Break out sessions
Finally, in the break out sessions, the implications and challenges for the individual work packages were further discussed.

  • For WP3, the discussion focused on formats. There are manynatural language annotation formats used, some with a long history, and these formats are often very closely connected to text analysis software. One of the reasons it may not be useful to WP3 to convert all tools and data to RDF is that performance cannot be guaranteed, and in some cases has already been proven to not be preserved when doing certain text analysis tasks in RDF. However, converting certain annotations, i.e. end results of processing to RDF could be useful here. We further talked about different types of use cases for WP3 that include LOD.
  • The WP4 break-out session consisted of about a dozen researchers, representing all working packages. The focus of the talk was on the expectations of the tools and data that were demonstrated throughout the day. Various persons were interested to apply QBer, the tool that allows one to turn csv files into Linked Data. The really exciting bit about this, is that the interest was shared by persons outside WP4, thus from persons usually working with text or audio-video sources. This does not just signal the interest in interdisciplinary research, but also the interest for research based on various data types. A second issue discussed was the need for vocabularies ((hierarchical) lists of standard terms). For various research fields such vocabularies do not yet exist. While some vocabularies can be derived relatively easily from existing standards that experts use, it will prove more difficult for a large range of variables. The final issue discussed was the quality of datasets. Should tools be able to handle ‘messy’ data? The audience agreed that data cleaning is the responsibility of the researcher, but that tools should be accompanied by guidelines on the expected format of the datafile.
  • In the WP5 discussion, issues around data privacy and copyrights were discussed as well as how memory institutions and individual researchers can be persuaded to make their data available as LOD (see image).

wp5 result

The day ended with some final considerations and some well-deserved drinks.

Share This:

Downscale2016 proceedings published


Proceedings screenshotI chose to publish the proceedings of Downscale2016 using Figshare. This gives a nice persistent place for the proceedings and includes a DOI. To cite the proceedings, use the text below. The proceedings is published using the CC-BY license.

Victor de Boer, Anna Bon, Cheah WaiShiang and Nana Baah Gyan (eds.) Proceedings of the 4th Workshop on Downscaling the Semantic Web (Downscale2016). Co-located with the 4th International Conference on ICT for Sustainability (ICT4S) Sep 1, 2016, Amsterdam, The Netherlands. doi:10.6084/m9.figshare.3827052.v1 



Share This: