I am an assistant professor (UD) at the User-Centric Data Science group at the Computer Science department of the Vrije Universiteit Amsterdam (VU). I am also a senior research fellow at Netherlands Institute for Sound and Vision. In my research, I combine (Semantic) Web technologies with Human-Computer Interaction, Knowledge Representation and Information Extraction to tackle research challenges in various domains. These include Cultural Heritage, Digital Humanities and ICT for Development (ICT4D). More information on these projects can be found on this site or through my CV .
A look back at UCDS at ICT.Open2018
Two weeks ago, ICT.Open2018 was held in Amersfoort. This event brings together Computer Science researchers from all over the Netherlands and our research group was present with many posters and presentations.
We even won a prize! (Well, a 2nd place prize, but awesome nonetheless). Xander Wilcke presented work on using Knowledge Graphs for Machine Learning. He was awarded the runner-up prize for best poster presentation at ICTOpen2018. Congrats!
The knowledge graph for end-to-end learning poster at #ictopen2018 check it out at stand nr 4 @UserCentricDS pic.twitter.com/3INrp63lN9
— Victor de Boer (@victordeboer) March 19, 2018
Ronald Siebes presented work in the ArchiMediaL project on reconstructing 4D street views from historical images.
Let Ronald Siebes tell you all about reconstructing #4D street views from historical image collections, as proposed by #ArchiMediaL @VU_Science @tudelft @UserCentricDS pic.twitter.com/6WQUGta4VC
— Victor de Boer (@victordeboer) March 20, 2018
Oana Inel presented her work on Named Entity Recognition and Gold Standard critiquing. She also demonstrated the Clariah MediaSuite.
@oana_inel shows that 'gold standards' in NER might not be gold, nor standard. #ICTOPEN2018 @UserCentricDS @VU_Science #crowdtruth pic.twitter.com/cSRIH3YUS7
— Victor de Boer (@victordeboer) March 19, 2018
Advanced digital data services for media scholars: @oana_inel demonstrating the @CLARIAH_NL MediaSuite at #ictopen2018 @VU_Science @benglabs #clariah #digitalhumanities pic.twitter.com/K6wzGFdHFR
— Victor de Boer (@victordeboer) March 19, 2018
Anca Dumitrache talked about using crowdsourcing as part of the Machine Learning life cycle.
@anca_dmtrch proposes #crowdsourcing to fix errors in distant supervision data in the machine learning session at #ICTOPEN2018 #crowdtruth @UserCentricDS @VU_Science pic.twitter.com/2QXNcSkdRN
— Victor de Boer (@victordeboer) March 19, 2018
Tobias Kuhn talked about Reliable Granular References to Changing Linked Data, which was previously published at ISWC2017.
Cristina Bucur introduced Linkflows: enabling a web of linked semantic publishing workflows
I talked myself a bit about current work in the ABC-Kb Network Institute project
@victordeboer presenting "UX Challenges of information organization: the assessment of language impairment in bilingual children" @ #ictopen2018 @networkinstvu @UserCentricDS @VU_Science pic.twitter.com/2CY4esa4vy
— Oana Inel (@oana_inel) March 20, 2018
All in all, this was quite a nice edition of the yearly event for our group. See you next year in Amersfoort!
An Augmented Reality App to Annotate Art
[This post is based on the Bachelor project by Jurjen Braam and reuses content from his thesis]
The value of Augmented Reality applications has been shown for a number of different tasks. Most of these show that AR applications add to the immersiveness of an experience. For his Bachelor Project, VU student Jurjen Braam researched to what extent AR technology makes sense for the task of annotating artworks.
To this end, Jurjen built a mobile application which allows experts or laypeople to add textual annotations to artworks in three different modes. One mode doesnt show the artwork, but allows for textual input, the 2nd mode shows the work in an image and allows for localised annotations. The last mode is the AR mode, which projects the artwork in the physical space, using the device camera and screen.
Jurjen evaluated the three modes through a small user study, which showed that immersion and enjoyment was highest in the AR mode but that this mode was least efficient. Also, participants indicated that for annotation tasks, larger screens would be preferable.
This research was a unique endeavour combining a proven technology (AR) and well-known task (Annotation) which identified interesting possibilities for follow-up research.
- Read more in Jurjen Braam’s BSc thesis
- Check out the code on Jurjen’s github: https://github.com/JurjenBraam/AR-Annotator
W4RA work featured in BBC article
A BBC web article “‘Siri, will talking ever top typing?’ By Padraig Belton features our W4RA work done on voice interfaces for farmer information in Northern Ghana.
Francis Dittoh talks about the need for specific information for farmers in their own language and discusses ongoing research into our Kasadaka system. Anna Bon talks more about the web of voices. Very nice to see our work recognized by international media!
Read more at http://www.bbc.com/news/business-43409952
Field trip to Ghana, feb 2018
As part of the ongoing W4RA efforts, the VU ICT4D team visited West-Africa once more. This time, we visited Mali, Burkina Faso and Ghana. I personally went to Ghana to talk to external PhD candidate Francis Dittoh and his colleagues at the University of Development Studies (UDS) and the SARI institute in Tamale, in Northern Ghana. 
After first flying to Accra, I first was able to have a short meeting with my good friend Nana Baah Gyan, who is now an ICT for Development expert in that city. I then flew to Tamale,. where we met up with Francis to talk about his PhD work on information systems for rural farmers.
After colleagues Anna Bon and Hans Akkermans arrived from Burkina Faso, we met with UDS vice-chancellor as well as the Dean of the Faculty of Engineering to discuss the ongoing collaboration of W4RA and VU with Prof Saa Dittoh, and Francis. We hope to formalize these collaborations in a Memorandum of Understanding between the two universities. We also discuss the ambition of UDS to set up a curriculum in software engineering as part of the undergraduate programs. Such a programme would include a yearly community service courses, where students go into the field. This is very interesting for us as it aligns well with the goals of the ICT4D course at VU. The programmes also include a French language course to allow for smoother cooperation with other Sahel countries (specifically Burkina Faso).
We also visited the Savanna Agriculture Research Institute (CSIR-SARI). The situation is comparable to that in Burkina. SARI governs the use of hybrid seeds, which cannot be reused year by year but result in a higher yield. SARI is interested in educating farmers on how to handle these seeds (fertilization, planting etc). There are three classes of seeds: 1) breeder seeds – grown at research institutes, 2) foundation seeds and 3) certified seeds – produced by ~1000 farmers monitored by ~10 seed companies, and certified by an external agency.
Potentially interesting for us is their connection to rural farmers. SARI now mostly does this through extension workers from the ministry of Agriculture. However, because of budget cuts, these are now reduced to roughly 1 extension worker to 2,500 farmers. Therefore, SARI is open to the idea of commucating with farmers directly, for example through a voice-based system. An interesting opportunity could be two-track strategy with a smartphone app for extension workers to allow them to do their work better and a companion voice app for farmers. SARI is very much interested in developing applications in a co-creation process. This would match the research proposal that Francis has submitted to SARI and UDS.
A personal highlight was the 10hr road trip that Francis offered me to join instead of taking a flight back to Accra. Ghana is a beautiful country that changes before your eyes on such a trip. My sincere hope is that the proposed collaborations will lead to many more visits to this great country. 
A Voice Service Development Kit for the Kasadaka platform
[This post is written by André Baart and describes his MSc thesis]
While the internet usage in the developing world is still low, the adoption of simple mobile phones is widespread. A way to offer the advantages of the internet to these populations is voice-based information systems. The KasaDaka voice-services platform is aimed at providing voice-services in the context of ICT for Development (ICT4D). The platform is based on a Raspberry Pi and a GSM modem, which enables affordable voice-service hosting, using the locally available GSM network. The platform takes into account the special requirements of the ICT4D context, such as limited internet connectivity and low literacy rates.
This research focuses on lowering the barrier to entry of voice-service development, by reducing the skill set needed to do so. A Voice Service Development Kit (VSDK) is developed that allows the development of voice-services by deploying and customizing provided building-blocks. These building blocks each represent a type of interaction that is often found in voice-services. (for example a menu, user voice input or the playback of a message) The researcher argues that the simplification of voice-service development is an essential step towards sustainable voice-services in the ICT4D context; As this increases the potential number of local voice-service developers, hremoving the dependency on foreign (and thus expensive) developers and engineers. This simplification should ideally be achieved by providing a graphical interface to voice-service development.
The VSDK was evaluated during the ICT4D course at the Vrije Universiteit Amsterdam, where students built applications for various ICT4D use-cases using the VSDK. Afterwards a survey was conducted, which provided insight on the students’ experiences with voice-service development and the VSDK. From the results of the evaluation is concluded that the building-block approach to voice-service development used in the VSDK, is successful for the development of simple voice-services. It allows newcomers to (voice-service) development, to quickly develop (simple) voice-services from a graphical interface, without requiring programming experience.
The VSDK combined with the existing KasaDaka platform provides a good solution to the hosting and development of voice-services in the ICT4D context.
More details can be found in the complete thesis.A slidedeck is included below. You can find the VSDK code on Andre’s Github: http://github.com/abaart/KasaDaka-VSDK
KasaDaka: a sustainable voice-service platform – Master thesis presentation André Baart from André Baart
Machine-to-machine communication in rural conditions: Realizing KasadakaNet
[This post describes research by Fahad Ali and is based on his Msc. thesis]
Contextual constraints (lack of infrastructure, low-literacy etc.) play an important role in ICT for Development (ICT4D) projects. The Kasadaka project offers a technological platform for knowledge sharing applications in rural areas in Sub-Saharan Africa. However, lack of stable internet connections restrict exchange of data between distributed Kasadaka instances, which leads us to research alternative ways of machine-to-machine (m2m) communication.
Fahad Ali’s research focuses on mobile elements and using wifi sneakernets for this m2m to enable information sharing between geographically distributed devices. He developed a Raspberry Pi-based device called the Wifi-donkey that can be mounted on a vehicle and facilitates information exchange with nearby devices, using the built-in wifi card of the rPi 3.The solution is based on Piratebox offline file-sharing and communications system built with free software and uses off-the-shelf Linux software components and configuration settings to allow it to discover and connect to nearby Kasadaka devices based using Wifi technologies.
We evaluated the solution by simulating a low resource setting and testing it by performing so-called “pass-bys” in an Amsterdam residential area. In these cases, SPARQL queries are exchanged between host and client devices and we measure amount of RDF triples transferred. This setup matches earlier case requirements as described in Onno Valkering’s work.Results show that the system works fairly reliably in the simulated setting. The machine-to-machine communication method can be used in various ICT4D projects that require some sort of data sharing functionality.
You can find out more about Fahad’s work through the following resources:
- Fahad Ali’s Msc Thesis [PDF]
- The codebase for the configuration files on github.com/fahad105/masterproject/
- His final presentation slides (embedded below)
DIVE+ collection enrichment paper wins best paper award at MTSR2017
Last week, I visited the 11th Metadata and Semantics Research Conference (MTSR2017) in Tallinn, Estonia. This conference brings together computer scientists. information scientists and people from the domain of digital libraries to discuss their work in metadata and semantics. The 2017 edition of the conference draws around 70 people which is a great size for a single-track conference with lively discussions. The paper included interesting tracks on Cultural Heritage and Library (meta)data as well as one on Digital Humanities.
On the last day I presented our paper “Enriching Media Collections for Event-based Exploration” [draft pdf], co-authored with the people in the CLARIAH and DIVE+ team working on data enrichment and APIs: Liliana Melgar, Oana Inel Carlos Martinez Ortiz, Lora Aroyo and Johan Oomen. The slides for the presentation can be found here on slideshare. We were very happy to hear that our paper was presented the MTSR2017 Best Paper Award!
In the paper, we present a methodology to publish, represent, enrich, and link heritage collections so that they can be explored by domain expert users. We present four methods to derive events from media object descriptions. We also present a case study where four datasets with mixed media types are made accessible to scholars and describe the building blocks for event-based proto-narratives in the knowledge graph
Dancing and Semantics
This post describes the MSc theses of Ana-Liza Tjon-a-Pauw and Josien Jansen.
As a semantic web researcher, it is hard to sometimes not see ontologies and triples in aspects of my private life. In this case, through my contacts with dancers and choreographers, I have since a long time been interested in exploring knowledge representation for dance. After a few failed attempts to get a research project funded, I decided to let enthusiastic MSc. students have a go to continue with this exploration. This year, two Information Sciences students, Josien Jansen and Ana-Liza Tjon-a-Pauw, were willing to take up this challenge, with great success. With their background as dancers they did not only have the necessary background knowledge at but also access to dancers who could act as study and test subjects.
The questions of the two projects was therefore: 1) How can we model and represent dance in a sensible manner so that computers can make sense of choreographs and 2) How can we communicate those choreographies to the dancers?
Josien’s thesis addressed this first question. Investigating to what extent choreographers can be supported by semi-automatic analysis of choreographies through the generation of new creative choreography elements. She conducted an online questionnaire among 54 choreographers. The results show that a significant subgroup is willing to use an automatic choreography assistant in their creative process. She further identified requirements for such an assistant, including the semantic levels at which should operate and communicate with the end-users. The requirements are used for a design of a choreography assistant “Dancepiration”, which we implemented as a mobile application. The tool allows choreographers to enter (parts of) a choreography and uses multiple strategies for generating creative variations in three dance styles. Josien evaluated the tool in a user study where we test a) random variations and b) variations based on semantic distance in a dance ontology. The results show that this latter variant is better received by participants. We furthermore identify many differences between the varying dance styles to what extent the assistant supports creativity.
In her thesis, Ana-Liza dove deeper into the human-computer interaction side of the story. Where Josien had classical ballet and modern dance as background and focus, Ana-Liza looked at Dancehall and Hip-Hop dance styles. For her project, Ana-Liza developed four prototypes that could communicate pieces of computer-generated choreography to dancers through Textual Descriptions, 2-D Animations, 3-D Animations, and Audio Descriptions. Each of these presentation methods has its own advantages and disadvantages, so Ana-Liza made an extensive user survey with seven domain experts (dancers). Despite the relatively small group of users, there was a clear preference for the 3-D animations. Based on the results, Ana-Liza also designed an interactive choreography assistant (IDCAT).
The combined theses formed the basis of a scientific article on dance representation and communication that was accepted for publication in the renowned ACE entertainment conference, co-authored by us and co-supervisor Frank Nack.
You can find more information here:
- Josien’s thesis (pdf)
- Josien’s presentation slides (pptx)
- Josien’s code on Github (includes android apk)
- Ana-Liza’s thesis (pdf)
- Ana-Liza’s presentation (pdf)
ABC-Kb Network Insitute project kickoff
VU’s Network Institute has a yearly Academy Assistant programme where small interdisciplinary research projects are funded. Within these projects, Master students from different disciplines are given the opportunity to work on these projects under supervision of VU staff members. As in previous years, this year, I also participate as a supervisor in one of these projects, in collaboration with Petra Bos from the Applied Linguistics department. And after having found two enthusiastic students: Dana Hakman from Information Science and Cerise Muller from Applied Linguistics, the project has just started.
Our project “ABC-Kb: A Knowledge base supporting the Assessment of language impairment in Bilingual Children” is aimed at supporting language therapists by (re-)structuring information about language development for bilingual children. Speech language therapists and clinical linguists face the challenge of diagnosing children as young as possible, also when their home language is not Dutch. Their achievements on standard (Dutch) language tests will not be reliable indicators for a language impairment. If diagnosticians had access to information on the language development in the Home Language of these children, this would be tremendously helpful in the diagnostic process.
This project aims to develop a knowledge base (KB) collecting relevant information on the specificities of 60 different home languages (normal and atypical language development), and on contrastive analyses of any of these languages with Dutch. To this end, we leverage an existing wiki: meertaligheidentaalstoornissenvu.wikispaces.com
DIVE+ in Europeana Insight
This months’ edition of Europeana Insight features articles from this year’s LODLAM Challenge finalists, which include the winner: DIVE+. The online article “DIVE+: EXPLORING INTEGRATED LINKED MEDIA” discusses the DIVE+ User studies, data enrichment, exploratory interface and impact on the cultural heritage domain.
The paper was co-authored by Victor de Boer, Oana Inel, Lora Aroyo, Chiel van den Akker, Susane Legene, Carlos Martinez, Werner Helmich, Berber Hagendoorn, Sabrina Sauer, Jaap Blom, Liliana Melgar and Johan Oomen
