Seth Teicher of the crowdsourcing platform CrowdFlower has published a blog post about the CrowdTruth project. He summarizes what the implications of the research project are and what’s next for data enrichment and Watson.
Read the full story at the CrowdFlower blog: Crowd’s Collective Wisdom vs. Experts: Who Makes IBM Watson Smarter?
On Monday 31st of August I presented the preliminary results of my work on sound representations during the weekly Artificial Intelligence meeting at the VU University Amsterdam. In this collaboration with Emiel van Miltenburg, a sound corpus is built with annotations on how people perceive these sounds. Sounds can often be interpreted in multiple ways, but tags in sound corpora do not directly relate to the acoustic features of sounds. Because of this limited representation of what can be heard in a sound, the ranking of search results is not optimal. In this research, we use crowdsourcing to build an annotated corpus of sounds from freesound.org with meaningful representations that are perceptually grounded. The presented slides can be seen below or on slideshare.
On Friday 11th of September I pitched the medical relation extraction work of my CrowdTruth colleague Anca Dumitrache at the third Amsterdam Data Science: Coffee and Data event. The purpose of this was to get in touch with researchers that have medical datasets that are for instance incomplete or contain errors. With our research, we want to investigate how we can improve the quality of this data. Several other interesting presentations on data science in the medical domain were given at this event, which was hosted on the top floor of the VU University Amsterdam. Together with Merel van Empel, we also presented our latest work on gamification of crowdsourcing for advancing biology using BioCrowd. Feel fee to try out the game and provide us with feedback.
Our research group received several awards at the IBM Extreme Blue Expo 2015. In this event, an array of speakers from IBM, Academia and the startup community presented their latest findings. The Shared University Research Award was granted to the Web & Media group of the VU University Amsterdam. This award is a global initiative by IBM to stimulate science and the collaboration between IBM and scientists. Furthermore, Lora Aroyo received a faculty award for her work on our project CrowdTruth, and Anca Dumitrache reveived a PhD Fellowship award for her work on medical relation extraction.
As part of the shared university research award we received access to the Watson Engagement Advisor Research platform, which will be used for collaborative research on methods for the training and evaluation of IBM Watson. In the upcoming months we will jointly host a Watson innovation course for students and Watson masterclasses for professionals. More info will follow on this at a later stage.
On Thursday 9th of October was the Netherlands eScience symposium in the Amsterdam Arena. This yearly event attracts scientists and researchers from many different disciplines. In the digital humanities track, Oana Inel of the CrowdTruth team gave a talk on the Dive+ project. This is a digital cultural heritage project in which innovative access to online collections is provided, with the purpose of supporting digital humanities scholars and online exploration for the general public. This project is supported by the Netherlands eScience center, and uses CrowdTruth for the crowdsourcing of events in historical data. The talk titled “Towards New Cultural Commons with DIVE+” can be seen below.
Want to learn about IBM Watson and cognitive computing? Sign up for the Watson Innovation course! The Watson Innovation course is a unique collaboration between the Vrije Universiteit and IBM, in which experts teach you all about cognitive computing and IBM Watson. You will learn how this artificial intelligence system is capable of answering questions from big data, and work in a joint team of Computer Science and Business Students with some of the latest technology.
Have you ever wondered how we could provide tourists in Amsterdam with the best experience? Now is your chance to develop ideas, business cases and real prototypes of Watson to answer all questions tourists have. Visit the course description page to find out all the details.
Today Lora Aroyo presented the first lecture of the Watson Innovation course at the Vrije Universiteit. The topic of the lecture was Cognitive Computing, IBM Watson and looking inside the mind of Watson. There was a high attendance of motivated bachelor and master students with various backgrounds, such as artificial intelligence, computer science, business administration, business analytics and information sciences. We are looking forward to see them develop their ideas with Watson.
As part of the Watson Innovation course at the Vrije Universiteit Amsterdam, I presented a lecture on crowdsourcing ground truth data through human computation. In the lecture I explained the need of cognitive systems for large amounts of annotated data, and how the wisdom of the crowd should be used to gather this data with the CrowdTruth methodology.
Today the students of the first Watson Innovation course by the Vrije Universiteit Amsterdam and IBM Netherlands presented their group work at the VU Intertain Lab. Representatives from Amsterdam Marketing and IBM Netherlands were present to evaluate the ideas, applications and business plans of the groups. The groups have been working on their Watson powered apps since last November, using the Watson Engagement Advisor and IBM Bluemix. The most interesting project groups will be selected to present their work again next Friday at IBM.
On Friday 22 January Gerard Smit (CTO for IBM Belgium, Netherlands, Luxembourg) and Prof. Hubertus Irth (Vice Dean and Research Director of the Vrije Universiteit Faculty of Earth and Life Sciences and Faculty of Sciences) officially launched the collaboration between the IBM Collaborative Innovation Center (CIC) and the VU Faculty of Sciences. In the event students of the Watson Innovation course pitched their projects to a mixed crowd of students, scientists, engineers and business clients.
In the Watson Innovation course, students used Watson to answer questions about Amsterdam, for which Amsterdam Marketing provided the data and use case. The app LocalBuddy was selected as winner, and the students received a prize for their achievements by Amsterdam Marketing.
Recently the CrowdTruth team got a paper accepted at ICT Open 2016. As part of this upcoming conference, I visited a masterclass on scientific poster design at NWO. The class was given by two professional designers.
The most important thing in your poster is having a clear message. This can be achieved by creating a visual focus. This means that you should not give all images the same size, but guide the reader visually with placement and size of text and images. You have to be able to read the main message from far away and can include the fine details smaller for when the reader is up close. In order to achieve this, there should only be one main focus point to start from.
After having a starting point, there should be a clear hierarchy throughout the poster. The amount of levels of information should be reduces as much as possible, for instance four or five maximum. Most of the content from your paper is not suitable for the poster, only use the most suitable parts, and optionally include more text with details using a small font size at the bottom. Organize the message systematically by using a grid so that all elements are aligned along this grid.
The typography is another very important but also often forgotten aspect of poster design. Choose one proper typography that is well readable and has enough options to variate in size and style. Though, try to minimize the differences in font size, matching the hierarchy structure of the content. Write easy to read sentences but make sure the lines are not too short or long to improve the readability.
The colors of the poster are also an important aspect. Do not use a picture or image with different colors behind a text, it usually makes it too difficult to read. Applying a drop shadow to solve this is not a good solution. Try to never use shadows. Instead, focus on having a high contrast between the text and background color.
For using images and graphics, apply the same rules as for text color. Choose the most important image and decide if it communicates with your audience. It is better to choose one powerful image than a lot of random images. The chronological order of the poster can be changed by positioning the main thing in an unusual position, but then this focus point and the continuing hierarchy must be very clear.
Finally, it is best with scientific posters to just put all logos in a clear line at the bottom in a color bar. They could also be placed vertically, although this is less common and tends to take up more space. When in doubt, just put something big in the poster to get the attention of the audience. Make the poster stand out from the 200 other ones in the same room.
Today we released version 2.0 of the CrowdTruth framework. In the update the data model of the platform is changed, so that data and crowdsourcing results can be managed and reused more easily. This allows for several new features that have been integrated, such as project management and permissions. Users can create projects and share their crowdsourcing jobs within these projects. The media search page has been updated to accommodate any type of data, where you can search through the media in the platform. Another improvement to the platform is the automatic setup of new installations. This makes it easier for new users to get started straight away. You can find a list of the changes in the change log. Try out the platform and get started!
Today I visited the Guide to the network society conference at the Vrije Universiteit Amsterdam. I attended a session on the digitization of cities, in which it became clear to me that cities are going to a major change. Due to digital communication and e-commerce, the need for a main city center with shops and facilities like banks has decreased drastically. Because of this, the way we look at urban development is changing.
An interesting talk was given by Eduardo Diaz on the work of GeoCraft. In this research minecraft is used as a tool for children to gather ideas on how do develop their new neighborhood. This can help the municipalities and government agencies better develop urban areas. I see this as a sort of gamified crowdsourcing, with the interesting aspect that rather than developing their own platform they used a game that many children already play. Using an existing platform can be a valuable approach for our crowdsourcing research. Though, limitations can arise with the functionality and design of such platform. Nevertheless, there are many games and forums where enthusiasts share their knowledge to form a sort collective intelligence. I believe this can be a valuable source of information for government agencies, which will become more important in the future.
The second session was about the success and doom scenarios of intelligent computers. Guszti Eiben, Zoltán Szlávik and Noel Sharkey each presented both their hypothetical success story and the worst case outcome of the recent developments in artificial intelligence and robotics. Guszti presented his view on the possibility of killer robots and the ethical responsibilities we have as researchers. From his work on evolutionary computing, he learned it is essential that in the future we allow robots to reproduce only in a centralized system. This allows the evolution of robotics to be stopped by disabling the central replication, in the case it were ever to become hostile to humanity. Zoltan made an interesting statement that it is not intelligent robots that threaten the future of humanity, but it is humans that will try to abuse power with these robots. Nevertheless his experience is that robots are already have a positive contribution to society, for instance in education and helping the elderly.
On the 22nd of March we presented our latest work on CrowdTruth at the ICT.OPEN 2016 conference. We are happy to announce that our poster received the best poster award in the Human and the Machine track. Furthermore, Anca Dumitrache gave a presentation and pitched our poster which resulted in the 2nd prize for best poster of the conference. It is a good signal that from the almost 200 posters the importance of the CrowdTruth initiative was recognized.
The aim of the ControCurator project is for modern information access systems to discover and understand controversial topics and events. This is done by 1) bringing together different types of crowds: niches of experts, lay crowds and engaged social media contributors; and 2) using machines in a joint active learning workflow for real-time and offline creation of adequate training data. The ControCurator system will consist of two end-user applications: a Controversy Barometer for identifying controversial claims in medical forums, and an Event Blender for summarization of high-profile and catastrophic events in broadcast news & social media. Both systems use the ControCurator platform for curating the data.
The ControCurator projects extends and validates the work on the Accurator (a SealincMedia project) by adding (non-expert) crowdsourcing annotations from CrowdTruth. Additionally, event interpretations will be added that are derived from the analysis and mining of user-generated data on social media through Crowdynews. These event interpretations will also be extended with event interpretations derived from analysis and mining of broadcast news through MediaNow. This will allow to expand even further the range of interpretations on specific topics and events with the media perspective, i.e. how news media manages catastrophic events, detecting controversial topics and events. Finally, these perspectives and interpretations will all be combined in joint temporal summarization of controversial streaming broadcast news events, to enable for user feedback to be incorporated in search and access algorithms.
I presented my doctoral consortium paper titled “Exploiting disagreement through open ended tasks for capturing interpretation spaces” at the PhD Symposium of ESWC 2016.
An important aspect of the semantic web is that systems have an understanding of the content and context of text, images, sounds and videos. Although research in these fields has progressed over the last years, there is still a semantic gap between data available of multimedia and metadata annotated by humans describing the content. This research investigates how the complete interpretation space of humans about the content and context of this data can be captured. The methodology consists of using open-ended crowdsourcing tasks that optimize the capturing of multiple interpretations combined with disagreement based metrics for evaluation of the results. These descriptions can be used meaningfully to improve information retrieval and recommendation of multimedia, to train and evaluate machine learning components and the training and assessment of experts.
From May 29th until June 2nd 2016, the 13th Extended Semantic Web Conference took place in Crete, Greece. CrowdTruth was presented by Oana Inel presenting her paper “Machine-Crowd Annotation Workflow for Event Understanding across Collections and Domains” and by Benjamin Timmermans presenting his paper “Exploiting disagreement through open-ended tasks for capturing interpretation spaces”, both in the PhD Symposium.
The Semantic Web group at the Vrije Universiteit Amsterdam was very well represented, with plenty of papers during the workshops and the conference. The paper on CLARIAH by Rinke, Albert, Kathrin among others won the best paper award at the Humanities & Semantic Web workshop. Here are some of the topics and papers that we found interesting during the conference.
EMSASW: Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web
In the Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web a keynote talk was given by Hassan Saif titled “Sentiment Analysis in Social Streams, the Role of Context and Semantics”. He explained that sentiment analysis is nothing more than extracting the polarity of an opinion. Through the Web 2.0 the sharing of opinions has become easier, increasing the potential of sentiment analysis. In order to find these opinions first opinion mining has to be performed, which is an integral part of sentiment analysis. Hassan compared several semantic solutions for sentiment analysis: SentiCircles which does not rely on the structure of texts but semantic representations of words in a context-term vector; Sentilo which is an unsupervised domain-independent semantic framework for sentence-level sentiment analysis; sentic computing, a multi-disciplinary tool for concept-level sentiment analysis that uses both contextual and conceptual semantics of words and can result in high performance on well structured and formal text.
Jennifer Ling and Roman Klinger presented their work titled “An Empirical, Quantitative Analysis of the Differences between Sarcasm and Irony”. They explained the differences between irony and sarcasm quite clearly. Irony can be split up into verbal irony which is the use of words for a meaning other than the literal meaning, and situational irony which is a situation where things happen opposite of what is expected. They made clear that sarcasm is ironic utterance, designed to cut or give pain. It is nothing more than a subtype of verbal irony. In tweets, they found that ironic and sarcastic tweets contain significantly less sentences than normal tweets.
PhD Symposium
Ghiara Ghidini and Simone Paolo Ponzetto organized a very nice PhD Symposium. They took care to assign for each student mentors that work in related domains and this made their feedback highly relevant and valuable. In this sense, we would like to thank to our mentors Chris Biemann, Christina Unger, Lyndon Nixon and Matteo Palmonari for helping us to improve our papers and for providing feedback during our presentations.
It was very nice to see that events present a high interest in the semantic web community. Marco Rovera presented his Phd proposal “A Knowledge-Based Framework for Events Representation and Reuse from Historical Archives” that aims to extract semantic knowledge from historical data in the context of events and make them available for different applications. It was nice to see that projects just as Agora and the Simple Event Model (SEM), developed at VU Amsterdam were mentioned in his work.
Another very interesting research project on the topic of human computation and crowdsourcing in order to solve problems that are still very difficult for computers was presented by Amna Basharat, “Semantics Driven Human-Machine Computation Framework for Linked Islamic Knowledge Engineering“. She envisioned hybrid human-machines workflows, where the skills and knowledge background of crowds and experts, together with automated approaches aim to improve the efficiency and reliability of semantic annotation tasks in specialized domains.
Vocabularies, Schemas and Ontologies
Céline Alec, Chantal Reynaud and Brigitte Safar presented their work “An Ontology-driven Approach for Semantic Annotation of Documents with Specific Concepts”. This is a collaboration with the weather company, where they use machine learning to classify things you can but also cannot do at a venue. This results in both positive and negative annotations. In order to achieve this, domain experts manually annotated documents and target concepts as either positive or negative. These target concepts were based on an ontology on tourist destinations with descriptive classes.
Open Knowledge Extraction Challenge
This year, the Open Knowledge Extraction Challenge was composed of 2 tasks and 2 submissions were selected for each of the tasks.
Task 1: Entity Recognition, Linking and Typing for Knowledge Base population
Task 2: Class Induction and entity typing for Vocabulary and Knowledge Base enrichment
Semantic Sentiment Analysis Challenge
This challenge consisted of two tasks, one for polarity detection of 1m amazon reviews in 20 domains, and one on entity extraction of 5k sentences in two domains.
IN-USE AND INDUSTRIAL TRACK
Mauro Dragoni presented his paper “Enriching a Small Artwork Collection through Semantic Linking”. A very nice project that highlights some of the issues that small museums and small museums collections encounter: data loss, no exposure, no linking to other collections, no multilinguality. One of the issues that they identified, poor linking to other collections is one of the main goals of our DIVE+ project&system – creating an event-centric browser for linking and browsing across cultural heritage collections. Working with small or local museums is very difficult due to poor data quality, quantity and data management. Attracting outside visitors is also very cumbersome since they have no real exposure and collection owners need to translate the data in multiple languages. As part of the Verbo-Visual-Virtual project, this research investigates how to combine NLP with Semantic Web technologies in order to improve the access to cultural information.
Rob Brennan presented the work on “Building the Seshat Ontology for a Global History Databank”, which is a collection of expert-curated body of knowledge about human history. They used an ontology to model uncertain temporal variables, and coding conventions in a wiki-like syntax to deal with uncertainty and disagreement. This allows each expert to define their interpretation of history. Different types of brackets are used to indicate varying degrees of certainty and confidence. However, in the tool they do not show all the possible values, just the likely ones. Three graphs were used for this model: the real geospatial data, the provenance and the annotations. Different user roles are supported in their tool, which they plan to use to model trust and the reliability of their data.
NATURAL LANGUAGE PROCESSING AND INFORMATION RETRIEVAL
In the paper “Towards Monitoring of Novel Statements in the News” Michael Färber stated that the increasing amount of information that is currently available on the web makes it imperative to search for novel information, and not only relevant information. The approach extracts novel statements in the form of RDF triples, where novelty is measured with regard to an existing KB and semantic novelty classes. One of the limitations of the system, that is considered as future work, is the fact that the system does not consider the timeline. Old articles could be considered novel if their information is not in the KB.
As a side note, we also consider novelty detection an extremely relevant task given the overwhelming amount of information available, and we made the first steps in tackling this problem by combining NLP methods and crowdsourcing (see Crowdsourcing Salient Information from News and Tweets, LREC 2016).
The paper “Efficient Graph-based Document Similarity” by Christian Paul, Achim Rettinger, Aditya Mogadala, Craig Knoblock and Pedro Szekely deals with assessing the similarity or relatedness among documents. They rank documents based on their relevance/similarity by first performing a search for surface forms of words in the document collection and then looking for co-occurrences of words in documents. They integrate semantic technologies (DBpedia, Wikidata, xLisa) to solve problems arising due to language ambiguity: dealing with heterogenous data (news articles, tweets), poor or no metadata available for images, videos among others.
Amparo E. Cano presented the work on “Semantic Topic Compass – Classification based on Unsupervised Feature Ambiguity Gradation”. For classification they used lexical features such as ngrams, entities and twitter features, and also semantic features from dbpedia. The feature space of a topic is semantically represented under the hypothesis that words have a similar meaning if they occur in a similar context. Related words for a given topic are found using wikipedia articles. They found that enriching the data with semantic features improved the recall of the classification. For evaluation three annotators classified the data, where data on which they did not agree was removed from the dataset.
SEMANTIC DATA MANAGEMENT, BIG DATA, SCALABILITY
“Implicit Entity Linking in Tweets” by Sujan Perera, Pablo Mendes, Adarsh Alex, Amit Sheth and Krishnaprasad Thirunarayan – is a new approach of linking implicit entities by exploiting the facts and the known context around given entities. To achieve this, they use the temporal factor to disambiguate entities that are present in tweets, i.e., identify domain entities that are relevant at the time t.
Keynotes
On Tuesday, Jim Hedler gave a keynote speech titled “Wither OWL in a knowledge-graphed, Linked-Data World?”. The topic of the talk was the question whether OWL is dead or not. In 2010 he claimed that semantics were coming to search. Some of the companies back then like Siri had success, but many did not. SPARQL has been adopted in the supercomputing field, but they are not yet a fan of RDF. Many large companies are also using semantic concepts, but not OWL. They are simply not linking their ontologies. Schema.org is now used in 40% of google crawls. It is simple, and this is good because it is used in 10 billion pages. It’s simplicity keeps the use consistent.
Ontologies and owl are like sauron’s tower. If you let one inconsistency in, it may fall over completely. The RDFS view is different: it does not matter if things mean different things, it is jut about linking things together. In the Web 3.0 there are many use cases for ontologies in web apps at web scale. There is a lof of data but few semantics. This explains why RDFS and SPARQL are used but not why OWL is not. The problem is that we cannot talk about the right things in OWL.
On Thursday, Eleni Pratsini – Lab Director, Smarter Cities Technology Center, IBM Research – Ireland had a keynote on “Semantic Web in Business – Are we there yet?”. Her work focuses on advancing science and technology in order to improve the overall cities’ sustainability. Applying semantic web in smart cities could be the main way to understand the city’s needs and further empowering it take smart decisions over the population and the environment.
We both pitched our doctoral consortium papers at the minute of madness session and presented it in the poster session. You can read more about Oana’s presentation here, and Benjamin’s presentation here.
By Oana Inel and Benjamin Timmermans
From 2 to 16th of July we organized the Big Data in Society Summerschool at the Vrije Universiteit Amsterdam. As part of our Collaborative Innovation Center with IBM, we presented an introduction of the technical and theoretical underpinnings of IBM Watson and discussed the use of big data and implications for society. We looked at examples of how the original Watson system can be adapted to new domains and tasks, and presented the CrowdTruth approach for gathering training and evaluation data in this context. The participating students, which ranged from bachelor to PhD level, said they learned a lot from the lectures and found the practical hands-on sessions very useful.
Brainstem tumors are a rare form of childhood cancer for which there is currently no cure. The Semmy Foundation aims to increase the survival of children with this type of cancer by supporting scientific research. The Center for Advanced Studies at IBM Netherlands is supporting this research by developing a cognitive system that allows doctors and researchers to quicker analyse MRI-scans and better detect anomalies in the brainstem.
In order to gather training data, a crowdsourcing event was held at the festival Lowlands, which is a 3-day music festival that took place from 19-21 August 2016 and welcomed 55k visitors. At the science fair, IBM had a booth that hosted both this research and showcase of the Weather stations of the Tahmo project with TU Delft.
In the crowdsourcing task, the participants were asked to draw the shape of the brainstem and tumor in an MRI scan. Gathering data on whether a particular layer of a scan contains the brainstem and determining its size should allow a classifier to recognize the tumors. Furthermore, the annotator quality can be measured with the CrowdTruth methodology by analysing the precision of the edges that were drawn in relation to their alcohol and drug use that we collected. The hypothesis is that people under influence can still make valuable contributions, but that these are of lower quality than sober people. This may make the reliability of online crowd workers more clear, because it is unknown under what conditions they make their annotations.
The initial results in the heatmap of drawn pixels give an indication of the overall location of the brainstem, but further analysis will follow on the individual scans in order to measure the worker quality and generating 3d models.
Today I gave a talk in our Weekly AI meeting on the topic of ControCurator. This is a project that I am currently working on, which has the goal to enable the discovery and understanding of controversial issues and events by combining human-machine active learning workflows.
In the talk I explained the issue of defining the space of a controversy, and how this relates to for instance wicked problems. You can see the slides below.
