Challenge Website: https://github.com/diegoref/ESWC-CLSA

General Chair

- Fabien Gandon (Inria, Sophia Antipolis, France)

Challenge Coordinators

- Elena Cabrio (Inria, Sophia Antipolis, France)

- Milan Stankovic (SEPAGE, Paris, France)

Challenge Chairs

- Mauro Dragoni, FBK, (Italy)

- Valentina Presutti, CNR STLAB Laboratory (Italy)

- Diego Reforgiato Recupero, CNR STLAB Laboratory (Italy) 

Important Dates

• March 27, 2015, 23:59 (Hawaii time): Challenge Paper Submission deadline
• April 9, 2015, 23:59 (Hawaii time): Notification
• April 24, 2015, 23:59 (Hawaii time): Camera-ready papers deadline
• May 25, 2015, 23:59 (Hawaii time): Test data set published
• May 27, 2015, 23:59 (Hawaii time): Communicate to the Challenge Track Chairs the final results of the evaluation
• May 31 - June 4, 2015: The Challenge takes place at ESWC-15

Motivation and Objectives

As the Web rapidly evolves, Web users are evolving with it. In an era of social connectedness, people are becoming increasingly enthusiastic about interacting, sharing, and collaborating through social networks, online communities, blogs, Wikis, and other online collaborative media. In recent years, this collective intelligence has spread to many different areas, with particular focus on fields related to everyday life such as commerce, tourism, education, and health, causing the size of the Social Web to expand exponentially.

The opportunity to automatically capture the opinions of the general public about social events, political movements, company strategies, marketing campaigns, and product preferences has raised growing interest within the scientific community, leading to many exciting open challenges, as well as in the business world, due to the remarkable benefits deriving from marketing prediction. The distillation of knowledge from such a large amount of unstructured information is an extremely difficult task, as the contents of today’s Web are perfectly suitable for human consumption, but remain hardly accessible to machines. Mining opinions and sentiments from natural language, involves a deep understanding of most of the explicit and implicit, regular and irregular, syntactical and semantic rules proper of a language. Existing approaches mainly rely on parts of text in which opinions and sentiments are explicitly expressed such as polarity terms, affect words and their co-occurrence frequencies. However, opinions and sentiments are often conveyed implicitly through latent semantics, which make purely syntactical approaches ineffective. This issue offers a research opportunity and an exciting challenge to the Semantic Web community. In fact, concept-level sentiment analysis aims to go beyond a mere word-level analysis of text and provide novel approaches to opinion mining and sentiment analysis that allow a more efficient passage from (unstructured) textual information to (structured) machine-processable data, in potentially any domain.

Concept-level sentiment analysis focuses on a semantic analysis of text through the use of web ontologies, semantic resources, or semantic networks, allowing the identification of opinion data which with only natural language techniques would be very difficult. By relying on large semantic knowledge bases, concept-level sentiment analysis steps away from blind use of keywords and word co-occurrence count, but rather relies on the implicit features associated with natural language concepts. Unlike purely syntactical techniques, concept-based approaches are able to detect also sentiments that are expressed in a subtle manner, e.g., through the analysis of concepts that do not explicitly convey any emotion, but which are implicitly linked to other concepts that do so.

Systems must have a semantics flavor (e.g., by making use of Linked Data or known semantic networks within their core functionalities) and authors need to show how the introduction of semantics can be used to obtain valuable information, functionality or performance. Existing natural language processing methods or statistical approaches can be used too as long as the semantics plays a main role within the core approach (engines based merely on syntax/word-count will be excluded from the competition).

Target Audience

The Challenge is open to everyone from industry and academia.

Tasks

The Concept-Level Sentiment Analysis Challenge is defined in terms of different tasks. The first task is elementary whereas the others are more advanced. In order to be accepted for the challenge, each system has to deal with the first task.

- Elementary Task: Polarity Detection

The basic task of the challenge is binary polarity detection. The proposed semantic opinion-mining engines will be assessed according to precision, recall and F-measure of detected polarity values (positive OR negative) for each review of the evaluation dataset. The problem of subjectivity detection is not addressed within this Challenge, hence participants can assume that there will be no neutral reviews.

- Advanced Task #1: Aspect-Based Sentiment Analysis

The output of this Task will be a set of aspects of the reviewed product and a binary polarity value associated to each of such aspects. So, for example, while for the Elementary Task an overall polarity (positive or negative) is expected for a review about a mobile phone, this Task requires a set of aspects (such as ‘speaker’, ‘touchscreen’, ‘camera’, etc.) and a polarity value (positive OR negative) associated with each of such aspects. Engines will be assessed according to both aspect extraction and aspect polarity detection using precision, recall and F-measure similarly as performed during the first Concept-Level Sentiment Analysis Challenge held during ESWC2014 and re-proposed at SemEval 2015 (http://www.alt.qcri.org/semeval2015/task12/).

- Advanced Task #2: Frame entities Identification

The Challenge focuses on sentiment analysis at concept-level. This means that the proposed engines must work beyond word/syntax level, hence addressing a concepts/semantics perspective. This task will evaluate the capability of the proposed systems to identify the objects involved in a typical opinion frame according to their role: holders, topics, opinion concepts (i.e. terms referring to highly polarised concepts). For example, in a sentence such as The mayor is loved by the people in the city, but he has been criticized by the state government (taken from Sentiment Analysis and Opinion Mining, Bing Liu, 2012), an approach should be able to identify that the people and state government are the opinion holders, is loved and has been criticized represent the opinion concepts, The mayor identifies a topic of the opinion. The proposed engines will be evaluated according to precision, recall and F-measure.

Evaluation Dataset

Systems will be evaluated against a testing dataset which will be released after a first-round of evaluation during the Conference. Participants are recommended to train and/or test their own systems using the Blitzer Dataset (http://cs.jhu.edu/%7Emdredze/datasets/sentiment). Precision, recall, F1-measure for all the tasks will be computed automatically by a tool that will be available for download so that each participant will be able assess their methods and make sure the output produced by their system is in compliance with the input required by the script.

Subjective Evaluation

A subjective evaluation will be performed by the members of the Advisory Board. For each system, reviewers will give a numerical score within the range [1-10] and details motivating their choice. The scores will be given to the following aspects:

• 1. Use of common-sense knowledge and semantics;
• 2. Computational time;
• 3. Graphical interface - including the number of features that is possible to query, usability of the system, appealing of the user interface;
• 4. Innovative nature of the approach including multi-language capabilities.

Objective Evaluation

For systems that can be tuned with different parameters, please indicate a range of up to 5 sets of settings. Settings with the best precision, recall, F-measure will be considered for judgment. The objective evaluation will be performed according to precision, recall, and F-measure analysis.

Judging and Prizes

We propose to award systems based on two criteria judged separately:

    Subjective: the system with the highest average score in items 1-4 above;

    Objective: the system with the highest score in precision, recall and F-measure analysis.

After the first round of evaluation a list of runners up will be defined. A certain number of systems with the highest scores within the subjective evaluation and a number of systems with the highest scores within the objective evaluation will be the finalists and will have to present their work in a conference session. The exact number will depend on the scores they get and on the Conference policy. They will have a slot of approximately 15 minutes. The judges will be present and will evaluate again the systems in more detail. The judges will then meet in private to discuss the entries and to determine the winners.

How to Participate

The following information has to be provided:

• * Abstract: no more than 200 words.
• * Description: It should contain the details of the system, including why the system is innovative, how it uses Semantic Web, which features or functions the system provides, what design choices were made and what lessons were learned. The description should also summarize how participants have addressed the evaluation tasks. Papers must be submitted in PDF format, following the style of the Springer's Lecture Notes in Computer Science (LNCS) series (http://www.springer.com/computer/lncs/lncs+authors), and not exceeding 12 pages in length.
• * Web Access: The application can either be accessible via the web or downloadable. If the application is not publicly accessible, password must be provided. A short set of instructions on how to use the application should be provided as well.

Papers are submitted in PDF format via the challenge's EasyChair submission pages (https://easychair.org/conferences/?conf=eswcclsa2015). 

Please share comments and questions with the challenge mailing list. The organizers will assist you for any potential issues that could be raised.

Mailing List

We invite the potential participants to subscribe to our mailing list in order to be kept up to date with the latest news related to the challenge. 

Mailing list web page is http://groupspaces.com/ESWC2015-CLSA. Users are invited to join it and then post messages using the address ESWC2015-CLSA@groupspaces.com

Program Committee

* Rada Mihalcea, University of North Texas (USA)

* Ping Chen, University of Houston-Downtown (USA)

* Yongzheng Zhang, LinkedIn Inc. (USA)

* Giuseppe Di Fabbrizio, Amazon Inc. (USA)

* Soujanya Poria, Nanyang Technological University (Singapore)

* Yunqing Xia, Tsinghua University (China)

* Rui Xia, Nanjing University of Science and Technology (China)

* Jane Hsu, National Taiwan University (Taiwan)

* Rafal Rzepka, Hokkaido University (Japan)

* Amir Hussain, University of Stirling (UK)

* Alexander Gelbukh, National Polytechnic Institute (Mexico)

* Bjoern Schuller, Technical University of Munich (Germany)

* Amitava Das, Samsung Research India (India)

* Dipankar Das, National Institute of Technology (India)

* Carlo Strapparava, Fondazione Bruno Kessler (Italy)

* Stefano Squartini, Marche Polytechnic University (Italy)

* Cristina Bosco, University of Torino (Italy)

* Paolo Rosso, Technical University of Valencia (Spain)

Additional information could be found at: https://github.com/diegoref/ESWC-CLSA