Home » Call for Challenge: Schema-agnostic Queries over Large-schema Databases (SAQ-2015)

Call for Challenge: Schema-agnostic Queries over Large-schema Databases (SAQ-2015)

Challenge Website: https://sites.google.com/site/eswcsaq2015/

General Chair

- Fabien Gandon (Inria, Sophia Antipolis, France)

Challenge Coordinators 

- Elena Cabrio (Inria, Sophia Antipolis, France)

- Milan Stankovic (SEPAGE, Paris, France)

Challenge Chairs

- Andre Freitas, University of Passau, Germany, DERI/Insight, Ireland 

- Christina Unger, Bielefeld University, Germany

Important Dates

  • February 16, 2015: Final version of challenge training set
  • March 23, 2015: Submission of the intent to participate
  • March 25, 2015: Challenge paper submission deadline (extended)
  • April 9, 2015: Paper results, report of the task results, and Test data set published
  • April 24, 2015: Camera-ready papers deadline

The Challenge in a Nutshell

To create a query mechanism that semantically matches schema-agnostic user queries to knowledge base elements.

Goal

To support easy querying over complex databases with large schemata, relieving users from the need to understand the formal representation of the data.

Relevance

The increase in the size and in the semantic heterogeneity of database schemas are bringing new requirements for users querying and searching structured data. At this scale it can become unfeasible for data consumers to be familiar with the representation of the data in order to query it. At the center of this discussion is the semantic gap between users and databases, which becomes more central as the scale and complexity of the data grows. Addressing this gap is a fundamental part of the Semantic Web vision. Schema-agnostic query mechanisms aim at allowing users to be abstracted from the representation of the data, supporting the automatic matching between queries and databases. This challenge aims at emphasizing the role of schema-agnosticism as a key requirement for contemporary database management, by providing a test collection for evaluating flexible query and search systems over structured data in terms of their level of schema-agnosticism (i.e. their ability to map a query issued with the user terminology and structure, mapping it to the dataset vocabulary). The challenge is instantiated in the context of Semantic Web datasets.

Submissions

SAQ-2015 invites submissions of papers targeting the following (or related) topics:

  • - Schema-agnostic query approaches and systems.
  • - Demonstrations of schema-agnostic query approaches.
  • - Usability and user-interface aspects for schema-agnostic queries.
  • - Formal models for schema-agnostic queries.
  • - Analysis of the semantic aspects involved in query-database matching.
  • - Evaluation methodologies for schema-agnostic queries.
  • - Entity search & schema-agnostic queries.
  • - Natural Language Interfaces & schema-agnostic queries.

About the Challenge

The challenge aims at providing an evaluation test collection for schema-agnostic query mechanisms, focsuing on Semantic Web scenarios. The large-schema and semantically heterogeneous nature of Semantic Web datasets brings schema-agnosticism as a fundamental data management concern for this community. The test collection supports the quantitative and qualitative evaluation of degree of schema-agnosticism of different approaches. Since addressing schema-agnostic queries is dependent on semantic approaches which need to cope with different types of semantic matching between query and dataset, the test collection explores different categories of semantic phenomena involved in the challenge of matching schema-agnostic queries. Each query is categorized according to the semantic mapping types. This categorization supports a fine-grained qualitative and quantitative interpretation of the evaluation results. Participating systems (semantic query/search engines) will receive a set of schema-agnostic queries over DBpedia 3.10 data. The task is to return the correct answers for the query associated with the right interpretation of the query under the evaluation dataset. Two categories of schema-agnostic queries (tasks) are available: 

Schema-agnostic SPARQL query

Consists of schema-agnostic queries following the syntax of the SPARQL standard. The syntax and semantics of operators are maintained, while different terminologies are used.

* Example I:

  • SELECT ?y {
  •      BillClinton hasDaughter ?x .
  •      ?x marriedTo ?y .
  • }

which maps to the following SPARQL query in the dataset vocabulary:

 

* Example II:

  • SELECT   ?x {
  •          ?x isA book .
  •          ?x by William_Goldman .
  •          ?x has_pages ?p .
  •          FILTER (?p > 300)
  •          }

which maps to the following SPARQL query in the dataset vocabulary:

Schema-agnostic keyword query

Consists of schema-agnostic queries using keyword queries. In this case the syntax and semantics of operators are different from the SPARQL syntax.

* Example I:

"Bill Clinton daughter married to"

* Example II:

"Books by William Goldman with more than 300 pages"

Evaluation

The challenge provides a gold standard with the correct answers for each query. Queries will be issued over DBpedia 3.10. A training dataset consisting of 25 queries will be made available for the participants. 100 queries will be used to evaluate the systems. In order to participate in the challenge, each system should submit the results in the format proposed by the challenge. The organizers will then automatically calculate precision, recall, mean reciprocal rank for each query and the associated averages. Participants are required to submit their query execution time, dataset enrichment time, and user-interaction disambiguation effort.

Prizes

A financial prize (to be announced) will be given to the authors of the best-performing system and/or most original contribution.

How to Participate

High quality original papers should be submitted via EasyChair: https://easychair.org/conferences/?conf=saq2015 Long papers should have at maximum 12 pages, and short papers 6 pages. All submissions must conform with the LNCS format (http://www.springer.de/comp/lncs/authors.html).

Program Committee

Stefan Bischof, WU Economics/Siemens, Austria

Edward Curry, DERI/Insight, Ireland 

Sebastian Walter, Bielefeld University, Germany

Siegfried Handschuh, University of Passau, Germany 

Pierpaolo Basile, University of Bari, Italy

Axel Ngonga, University of Leipzig, Germany

Andre Freitas, University of Passau, Germany 

Christina Unger, Bielefeld University, Germany

Stephane Campinas, DERI/Insight, Ireland

Honghan Wu, University of Aberdeen, UK 

Vanessa Lopez, IBM Research, Ireland

Ahmed El-Roby, University of Waterloo, Canada

Tran Thanh, Graphinder/San Jose State University, USA

Pablo Mendes, IBM Research, USA

Francesco Guerra, University of Modena, Italy

Elena Cabrio, INRIA Sophia-Antipolis, France

Souleiman Hasan, DERI/Insight, Ireland 

Saeedeh Shekarpour, Bonn University, Germany

 

Additional information is available at: https://sites.google.com/site/eswcsaq2015/