Task Outline
The NTCIR-11 Math-2 is a shared task for retrieving mathematical formulae in scientific documents. Given a query representing a target formula expressed in a standard XML notation, namely MathML (http://www.w3.org/Math/), the participating systems should return a ranked list of the relevant retrieval units containing a formula matching the query.
Dataset
The target dataset of NTCIR-11 Math-2 consists of about 100,000 scientific articles (in English) that were converted from LaTeX to an XHTML format by the KWARC project (http://kwarc.info/). The dataset will include about 35M math formulae (estimated; including monomial expressions) and can be used for research into topics such as tree structure search, semantic information retrieval using natural language and text analysis, or utilization and linking of mathematical knowledge. Initially, we planned to re-use the NTCIR-10 dataset, except for the changes affecting the new retrieval units. However, we decided to regenerate the dataset using the latest version of the conversion tool (LaTeXML). This will improve the consistency and the quality of math formulae in the dataset. yBased on this, Math-2 dataset will contains documents resampled from the same ArXiv dataset. Also, for NTCIR-11 Math-2, each document in the corpus will be divided into smaller subsections, the relevance of which to a given query should be estimated by the participating systems. The participating systems should return a ranked list of the retrieval units of the documents containing a formula matching the query.
Queries
Given a set of queries, systems must return a ranked list of search results. Total number of queries will be around 50.
A query contains:
- Topic ID
- Query (formula + key words)
-
Description (short description of what a user is looking for)(topics for Math-2 will not contain natural language description) - Narrative (precise description of the user situation and information need and relevancy criteria; used only for assessment and will not be included in the query set delivered to participants)
The task is designed so that all the topics include multiple relevant documents at least a single relevant document in arxiv.org.
For NTCIR-11 Math-2 topic format, please refer
Formats for topics and submissions for NTCIR-11 Math-2 Task with sample topics.
For NTCIR-10 Math Pilot Task Definition, please refer
NTCIR10-math-topics (printed version).
(Note: there may be a change in the query format for Math-2)
Runs
The results should contain compulsory automatic runs using query only field.
Also, we encourage participants to submit manual runs (with manually generated queries).
The results should include the score of the returned retrieval units and may optionally include supporting evidence (formulaID).
For NTCIR-11 Math-2 submission format, please refer
Formats for topics and submissions for NTCIR-11 Math-2 Task.
Relevance Judgment
The frequent 50 retrieval units will be selected from the union of the top 100 1,000 documents from each run and assessed by human reviewers. We plan to apply double relevance assessment for every retrieval unit. After the assessment, the list of documents and the relevance judgment will be returned to the participants. (Note that the evaluation is pooling based; not all the submitted documents will be assessed by the organizers.)
Wikipedia Open Task
- In addition to the regular Math-2 task there is an optional free
Wikipedia subtask that uses the same topic and submission format.
Please refer to the http://ntcir11-wmc.nii.ac.jp for further information, the query and dataset download.
Resources from NTCIR-10 Math Pilot Task
- 100,000 XHTML documents transformed into XHTML+MathML using the LATEXML converter (7.2G).
The NTCIR-10 document set is available from the KWARC Project; if you wish to obtain it, please contact ntcadm-math@nii.ac.jp.
- Annotated Dataset for Math Description Extraction
45 annotated papers to help semantic formula search studies. All the natural language descriptions of mathematical expressions is manually annotated. Also, all the mathematical expressions in the dataset are expressed using MathML Parallel Markup.
For details, please refer to the evaluation documentation for NTCIR-Math Math Understanding Subtask and also http://ntcir-math.nii.ac.jp/ntcir11-math/descext/
- For the distribution of topics and relevance judgment at NTCIR-10 Math Pilot Task, please refer to http://ntcir-math.nii.ac.jp/ntcir11-math/ntcir10-math/datasets-2/