Welcome to the NTCIR-13 WWW Task!


NTCIR-13 WWW is an ad hoc Web search task, which we plan to continue for at least three rounds (NTCIR 13-15). Information access tasks have diversified: currently there are various novel tracks/tasks at NTCIR, TREC, CLEF etc. This is in sharp contrast to the early TRECs where there were only a few tracks, where the ad hoc track was at the core. But is the ad hoc task a solved problem? It seems to us that researchers have moved on to new tasks not because they have completely solved the problem, but because they have reached a plateau. Ad hoc Web search, in particular, is still of utmost practical importance. for users, but they are black boxes. We believe that IR researchers should continue to study and understand the core problems of ranked retrieval and advance the state of the art.


The main task of WWW is a traditional ad hoc task. We will also consider a session-based subtask. Pooling and graded relevance assessments will be conducted as usual. After releasing the evaluation results, we will work with participants to conduct an organized failure analysis, following the approach of the Reliable Information Access workshop. We believe that progress cannot be achieved if we just keep looking at mean performance scores. More information is provided in the following figure.


Tentative Schedule


  • Jul-Aug 2016  Corpora released to registered participants
  • Aug-Sep 2016  Designing and constructing topics
  • Oct 2016-Jan 2017  User behavior data collected for the topics
  • Feb-Mar 2017  User behavior data released to registered participants
  • Apr 2017  Task registration due
  • May 2017  Topics released; runs received
  • Jun-Jul 2017  Relevance assessments
  • Sep 1, 2017 Results and Draft Task overview released to participants
  • Oct 1, 2017 Participants’ draft papers due
  • Nov 1, 2017 All camera ready papers due
  • Nov 2017 Pre-NTCIR-13 WWW Workshop on Failure Analysis in Beijing
  • Dec 2017 NTCIR-13 Conference



Participants can…

  • Easily participate in this standard ad hoc task, and monitor improvements over 4.5 years (3 NTCIR rounds).
  • Leverage user behavior data + new Sogou corpus and query log
  • Discover what cannot be discovered in a single-site failure analysis


To participate in the 13th NTCIR WWW task, please read How to participate NTCIR-13 task and register via NTCIR-13 online registration page.




If any questions, feel free to contact the organizers by ntcirwww@list.waseda.jp !

Task Design


  • An ad hoc web search task, which we plan to continue for at least three rounds (NTCIR 13-15).
  • Some user behavior data will be provided to improve search quality.
  • Evaluation measures will also take into account user behavior, along with traditional ranked retrieval measures.
  • English and Chinese subtasks for NTCIR-13 (+Japanese for NTCIR-14)
  • Collective failure analysis at a pre-NTCIR-13 workshop



  • For English Subtask, we will use ClueWeb-12 Category-B dataset.
  • For Chinese Subtask, we are excited to use the latest SogouT-16, which is the updated version of SogouT-12. SogouT-16 contains more than 1.2 billion Webpages (about 90 TB before compression). Don’t worry about the indexing, we will provide a smaller “Category-B” version.
  • For both of the two subtasks, a baseline ranking and corresponding Webpages will be released to help you focus on your own ranking algorithm.
  • In the near future, the updated  SogouQ-16 (querylog) will also be public to promote research!



to be announced soon.



to be announced soon.

© copyright 2016. All rights reserved.