The Dataset in "Investigating Examination Behavior in Mobile Search" (WSDM '20)

In this page, we provide our dataset used in our paper “Investigating Examination Behavior in Mobile Search” which was accepted by WSDM 2020 as a full paper. This dataset was constructed to support research on mobile search and other information retrieval related technologies. It consists of 60 search tasks and corresponding user behavior data as well as annotations collected in the user study.

Tasks

  • eye_mobile_task.tsv: tasks used in the paper
  • eye_mobile_position.tsv: the information of isVertical, result_type and position (defined as a rectangle with 4 boundaries) of each result .
  • SERP: SERPs used in the paper, which were crawled from Sogou on August 7th, 2019.

EyeTracking

  • FileFolder Name is “{Student ID}” (e.g., S2019110501)
  • FileName is “Task ID” (e.g., 21.csv)
  • Data Field Name:
    • TimeStamp
    • x: position x of eye fixation on the screen
    • y: position y of eye fixation on the screen
    • duration: fixation duration/saccade duration is total duration for current consecutive saccade
    • event: fixation / saccade (saccade is not used in the paper)

UserLog

  • Actions:

    • TASK_BEGIN: the beginning time to start a task
    • TASK_END: the ending time to finish a task
    • SCROLL: scroll on the SERP
    • YCHANGE: the offset of SERP
    • FLING: fling motion
    • SINGLETAPUP: a single tap on the screen
    • LONGPRESS: long press on the screen
    • CLICK_RESULT: click on a result
    • JUMP_TO_PAGE: switch to another page
    • BACK_TO_SERP: switch back to the SERP

  • Format

    For each action, we record its timestamp and the corresponding information such as the position on mobile screen.

Annotation

We collected some annotations, including result relevance, click necessity and result type, by crowdsourcing:

  • eye_mobile_click_necessity.tsv: three-grade click necessity
  • eye_mobile_landingpage_relevance.tsv: four-grade page relevance
  • eye_mobile_result_type.tsv: 18 result types
  • eye_mobile_snippet_relevance.tsv: four-grade snippet relevance

Download

You can download our dataset for your research by clicking here. If you have any problem, please contact us (zhengyk13 at gmail.com).

Citation

If you use this dataset in your research, please add the following bibtex citation in your references. A preprint of this paper can be found here WSDM2020-Zheng.

@inproceedings{zheng2020investigating,
  author = {Zheng, Yukun and Mao, Jiaxin and Liu, Yiqun and Sanderson, Mark and Zhang, Min and Ma, Shaoping},
  title = {Investigating Examination Behavior in Mobile Search},
  booktitle = {Proceedings of the Thirteen ACM International Conference on Web Search and Data Mining},
  series = {WSDM '20},
  year = {2020},
  publisher = {ACM},
  address = {New York, NY, USA}
}