Guidelines

Research groups or individuals interested in participating in the track should do the following:

1. Register (see http://www.clef-campaign.org). Upon registration, every participant will receive instructions to download the appropriate document collections from the CLEF FTP site.

2. Email the track organisers (Doug Oard and Gareth Jones) indicating your wish to participate, and the languages (user and document languages) that will be used in your experiment. Once registration for CLEF is confirmed, participants will receive instructions to proceed.

Desigining Official Runs

We will accept some maximum number runs from each site for official scoring. Sites are encouraged to score additional runs locally using the relevance judgments that will be released with the official results if there are additional contrastive conditions that they wish to explore.

In keeping with the goals of CLEF, the use of non-English queries is encouraged.

Some submitted runs will be used to create pools for relevance assessment (i.e., "judged runs"); others will be scored using those judgments (i.e., "unjudged runs"). The number of judged runs for each site will depend on the total number of submissions received. Participants are requested to prioritize their runs in descending order of preference for inclusion in the pool in such a way that selecting the runs assigned the highest priority will result in the most diverse judgment pools. The required (automatic monolingual TD ASR) run need not be assigned a high priority if a different assignment would result in greater pool diversity.

Which Topics to Run

Test topics will be selected to contain enough known relevant segments (or playback points in Czech collection) to avoid instability in the evaluation measures, and few enough known relevant segments (or playback points) that we believe the judged set to be reasonably complete. We will report official evaluation results only on the selected evaluation topics.

Submission Format (for at least English collection)

For each run, the top-1000 results will be accepted in the format:

topicid Q0 docno rank score tag

topicid - topic number
Q0 - unused field (the literal 'Q0')
docno - "document" id taken from the DOCNO field of the segment
rank - rank assigned to the segment (1=highest)
score - computed degree of match between the segment and the topic
tag - unique identifier for this run (same for every topic and segment)

Participating sites should each adopt some standard convention for tags that begins with a unique identifier for the site and ends with a unique run identifier. Please try to avoid tags longer than about 10 characters. For example, the University of Ipsilanti might create a tag: UipsASR1b

What to Put in Your README File

Please provide the institution name, name of the contact person at that institution, and complete contact information for that person (email, phone, postal address, and fax) and complete a questionnaire for each run.

Where to Send Your Submission

Runs should be submitted via email to Jianqiang Wang (wangjq at glue dot umd dot edu) in a single compressed file (.zip or .tar.gz). The compressed file should contain:

• Each run in a separate file
• One README with the completed questionnaires for all submitted runs.

Please ensure that the number of completed questionnaires in the README matches the number of runs. Submissions will be acknowledged within 24 hours; participants must ensure that acknowledgment is received!