Guidelines

Research groups or individuals interested in participating in the track should do the following:

1. Register (see http://www.clef-campaign.org). Upon registration, every participant will receive instructions to download the appropriate document collections from the CLEF FTP site.

2. Email the track organisers (Doug Oard and Gareth Jones) indicating your wish to participate and the languages (user and document languages) that will be used in your experiment. Once registration for CLEF is confirmed, participants will receive instructions to proceed.

Desigining Official Runs

We will accept a maximum of 5 runs from each site for official scoring. Sites are encouraged to score additional runs locally using the relevance judgments that will be released with the official results if there are additional contrastive conditions that they wish to explore.

At least one submitted run should use automatically constructed queries from the English title and description fields of the topics (i.e., an automatic monolingual "TD" run) and an index that was constructed without use of human-created metadata (i.e., indexing derived from some combination of ASRTEXT2003A, ASRTEXT2004A, AUTOKEYWORD2004A1, and AUTOKEYWORD2004A2, including the optional use of synonyms and/or broader terms for one or both of the AUTOKEYWORD fields). The other submitted runs can be created in whatever way best allows the sites to explore the research questions in which they are interested (e.g., comparing monolingual and cross-language, comparing automatic recognition with metadata, or comparing alternative techniques for exploiting ASR results). In keeping with the goals of CLEF, the use of non-English queries is encouraged.

Some submitted runs will be used to create pools for relevance assessment (i.e., "judged runs"); others will be scored using those judgments (i.e., "unjudged runs"). The number of judged runs for each site will depend on the total number of submissions received. Participants are requested to prioritize their runs in descending order of preference for inclusion in the pool in such a way that selecting the runs assigned the highest priority will result in the most diverse judgment pools. The required (automatic monolingual TD ASR) run need not be assigned a high priority if a different assignment would result in greater pool diversity.

Which Topics to Run

Results for all 75 topics should be submitted. 38 of the 75 are from the training collection; pools formed from those will be used to enrich the relevance judgments for that collection. At least 25 evaluation topics will be selected from the remaining 37 topics. These will be selected to contain enough known relevant segments to avoid instability in the mean uninterpolated average precision measure and few enough known relevant segments that we believe the judged set to be reasonably complete. We will report official evaluation results only on the selected evaluation topics.

Submission Format

For each run, the top-1000 results will be accepted in the format:

topicid Q0 docno rank score tag

topicid - topic number
Q0 - unused field (the literal 'Q0')
docno - "document" id taken from the DOCNO field of the segment
rank - rank assigned to the segment (1=highest)
score - computed degree of match between the segment and the topic
tag - unique identifier for this run (same for every topic and segment)

Participating sites should each adopt some standard convention for tags that begins with a unique identifier for the site and ends with a unique run identifier. Please try to avoid tags longer than about 10 characters. For example, the University of Ipsilanti might create a tag: UipsASR1b

What to Put in Your README File

Please provide the institution name, name of the contact person at that institution, and complete contact information for that person (email, phone, postal address, and fax) and complete the following questionnaire for each run:

1. Judgment pool priority (1=highest, 5=lowest)
2. Automatic or Human indexing? (List "Automatic" for indexing based on any combination of ASRTEXT2003A, ASRTEXT2004A, AUTOKEYWORD2004A1, and AUTOKEYWORD2004A2, including the optional use of synonyms and/or broader terms for one or both of the AUTOKEYWORD fields).
3. Automatic or Manual search? (list "Manual" for runs with any human intervention, including any changes to the system implementation after reading the evaluation topics, any manual editing of the contents of the topic fields that are used, and any adjustments made to system output based on human examination of the system results. Bug fixes to the system after examining the topics result in manual runs.
4. Query language used
5. Query fields used
6. Document fields indexed
7. For each keyword field (MANUALKEYWORD, AUTOKEYWORD2004A1, AUTOKEYWORD2004A2):
   • Synonyms used?
   • Number of broader term levels? (zero if not used)

Where to Send Your Submission

Runs should be submitted via email to Ryen White (ryen@umd.edu) in a single compressed file (.zip or .tar.gz). The compressed file should contain:

• Each run in a separate file
• One README with the completed questionnaires for all submitted runs.

Please ensure that the number of completed questionnaires in the README matches the number of runs. Submissions will be acknowledged within 24 hours; participants must ensure that acknowledgment is received!