Spoken Dialogue Systems Evaluation

Teams participating in this task should develop a Spoken Dialogue System (SDS) operating in an assigned domain, examples might be: railways time-table enquiries, hotel and/or flight reservation.
Precise requirements including applicative domain, domain-related database (same data will be given to all participants), task specifications and evaluation criteria are provided in the detailed guidelines.
Evaluation of the participating systems will be conducted in the period 10th - 20th September 2009 by a panel of four experts that will directly call the system either on fixed telephony line or by SIP softphone.
Each expert will call each system at least 10 times.

The evaluation, conducted following internationally accepted procedures, will be based on how the system accomplishes the specified task and on the constraints under which the system operates.
Participants will be asked to produce audio recordings of all received calls, which will be used by the referees in the evaluation process. In some cases participants could be required to provide orthographic transcription of some or all produced dialogues.

Audio file format and transcription criteria will be included in the task specification document.

Task materials

• Detailed Guidelines

Data Distribution NEW

Data can be downloaded at this address: http://evalita.fbk.eu/evalita-SDS-database.zip

Organizers
Giuseppe Riccardi (Uni. Trento), Francesco Cutugno (Uni. Napoli "Federico II"), Roberto Pieraccini (Speechcycle, New York) and Morena Danieli (Loquendo, Torino)