Download M91-1038

files on Friday . However, for your own sake, minimize the accessibility of those files ,
e .g ., put them in a protected directory of someone who is not directly involved i n
system development .
2 .2 GENERATING THE SYSTEM RESPONSE TEMPLATE S
There are 100 texts in tst2-muc3, and the message IDs have the following format :
TST2-MUC3-nnnn . Without looking at the texts, run your system against the file an d
name the output file response-max-tradeoff.tst2 .
You are to run the required test only once -- you are not permitted to make an y
changes to your system until the test is completed . If you get part way through the
test and get an error that requires user intervention, you may intervene only to th e
extent that you are able to continue processing with the NEXT message . You are no t
allowed to back up !
Notes :
1)
If you run short on time and wish to break up tst2-muc3 and run portions of i t
in parallel, that's fine as long
as you are truly running in parallel with a
i .e ., th e
single system or can completely simulate a parallel environment,
systems are identically configured .
You must also be sure to concatenate th e
outputs before submitting them to the scoring program .
2)
No debugging of linguistic capability can be done when the system breaks .
For example, if your system breaks when it encounters an unknown word an d
your only option for a graceful recovery is to define the word, then abor t
processing and start it up again on the next test message .
3)
If you get an error that requires that you reboot the system, you may do so, bu t
you must pick up processing with the message FOLLOWING the one that wa s
being processed when the error occurred . If, in order to pick up -processing a t
that point, you need to create a new version of tst2-muc3 that excludes th e
messages already processed or you need to start a new output file, that's ok . B e
sure to concatenate the output files before submitting them to the scorin g
program .
2 .3 SCORING THE SYSTEM RESPONSE TEMPLATE S
2 .3 .1 SCORING ALL SYSTEM RESPONSES FOR OFFICIAL, REQUIRED TES T
Run the scoring program on the system response templates, using key-tst2 as th e
answer key and entering config .el as the argument to initialize-muc-scorer . (Th e
config file contains arguments to the define-muc-configuration-options function ,
which you will have to edit to supply the proper pathnames) . When you enter th e
scoring program, type "is" so that the score buffer will contain detail table s
(template by template) as well as the final summary table . Save the score buffe r
(*MUC Score Display*) to a file called scores-max-tradeoff .tst2 .
Note : During the interactive scoring, make use of the guidelines (supplie d
separately) for interactively assigning full and partial credit . Also refer to key-tst2 notes (in the ftp directory) for NOSC's comments on how the answer key wa s
generated . See section 5, below, for information on the plans for handling th e
rescoring of results .
B–3