Download Test review information and application pack for test publishers

The British
Psychological Society
Promoting excellence in psychology
Test review information
and application pack for
test publishers/distributors
www.psychtesting.org.uk
Contents
Introduction............................................................................................................................
3
Information for first-time submissions/new tests
How do I submit a test for test review? ......................................................................................
3
Checklist for new submissions into the test review process ..........................................................
3
Materials required for submission into the test review process .....................................................
4
Applying for a Certificate of Test Registration for tests previously reviewed ....................................
5
Information relating to renewal of a Certificate of Test Registration or submission for
a test review update
Renewing a Certificate of Test Registration after a five-year cycle, or up to 15 years since ..............
the test review was published when there have been no substantive changes to the test
6
Renewing a Certificate of Test Registration when a test review was published ...............................
15 or more years ago
6
Renewing a Certificate of Test Registration after a five-year cycle when there have been ................
substantive changes to the test since the Certificate of Test Registration was granted
6
Renewing a Certificate of Test Registration inside the standard five-year test registration cycle ......
when there have been substantive changes to the test since the Certificate of Test Registration
was granted
7
Checklist for renewal of a Certificate of Test Registration and submission for a .............................
test review update
7
Flowcharts summarising test review processes............................................................................
8
Appendices
Appendix A: Application Form and Terms and Conditions ............................................................ 10
Appendix B: Factual Information Form for submission into the test review process ........................ 13
Appendix C: Editorial Panel Review Update Pro Forma................................................................ 25
Appendix D: Schedule of Fees for Test Reviews .......................................................................... 38
Appendix E: Renewal of Certificate of Test Registration............................................................... 39
Introduction
The British Psychological Society (BPS) has been operating a test review process for a number of years
and to date has reviewed over 150 psychological tests. In July 2013 the BPS made some changes to its
test review process. This document is aimed both at test publishers and test distributors who are doing
the following:
●
Submitting a specific test into the test review process for the first time.
●
Renewing a Certificate of Test Registration.
●
Applying for an updated test review (and submitting amended or new materials).
Information for first-time submissions/new tests
How do I submit a test for test review?
If you are applying for test review for the first time, or submitting a new test into this process, you will
find a checklist immediately below to work through in order to ensure you submit the required materials
to enter into the test review process.
We recommend a final check of the contents of your parcel before submitting materials for test review
to the BPS Psychological Testing Centre (PTC). Any omissions may impede the registration and review
process or affect the evaluation of the test, so it is important to ensure you submit all relevant materials
outlined below. (Please also note that there is a charge levied of £510 for the later submission of
additional materials when the materials have not been presented to the PTC at the outset of the process.)
Checklist for new submissions into the test review process
●
Step 1 can be completed by an administrator:
Complete Application Form and Terms and Conditions (Appendix A)
●
Step 2 can be completed by an authorised officer designated by the test publisher/distributor:
Complete the editorial panel test review factual information form
●
Step 3 can be completed by an administrator
Ensure all test materials detailed in this document on pages 4 and 5 and all necessary forms
outlined above are submitted with the application for test review.
A quick summary of test materials to submit for entry into the test review process is outlined below.
Four copies of each:
■ Question booklets/answer sheets. If the test is administered online, access for four people to the
online test material is required.
■ Scoring keys and instructions.
■ Norming information.
■ Computer generated reports.
■ Manuals and training notes.
Please note that it is possible to submit some of the materials outlined above in electronic format.
Publishers wishing to do this need to contact the PTC first about how they should go about this.
●
Step 4 can be completed by an administrator
Check relevant fees on the Schedule of Fees in Appendix D
3
Materials required for submission into the test review process
In order to produce a fair and objective test review it is important that our reviewers have all the
necessary information about your instrument. As each instrument and the supporting information are
formatted differently it is difficult to be specific about what is required, however, the following list
should help you ensure that you send the appropriate information. Where both paper and computer or
web-based versions of the instrument exist, access to all forms should be supplied.
Where information is lacking this is likely to be reflected in the final review. Please submit the following
test materials to the PTC to enter in to the test review process.
Please note that submission of four copies of materials is required for a market ready test entering the
test review process. Where an Editorial Panel Review Update is being pursued, only two copies of test
materials are required. Please see page 7.
Instrument:
Please supply four copies of the following:
● copies of question booklet and answer sheet and/or access to computer-based administration
system
● copies of scoring keys and instructions and/or access to computer/bureau scoring and description of
scoring method.
● copies of norming information and/or access to computer/bureau-based norming with description of
process.
● copies of computer-generated reports and/or access to computer/bureau-based report generation.
For each different computer- generated report four different reports should be provided:
1. A report for a typical profile.
2. A report for a profile made up of lowest possible scores (on each scale).
3. A report for a profile made up of mean scores (on each scale).
4. A report for a profile made up for highest possible scores (on each scale).
Alternatively reviewers can be provided with access to the report generator and will create their own
review reports.
Terms of Supply and Factual Information:
● Four copies of test catalogues or other information describing terms of supply, packaging and costs
of materials reports etc including details of any set up costs, licensing fees as well as ongoing costs.
Documentation:
● Four copies of any documentation supplied to test users. This is often structured differently but
would include User Manuals, Technical Manuals and Training Notes where these contained
additional information.
In addition references to any published work relating to the instrument should be submitted.
Where the standard documentation does not cover all of the information required to rate the instrument
against the criteria used in EFPA Review Model then additional information should be supplied to
address these issues. This might include published papers, technical reports, internal reports and client
reports covering such issues as:
● Test development processes.
● Validation work, for example, construct studies, criterion related validity.
● Reliability analyses.
● Sample descriptions for norm groups and other research.
● Equivalence studies, for example, paper and pencil versus computer versions; alternate forms.
● Development report or algorithms for computer-generated reports.
4
Please specify whether the information supplied is available to test users and how (e.g. on request,
supplied as part of training materials, etc.). Occasionally some of the information required may be
considered proprietary (e.g. scoring algorithms). Please clearly mark anything which is confidential and
indicate on your test review Application Form the documents where this applies. These will then be
treated as ‘commercial in confidence’ by the Editorial Panel and the reviewers allocated to the specific
test review.
Where possible technical documents should be supplied in paper form. CD ROMs as well as PC or web
access codes should be supplied to enable review of computer-based materials. Please note four copies
of all information are required (including access codes).
Please check that the information you send covers the required review areas. You can check the criteria
that will be used in the test review process in the EFPA Review Model located on the PTC website:
www.psychtesting.org.uk
Applying for a Certificate of Test Registration for tests previously reviewed
You will need to submit materials for a full test review and the outcome of this process will provide the
basis for the decision about the award of a Certificate of Test Registration.
What should I do now? Please complete the pro formas in Appendix A and B. A fee will be payable for
full test review. Please see the Schedule of Fees in Appendix D for details.
5
Information relating to renewal of a Certificate of
Test Registration or submission for an
Editorial Panel Review Update
Renewing a Certificate of Test Registration after a five-year cycle, or up to 15 years
since the test review was published when there have been no substantive changes to
the test
This guidance applies to those:
● who have been issued a Certificate of Test Registration for the first time, which is about to expire at
the end of a five-year period, and who wish to renew this for a further five-year period;
● who have been issued a Certificate of Test Registration for the second time, which is about to expire
at the end of a further five-year period, and who wish to renew this for another five-year period;
● where there have been no substantive changes to the test (such as new data, revised items or new
modes of presentation which will impact on the norms, validity and reliability of the test), and the
accompanying test review is up to 15 years old, you will need to sign an affirmation to this effect.
Please see Appendix E. In addition you will need to settle the renewal invoice for a Certificate of
Test Registration and renew your entry on the Directory of Test Publishers/Distributors annually. This
enables your test to be re-registered and for your editorial panel test review to continue to appear on
the PTC website.
Renewing a Certificate of Test Registration when a test review was published
15 or more years ago
If you wish to renew your Certificate of Test Registration for the third time and the accompanying test
review is 15 years or more old, the test will need to re-enter the full test review process. This will
compromise evaluating the test to determine whether it meets criteria for a Certificate of Test
Registration, followed by an Editorial Panel Test Review.
For details of the full test review process, please view pages 3 to 4 and complete pro formas in
Appendix A and B. Please also refer to the Schedule of Fees in Appendix D.
If you choose not to pursue re-submission of your test into the full review process, the existing test
review will subsequently feature in our test review archive area of the PTC website.
Renewing a Certificate of Test Registration after a five-year cycle when there have been
substantive changes to the test since the Certificate of Test Registration was granted
The guidance below relates to a test which has:
● been registered five years ago and is due for renewal of the Certificate of Test Registration;
● has a published editorial panel test review on the PTC website (which is up to 15 years old);
● subsequently had substantive changes made to it since the Certificate of Test Registration was
issued, such as new data, revised items or new modes of presentation which will impact on the
norms, validity and reliability of the test.
An Editorial Panel Review Update will be required. This requires an applicant to highlight where there
have been no subsequent changes to factual information since the original submission, as well as
identifying and cross-referring to the location within test materials of where there have been substantive
changes made to the test.
6
Renewing a Certificate of Test Registration inside the standard five-year test
registration cycle when there have been substantive changes to the test since the
Certificate of Test Registration was granted
The BPS will accept requests for an Editorial Panel Review Update inside of the established five-year
cycle for test registration where there have been substantive changes made to a test since the award of
a Certificate of Test Registration. However, there are higher fees payable in order to accommodate these
requests inside of the five-year cycle. Please refer to the Schedule of Fees in Appendix D.
What should I do now?
The checklist below prompts you about which pro formas to complete. Please ensure that you look at
the Schedule of Fees in Appendix D. These vary according to whether your test has reached the end of
a five-year cycle for test registration, or whether you are applying for an update Editorial Panel Review
Update inside a five-year cycle.
The Senior Editor, in consultation with the Consultant Editor, will determine the extent of the
substantive changes to the test and they reserve the right to come back to a publisher to state that the
materials will need to re-enter a full test review rather than an update review.
Checklist for renewal of a Certificate of Test Registration and submission for an
Editorial Panel Review Update if:
●
you have reached the end of a five year cycle for test registration or your test is inside the standard
five-year renewal cycle;
● and there have been substantive changes made to the test since initial registration;
● and the test review is up to 15 years old.
●
Step 1 can be completed by an administrator:
Complete Application Form and Terms and Conditions (Appendix A)
●
Step 2 can be completed by an administrator:
Complete Section 6 of the Factual Information Form for submission into the test review process
(Appendix B)
●
Step 3 can be completed by a senior administrator within the test publishing company, or compiled
by a Chartered Psychologist:
Complete the Editorial Panel Test Review Update pro forma (Appendix C)
●
Step 4 can be completed by an administrator under the supervision of a Chartered Psychologist:
Ensure test materials detailed in this document on pages 4 and 5 and all necessary forms outlined
above are submitted with the application for test registration and an Editorial Panel Review Update.
Please remember only two copies of test materials are required for test registration followed by an
Editorial Panel Review Update (not four copies). Where there have been changes to items, scoring
or computer generated reports it is essential that these are submitted with your application.
●
Step 5 can be completed by an administrator:
Check the Schedule of Fees in Appendix D. These will vary according to whether you have reached
the end of a five-year cycle, or are inside this timeframe.
7
Flowcharts summarising test review processes
New test review procedure flow chart
8
Submitting for re-registration of a test following a five-year cycle – Editorial Panel Review Update
9
Appendix A: Application Form and Terms and
Conditions
Company details
Please complete the details below
Company or Business Name: ...........................................................................................................
BPS Reference Number (if applicable): ............................................................................................
Company address:
....................................................................................................................................................
....................................................................................................................................................
....................................................................................................................................................
Postcode: ......................................................................................................................................
Company telephone number: ...........................................................................................................
Company email address: .................................................................................................................
Company website: ..........................................................................................................................
We will need to correspond with you during the test registration and test review process.
Your details will not be released to any third party and are for office use only.
Name of Administrator overseeing application for test registration and test review:
....................................................................................................................................................
Administrator’s job title:..................................................................................................................
Administrator’s email address:.........................................................................................................
Administrator’s direct dial number: ..................................................................................................
10
Terms and Conditions and statement from test publishers/distributors
Please read the terms and conditions relating to the test review process and sign to indicate your
agreement to the terms. No consideration of test materials will be made without a signed terms and
conditions statement.
I understand that:
● Only test materials available/published in the UK, or intended for publication and sale in the UK
can enter the test review process.
● As a general rule the submission of four sets of test materials is required from a test
publisher/distributor in order to submit for test review. The test materials will be retained by the
Test Reviews Editorial Panel at the end of the process.
● I have the right to determine whether certain test materials are commercially sensitive. I will have
the opportunity to highlight these to the BPS and the Test Reviews Editorial Panel will be informed
and asked to sign a non-disclosure agreement. If I request that the test materials are returned to me
at the end of the review process, I will organise a courier to return the materials and will bear the
cost of this. Please contact the BPS in relation to this, at the point of application.
● The Test Reviews Editorial Panel is required to declare any intellectual or commercial relationship
with a test/test publisher and that this may mean that an alternative reviewer/editor will be
allocated a specific test in their place.
● The identity of the test reviewers and Consultant Editor involved in the review of a test will remain
confidential up to the point of a published editorial panel test review. Any contact from a publisher
in relation to test review must be directed via the contact in the PTC.
● There is a fee payable of £510 for the later submission of updated or amended test materials to
consider in the test review process. This is payable once a review is underway and the fee must be
paid before continuation of the test review.
● I have the right to withdraw a draft Editorial Panel Test Review from publication. In this case the
Society will indicate on the PTC website that the review has not been published. A statement
indicating that there are a variety of reasons why a review may not have been published will appear
on the website, and enquirers will be directed to the publisher if they wish to seek further
clarification. In addition the BPS will remove registered test status from the test and the right to
use the BPS registered test logo will be removed from the publisher. I understand that if I request
all test materials to be returned that I will arrange for a courier to collect the materials and will
meet the cost for this.
● Upon award of a Certificate of Test Registration, the test publisher/distributor can affix to the test in
paper or electronic form a notice that the instrument has been registered as a test by the PTC, and
can use the BPS registered test logo, as a ‘quality mark’.
● A full copy of the wording on a Certificate of Test Registration must be accessible to test takers,
either in print on the test booklet or as an on-screen page for a computer-based or online test. The
Certificate of Test Registration and the associated BPS registered test logo shall relate only to the
test which has been registered, and only to the language version covered in the submission to the
BPS. It shall not be deemed to be an endorsement or registration of the author, publisher or
distributor, and may not be used in that way.
● A Certificate of Test Registration and an Editorial Panel Review of a test will relate specifically to
the English language version and this should be made explicit to clients and enquirers by the test
publisher/distributor if they are granted a Certificate of Test Registration and a License to Distribute
Test Reviews.
● A test can be registered for a five year period and is subject to a re-registration and renewal process
thereafter. Should renewal fees remain unsettled beyond the settlement period the test will be
removed from the list of registered tests on the PTC website and the use of the logo by the
publisher must be suspended. The test review will also be removed from the PTC website. Should a
publisher subsequently settle their fee at a later point (within a maximum of 12 months), the logo
for the specific test shall be reinstated on the PTC website and the publisher informed. The five
year registration period is calculated from the original date of expiry and not from the subsequent
late payment date.
11
●
Tests which have undergone substantive changes which are inside a five-year test registration cycle,
or are awaiting re-registration at the end of a five year cycle, will first be considered for an Editorial
Panel Review Update. Should the Senior Test Reviews Editor and Consultant Editor consider that
the substantive changes are such that a full Editorial Panel Test Review is required, the test
publisher/ distributor will be advised of this. There will be a difference in fees payable by the
publisher/ distributor in this eventuality.
● For the period in which a test review is published on the PTC website, a test publisher will be
required to hold an annually renewable subscription for an enhanced entry on the Directory of Test
Publishers/Test Distributors. Failure to maintain an enhanced entry on the directory will mean the
removal from the PTC website of registered test status and a published test review.
Should a publisher subsequently settle their fee at a later point (within a maximum of 12 months),
the logo for the specific test shall be reinstated on the PTC website, the review published again on
the website and the publisher informed. Your Certificate of Test Registration expiry date will not be
extended in such circumstances.
Please sign the statement below:
I confirm that the consent of the copyright holder to submit the test ............................................
into the BPS test review process has been obtained.
I agree to the terms and conditions for test registration and review above.
Print name:...................................................... Position: .........................................................
Signature:........................................................ Date: ..............................................................
12
Appendix B: Factual Information form for
submission into the test review process
To submit your test into the test review process, please complete the Factual Information Form below.
Reference numbers in the left-hand column relate to criteria in the EFPA Model for the Description and
Evaluation of Tests. Please see the PTC website www.psychtesting.org.uk for details.
N.B. If you have applied for re-registration of a test because there have been substantive changes
made to it, and now also need to submit the test for an Editorial Panel Review Update, please view
Appendix C for the appropriate pro forma to complete and return in this instance.
Section 1: Description of the Instrument: General information and classification
1.1
Instrument name (local version)
Short version test name
1.2
Original test name (if the local version is
an adaptation
1.4
Authors of the original test
1.3
Authors of the local ldaptation
1.7
Local test publisher/distributor
1.8
Publisher of the original version of the
test (if different to current publisher/
distributor
1.9.1
Date of publication of current revision/
edition
1.9.2
Date of publication of adaptation for
local use
1.9.3
Date of publication of original test
13
Section 2: Classification
1.10.1
Contents domain
(please tick those that apply)
You should identify the content domains
specified by the publisher. Where these are
not clear, this should be indicated and you
should judge from the information provided
in the manual (standardisation samples,
applications, validation, etc.)
what the most appropriate answers are
for 1.10.1.
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
1.10.2
1.10.3
14
Intended or Main Area (s) of Use
(please tick those that apply)
You should identify the intended areas of
uses specified by the publisher. Where
these are not clear, this should be indicated
and you should judge from the information
provided in the manual (standardisation
samples, applications, validation, etc.)
what the most appropriate answers are
for 1.10.2.
■
Intended Mode of Use (conditions under
which the instrument was standardised and
validated) (tick one box only)
This section is important as it identifies
whether the instrument has been designed
with the intention of it being used in
unsupervised or uncontrolled
administration conditions.
This item should be answered from
information provided by the publisher and
checked for accuracy.
■
■
■
■
■
■
■
■
■
Scholastic attainment
General ability
Verbal ability
Numerical ability
Spatial ability
Non-verbal ability
Perceptual speed
Memory
Manual skills/dexterity
Personality – Trait
Personality – Type
Personality – State
Cognitive Styles
Motivation
Values
Interests
Beliefs
Disorder and pathology
Group function
Family function
Organisational function, aggregated
measures, climate, etc.
School or educational function
Other (describe below)
Psycho-clinical
Psycho-neurological
Forensic
Educational
Work and Occupational
Counselling, advice, guidance and
career choice
General health, life and well-being
Sports and Leisure
Other (describe below)
Unsupervised administration without
control over the identity of the test taker
and without full control over the
conditions of administration (e.g. open
access internet delivered test; test
available for purchase from bookstores)
■ Controlled by unsupervised
administration. Control over conditions
(timing, etc.) and some control of
identify of the test taker (e.g. tests
administered over the internet but only
to known individuals – password
restricted access)
■ Supervised and controlled
administration. Test administration
under the control of a qualified
administrator or proctor
■ Managed administration. Test
administration only provided through
specified testing centres (e.g. licensing
Section 2: Classification (continued)
and certification assessment
programmes)
1.10.4
Description of the populations for which
the test is intended
For some tests this may be very general
(e.g. adults), for others it may be more
specific (e.g. manual workers or boys aged
10 to 14).
1.10.5
Number of scales and brief description of
the variable or variables measured by the
instrument
This item should be answered from
information provided by the publisher.
Please indicate the number of scales
(if more than one) and provide a brief
description of each scale if its meaning is
not clear from its name. Reviews of the
instrument should include discussion of
other derived scores where these are
commonly used with the instrument and are
described in the standard documentation,
for example, 16PF criterion scores and
Adjustment Specification Equation scores –
but not scales which are ‘add-ons’, for
example, the use of 16PF or OPQ scores to
generate Belbin team-type measures.
1.11
Items format (select one)
This item should be answered from
information provided by the publisher.
Note that it is important not to confuse
multiple choice same scale with multiple
choice different scale item formats. The
latter, ipsative, formats require test takers
to make choices between sets of two or
more items drawn from different scales.
For ‘Forced-choice mixed-scale
alternatives’, the test taker has to select
which of two scales are more like them or
with which of two statements they most
agree. For multiple-choice version, there
may be three or more statements drawn
from an equivalent number of different
scales. Typically these statements may
have to be ranked or the most- and leastlike-me options selected.
1.12
■
■
■
■
■
■
■
■
Open
Multiple choice, same scale alternatives
Bipolar adjectives
Likert ratings
Forced choice, mixed scale alternatives
(ipsative) – see Notes for explanation.
Multiple choice, mixed scale alternatives
(ipsative) – see Notes for explanation.
Adjective pair of sets, mixed scales
(ipsative)
Other (describe below)
Number of test items
This item should be answered from
information provided by the publisher.
If the instrument has several scales, make
clear whether you are indicating the total
number of items or the number of items
for each scale. Where items load on more
than one scale, this should be documented.
15
Section 2: Classification (continued)
1.13
1.14
1.15
16
Administration Mode(s)
This item should be answered from
information provided by the publisher.
If any special pieces of equipment (other
than those indicated in the list of options,
e.g. VCR, tape recorder) are required, they
should be described here. In addition, any
special testing conditions should be
described. ‘Standard testing conditions’ are
assumed to be available for proctored/
supervised assessment. These would
include a quiet, well-lit and well-ventilated
room with adequate desk-space and
seating for the necessary administrator(s)
and candidate(s).
■
Response Mode
This item should be answered from
information provided by the publisher.
If any special pieces of equipment (other
than those indicated in the list of options,
e.g. VCR, tape recorder) are required, they
should be described here. In addition, any
special testing conditions should be
described. ‘Standard testing conditions’ are
assumed to be available for proctored/
supervised assessment. These would include
a quiet, well-lit and well-ventilated room with
adequate desk-space and seating for the
necessary administrator(s) and candidate(s).
■
Time required for administering the
instrument
This item should be answered from
information provided by the publisher.
The response to this item can be broken
down into a number of components.
In most cases, it will only be possible to
provide general estimates of these rather
than precise figures. The aim is to give the
potential user a good idea of the time
investment associated with using this
instrument. Do NOT include the time
needed to become familiar with the
instrument itself. Assume the user is
experienced and qualified.
Preparation time (the time it takes the
administrator to prepare and set out the
materials for an assessment session).
● Administration time per session: This
includes the time taken to complete all
the items and an estimate of the time
required to give instructions, work
through example items and deal with
any debriefing comments at the end of
the session.
■
■
■
■
■
■
Interactive individual administration
Supervised group administration
Computerised locally-installed
application – supervised/proctored
Computerised web-based application –
supervised/proctored
Computerised locally-installed
application – unsupervised/
self-assessment
Computerised web-based application –
unsupervised/self-assessment
Other (indicate)
Oral interview
Paper and pencil
■ Manual operations
■ Computerised
■ Other (indicate)
■
Preparation:
Administration:
Scoring:
Analysis:
Feedback:
Section 2: Classification (continued)
●
Scoring: The time taken to obtain the
raw-scores.
● Analysis: The time taken to carry out
further work on the raw scores to derive
other measures and to produce a
reasonably comprehensive interpretation
(assuming you are familiar with the
instrument).
● Feedback: The time required to prepare
and provide feedback to a candidate.
It is recognised that time for the last two
components could vary enormously –
depending on the context in which the
instrument is being used. However, some
indication or comments will be helpful.
1.16
Indicate whether different forms of the
instrument are available (genuine or
pseudo-parallel forms, short versions,
computerised versions, etc.).
If computerised versions do exist, describe
briefly the software and hardware
requirements.
Report whether or not there are alternative
versions (forms) of the instrument available
and describe the applicability of each form
for different groups of people. In some
cases, different forms of an instrument are
meant to be equivalent to each other, that
is, alternate forms. In other cases, various
forms may exist for quite different groups
(e.g. a children’s form and an adult’s
form). Where more than one form exists,
indicate whether these are
equivalent/alternate forms, or whether they
are designed to serve different functions,
for example, short and long version;
ipsative and normative version.
17
Section 3: Measurement and Scoring
1.17
Scoring procedure for the test
This item should be completed by
reference to the publisher’s information
and the manuals and documentation.
Bureau services are services provided by
the supplier – or some agent of the
supplier – for scoring and interpretation. In
general these are optional services.
If scoring and/or interpretation can be
carried out ONLY through a bureau service,
then this should be stated in the review –
and the costs included in the recurrent
costs section.
■
■
■
■
■
■
■
Computer scoring with direct entry of
responses by test taker
Computer scoring with manual entry of
responses from the paper response form
Computer scoring by Optical Mark
Reader entry of responses from the
paper response form
Simple manual scoring key – clerical
skills only required
Complex manual scoring – requiring
training in the scoring of the instrument
Bureau-service – for example, scoring by
the company selling the instrument
Other (describe below)
1.18
Scores
This item should be completed by
reference to the publisher’s information
and the manuals and documentation.
Brief description of the scoring system to
obtain global and partial scores, correction
for guessing, qualitative interpretation aids,
etc.
1.19
Score transformation for standard scores
■
1.20
Scales used (tick all that apply)
This item should be completed by
reference to the publisher’s information
and the manuals and documentation.
Percentile Based Scores
■ Centiles
■ 5-grade classification: 10:20:40:20:10
centile splits
■ Deciles, and other equi-percentile
classifications
Normalised – scores obtained by use of
normalisation look-up table
■ Not-normalised – scores obtained by
linear transformation
Standard Scores
■ Z-scores
■ IQ deviation quotients, etc. (e.g. mean
100, SD=15 for Weschler or 16 for
Stanford-Binet)
■ College Entrance Examination Board
(e.g. SAT mean=500, SD=100)
■ Stens,
■ Stanines, C-scores
■ T-scores
■ Other (please describe)
18
Section 4: Computer-generated reports
Note that this is purely descriptive. Evaluations of the reports would be part of the Evaluation section of
the review.
For instances where there are multiple generated reports available please complete items below for each
report (please copy pages if necessary), This classification system could be used to describe two reports
provided by a system, for example, Report 1 may be intended for the test taker or other untrained users,
and Report 2 for a trained test user who is competent in the use of the instrument and understands how
to interpret it.
1.21
Are computer-generated reports available
with the instrument?
If the answer to 1.21 is ‘YES’ then the
following classification should be used to
classify the types of reports available. For
many instruments, there will be a range of
reports available. Please complete a
separate form for each report.
■
■
Yes (complete sections below)
No (move to section 1.23)
Number of computer-generated reports
available?
1.21.0
Name or description of report
(see introduction to this section)
1.21.1
Media (select one)
Reports may consist wholly of text or
contain text together with graphical or
tabular representations of scores (e.g. sten
profiles). Where both text and data are
presented, these may simply be presented
in parallel or may be linked, so that the
relationship between text statements and
scores is made explicit.
■
1.21.2
Complexity (select one)
Some reports are very simple, for example,
just substituting a text unit for a sten score
in a scale-by-scale description. Others are
more complex, involving text units which
relate to patterns or configurations of scale
scores and which consider scale interaction
effects.
■
1.21.3
Report structure (select one)
Structure is related to complexity.
■
Text only
Unrelated text and graphics
■ Integrated text and graphics
■ Graphics only
■
Simple (For example, a list of
paragraphs giving scale descriptions)
■ Medium (A mixture of simple
descriptions and some configural
descriptions)
■ Complex (Contains descriptions of
patterns and configurations of scale
scores, and scale interactions)
■
■
■
■
Scale-based (where the report is built
around the individual scales)
Factor-based (where the report is
constructed around higher order factors
– such as the ‘Big Five’ for personality
measures.
Construct-based – where the report is
built around one or more sets of
constructs (e.g. in a work setting these
could be such as team types, leadership
styles, tolerance to stress, etc.) which are
linked to the original scale scores.
Criterion-based where the reports
focuses on links with empirical
outcomes (e.g. training potential, job
performance, absenteeism, etc.).
Other
19
Section 4: Computer Generated Reports continued
Sensitivity to context (select one)
When people write reports they tailor the
language, form and content of the report to
the person who will be reading it and take
account of the purpose of the assessment
and context in which it takes place. A
report produced for selection purposes will
be different from one written for guidance
or development; a report for a middle-aged
manager will differ from that written for a
young person starting out on a training
scheme and so on.
■
1.21.5
Clinical-actuarial (select one)
Most reports systems are based on clinical
judgment. That is, one or more people who
are ‘expert-users’ of the instrument in
question will have written the text units.
The reports will, therefore, embody their
particular interpretations of the scales.
Some systems include actuarial reports
where the statements are based on
empirical validation studies linking scale
scores to, for example, job performance
measures.
■
1.21.6
Modifiability (select one)
The report output is often fixed. However,
some systems will produce output in the
form of a file that can be processed by the
user.
■
Degree of Finish (select one)
A related issue is the extent to which the
system is designed to generate integrated
text – in the form of a ready-to-use report –
or a set of ‘notes’, comments, hypotheses,
etc. The latter is clearly of far more use
when the text is available to the user in
modifiable form and can form the basis for
the user’s own report. In many cases,
reports are designed to a very high
standard of presentation, having a
‘published’ appearance and quality.
■
Transparency (select one)
Systems differ in their openness or
transparency to the user. An open system
is one where the link between a scale score
and the text is clear and unambiguous.
Such openness is only possible if both text
and scores are presented and the links
between them made explicit. Other
systems operate as ‘black boxes’, making it
difficult for the user to relate scales scores
to text.
■
1.21.4
1.21.7
1.21.8
20
One version for all contexts
Pre-defined context-related versions
■ User definable contexts and editable
reports
■
Based on clinical judgement of
one expert
■ Based on empirical/actuarial
relationships
■ Based on clinical judgement of group
of experts
Not modifiable (fixed print-only output)
Limited modification (limited to certain
areas, e.g. biodata fields)
■ Unlimited modification (e.g. through
access to Word Processor document file)
■
■
Publication quality
Draft quality
Clear linkage between constructs scores
and text
■ Concealed link between constructs,
scores and text
■ Mixture of clear/concealed linkage
betweenconstructs, scores and text
Section 4: Computer Generated Reports continued
1.21.9
1.21.10
Style and tone (select one)
Systems also differ in the extent to which
they offer the report reader guidance or
direction. Some are stipulative: ‘Mr X is
very shy and will not make a good
salesman…’. Others are designed to
suggest hypotheses or raise questions:
‘From his scores on scale Y, Mr X appears
to be very shy. If this is the case, he could
find it difficult working in a sales
environment. This needs to be explored
further with him.’
■
Directive
Guidance
■ Other
■
Intended recipients (select all that apply)
Reports are generally designed to address
the needs of one or more categories of
users. Users can be divided into four main
groups:
(a) Qualified test users. These are people
who are sufficiently knowledgeable and
skilled to be able to produce their own
reports based on scale scores. They should
be able to make use of reports that use
technical psychometric terminology and
make explicit linkages between scales and
descriptions. They should also be able to
customise and modify reports.
■
Qualified test users
(b) Qualified system users. While not
competent to generate their own reports
from a set of scale scores, people in this
group are competent to use the outputs
generated by the system. The level of
training required to attain this competence
will vary considerably, depending on the
nature of the computer reports (e.g. traitbased versus competency-based, simple or
complex) and the uses to which its reports
are to be put (low stakes or high stakes).
■
Qualified system users
(c) Test takers. The person who takes the
instrument will generally have no prior
knowledge of either the instrument or the
type of report produced by the system.
Reports for them will need to be in
language that makes no assumptions about
psychometric or instrument knowledge.
■
Test takers
(d) Third parties. These include people –
other than the candidate – who will be
privy to the information presented in the
report or who may receive a copy of the
report. They may include potential
employers, a person’s manager or
■
Third parties
21
Section 4: Computer Generated Reports continued
supervisor or the parent of a young person
receiving careers advice. The level of
language required for people in this
category would be similar to that required
for reports intended for Test takers.
1.22
Do distributors offer a service to correct
and/or develop computerised reports?
(select one)
■
■
Yes
No
Section 5: Supply, Condition and Costs
1.23
1.24
1.25.1
22
Documentation provided by the distributor
as part of the test package
(select all that apply)
■
Methods of publication
(select all that apply)
For example, technical manuals may be
kept up-to-date and available for
downloading from the internet, while user
manuals are provided in paper form or on a
CD.
■
Start-up costs
Price of a complete set of materials (all
manuals and other material sufficient for
at least one sample administration).
Specify how many candidates could be
assessed with the materials obtained for
start-up costs, where these costs include
materials for recurrent assessment
This item should try to identify the ‘set-up’
cost. That is the costs involved in obtaining
a full reference set of materials, scoring
keys and so on. It only includes training
costs if the instrument is a ‘closed’ one –
where there will be an unavoidable specific
training cost, regardless of the prior
qualification level of the user. In such
cases, the training element in the cost
should be made explicit. The initial costs
do NOT include costs of general-purpose
equipment (such as computers, cassette
tape recorders and so on). However, the
need for these should be mentioned. In
general, define: any special training costs;
costs of administrator’s manual; technical
manual(s); specimen or reference set of
materials; initial software costs, etc.
User manual
Technical (psychometric) manual
■ Supplementary technical information
and updates (e.g. local norms, local
validation studies, etc.)
■ Books and articles of related interest
■ Combinations of the above (specify)
■
■
■
■
■
■
Paper
PC – Diskettes
PC – CD/ROM
Internet download
Live internet (instrument runs in a
web browser)
Other (specify)
Section 5: Supply, Condition and Costs continued
1.25.2
Recurrent costs
This item is concerned with the ongoing
cost of using the instrument. It should give
the cost of the instrument materials
(answer sheets, non-reusable or reusable
question booklets, profile sheets, computer
usage release codes or ‘dongle’ units, etc.)
per person per administration. Note that in
most cases, for paper-based administration
such materials are not available singly but
tend to be supplied in packs of 10, 25
or 50.
Itemise any annual or per capita licence
fees (including software release codes
where relevant), costs of purchases or
leasing re-usable materials, and per
candidate costs of non-reusable materials.
1.26.1
Prices for reports generated by user
installed software
1.26.2
Prices for reports generated by postal/
fax bureau service
1.26.3
Prices for reports by internet service
1.27
Prices for other bureau services: Correcting
or developing automatic reports
1.28
Test-related qualifications required by the
supplier of the test (select all that apply)
1.28 concerns the user qualifications
required by the supplier. Where the
qualification requirements are not clear
this should be stated under ‘Other’ not
under ‘None’. ‘None’ means that there is
an explicit statement regarding the lack of
need for qualification.
■
■
■
■
■
■
1.29
Professional qualifications required for use
of the instrument (select all that apply)
1.29 concerns the user qualifications
required by the supplier. For this section,
where the publisher has provided user
qualification information, this should be
noted against the categories given. Where
the qualification requirements are not clear
this should be stated under ‘Other’ not
under ‘None’. ‘None’ means that there is
an explicit statement regarding the lack of
need for qualification.
■
■
■
■
■
■
■
None
Test specific accreditation
Accreditation in general achievement
testing: measures of maximum
performance in attainment
Accreditation in general ability and
aptitude testing: measures of maximum
performance in relation to potential for
attainment
Accreditation in general personality and
assessment: measures of typical
behaviour, attitudes and preferences
Other (specify)
None
Practitioner psychologist with
qualification in the relevant area of
application
Practitioner psychologist
Research psychologist
Non-psychologist academic researcher
Practitioner in relevant related
professions (therapy, medicine,
counselling, education, human
resources, etc.)
Holder of BPS Certificate of Registration
in Test User Occupational Ability
(formerly Level A)
23
Section 5: Supply, Condition and Costs continued
■
Holder of BPS Certificate of Registration
in Test User Educational Ability
(formerly CCET/Level A Educational)
■ Holder of BPS Certificate of Registration
in Test User Occupational Personality
(formerly Level B)
■ Other (indicate)
24
Appendix C: Editorial Panel Review Update
Pro Forma
This form is for the purpose of submitting for an Editorial Panel Review Update where there is already a
published test review appearing on the PTC website, but a publisher/distributor has made substantive
changes to the test since the publication of the initial review.
1. Please answer all of Sections 1 to 5, even if there have been no factual changes.
2. If there is new information, please highlight the items that have changed and note in the table any
information that may be relevant. Otherwise, please leave these sections blank.
3. Where there are changes to items, scoring or computer-generated reports please send relevant
documents to support this.
4. For Sections 6 and 7 if you have made changes,
● in the first column (rating in last review) give the star rating allocated for the item in the last
review, or the verbal judgement made, and enclose supporting documentation in a copy of your
manuals.
● Do not indicate what you would like the new rating to be.
● For any amendments/further information supplied/or further studies that are relevant to the
item, please indicate what these are in the column on the right, with dates and page references
to your manuals where appropriate.
● Do not send research articles without stating where they are relevant and where the relevant
information can be found.
Section 1: Description of the Instrument – General information and classification
Please answer all boxes even if there have been no factual changes.
Date of current review
Instrument name (local version)
Original test name (if the local version is
an adaptation)
Do you have permission of the copyright
holder for this update?
1.4
Authors of the original test
1.3
Authors of the local adaptation
1.7
Local test publisher/distributor
1.8
Publisher of the original version of the
test (if different to current publisher/
distributor
1.9.1
Date of publication of current revision/
edition
1.9.2
Date of publication of adaptation for
local use
1.9.3
Date of publication of original test
25
Section 2: Classification
Only indicate those items that have changed from the previous test review.
1.10.1
Contents domain
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
■
1.10.2
Intended or Main Area (s) of Use
Only indicate those that have changed.
■
■
■
■
■
■
■
■
■
1.10.3
Intended Mode of Use (conditions under
which the instrument was standardised and
validated)
Only indicate those that have changed.
■
■
■
■
■
26
Scholastic attainment
General ability
Verbal ability
Numerical ability
Spatial ability
Non-verbal ability
Perceptual speed
Memory
Manual skills/dexterity
Personality – Trait
Personality – Type
Personality – State
Cognitive Styles
Motivation
Values
Interests
Beliefs
Disorder and pathology
Group function
Family function
Organisational function, aggregated
measures, climate, etc.
School or educational function
Other (describe below)
Psycho-clinical
Psycho-neurological
Forensic
Educational
Work and Occupational
Counselling, advice, guidance and
career choice
General health, life and well-being
Sports and Leisure
Other (describe below)
Unsupervised administration without
control over the identity of the test taker
and without full control over the
conditions of administration (e.g. open
access internet delivered test; test
available for purchase from bookstores)
Controlled by unsupervised
administration.
Control over conditions (timing, etc.)
and some control of identify of the test
taker (e.g. tests administered over the
internet) but only to known individuals –
password restricted access)
Supervised and controlled
administration
Test administration under the control of
a qualified administrator or proctor
Section 2: Classification continued
1.10.4
Description of the populations for which
the test is intended
1.10.5
Number of scales and brief description of
the variable or variables measured by the
instrument
1.11
Items format (select one)
Only indicate if these have changed.
■
Managed administration.
Test administration only provided
through specified testing centres (e.g.
licensing and certification assessment
programmes)
■
Open
Multiple choice, same scale alternatives
Bipolar adjectives
Likert ratings
Forced choice, mixed scale alternatives
(ipsative) – see Notes for explanation.
Multiple choice, mixed scale alternatives
(ipsative) – see Notes for explanation
Adjective pair of sets, mixed scales
(ipsative)
Other (describe below)
■
■
■
■
■
■
■
1.12
Number of Test Items
1.13
Administration Mode(s)
■
■
■
■
■
■
■
1.14
Response Mode
■
■
■
■
■
1.15
Time required for administering the
instrument
■
■
■
■
■
Interactive Individual Administration
Supervised Group Administration
Computerised locally-installed
application – supervised/proctored
Computerised web-based application –
supervised/proctored
Computerised locally-installed
application – unsupervised/
self-assessment
Computerised web-based application –
unsupervised/self-assessment
Other (indicate)
Oral interview
Paper and pencil
Manual operations
Computerised
Other (indicate)
Preparation
Administration
Scoring
Analysis
Feedback
27
Section 2: Classification continued
1.16
28
Indicate whether different forms of the
instrument are available(genuine or
pseudo-parallel forms, short versions,
computerised versions, etc.).
If computerised versions do exist, describe
briefly the software and hardware
requirements.
Report whether or not there are alternative
versions (forms) of the instrument available
and describe the applicability of each form
for different groups of people. In some
cases, different forms of an instrument are
meant to be equivalent to each other, that
is, alternate forms. In other cases, various
forms may exist for quite different groups
(e.g. a children’s form and an adult’s
form). Where more than one form exists,
indicate whether these are
equivalent/alternate forms, or whether they
are designed to serve different functions,
for example, short and long version;
ipsative and normative version.
Section 3: Measurement and Scoring
Only indicate those items that have changed from the previous test review.
1.17
Scoring procedure for the test
■
■
■
■
■
■
■
1.18
Scores
Brief description of the scoring system to
obtain global and partial scores, correction
for guessing, qualitative interpretation aids,
etc.).
1.19
Score transformation for standard scores
■
1.20
Scales used (tick all that apply)
■
Computer scoring with direct entry of
responses by test taker
Computer scoring with manual entry of
responses from the paper response form
Computer scoring by Optical Mark
Reader entry of responses from the
paper response form
Simple manual scoring key – clerical
skills only required
Complex manual scoring – requiring
training in the scoring of the instrument
Bureau-service, for example, Scoring by
the company selling the instrument
Other (describe below
Normalised – scores obtained by use of
normalisation look-up table
■ Not-normalised – scores obtained by
linear transformation
■
■
■
■
■
■
■
■
■
■
Centiles
5-grade classification: 10:20:40:20:10
centile splits
Deciles, and other equi-percentile
classifications
Standard scores
Z-scores
IQ deviation quotients, etc. (e.g. mean
100, SD=15 for Weschler or 16 for
Stanford-Binet)
College Entrance Examination Board
(e.g. SAT mean=500, SD=100)
Stens,
Stanines, C-scores
T-scores
Other (please describe)
29
Section 4: Computer-generated reports
After completing 1.2.1 and 1.22, only indicate those items that have subsequently changed since the
previous test review.
1.21
Are Computer-generated reports available
with the instrument?
1.21.0
Name or description of Report
Please repeat this section for each new
report and include a copy of the new
report)
1.21.1
Media (select one)
■
■
Yes (complete sections below)
No (move to Section 5)
■
Text only
Unrelated text and graphics
■ Integrated text and graphics
■ Graphics only
■
1.21.2
Complexity (select one)
■
1.21.3
Report structure (select one)
■
Simple (For example, a list of
paragraphs giving scale descriptions)
■ Medium (A mixture of simple
descriptions and some configural
descriptions)
■
■
■
■
1.21.4
Sensitivity to context (select one)
■
1.21.5
Clinical-actuarial (select one)
■
1.21.6
Modifiability (select one)
■
Scale-based where the report is built
around the individual scales)
Factor-based where the report is
constructed around higher order factors
such as the ‘Big Five’ for personality
measures
Construct-based where the report is built
around one or more sets of constructs
(e.g. in a work setting these could be
such as team types, leadership styles,
tolerance to stress, etc.) which are linked
to the original scale scores
Criterion-based where the reports
focuses on links with empirical outcomes
(e.g. training potential, job
performance, absenteeism, etc.)
Other
One version for all contexts
■ Pre-defined context-related versions
■ User definable contexts and editable
reports
Based on clinical judgement of one
expert
■ Based on empirical/actuarial
relationships
■ Based on clinical judgement of group of
experts
Not modifiable (fixed print-only output)
Limited modification (limited to certain
areas, e.g. biodata fields)
■ Unlimited modification (e.g. through
access to Word Processor document file)
■
1.21.7
Degree of Finish (select one)
■
■
30
Publication quality
Draft Quality
Section 4: Computer Generated Reports continued
1.21.8
Transparency (select one)
■
1.21.9
Style and Tone (select one)
■
Clear linkage between constructs scores
and text
■ Concealed link between constructs,
scores and text
■ Mixture of clear/concealed linkage
between constructs, scores and text
Directive
Guidance
■ Other
■
1.21.10
1.22
Intended recipients (select all that apply)
■
Do distributors offer a service to correct
and/or develop computerised reports?
(select one)
■
Qualified test users
■ Qualified system users
■ Test takers
■ Third parties
■
Yes
No
Section 5: Supply, condition and costs
Please do not include documents and manuals which are only used for training and development.
Please complete 1.25.1, 1.25.2, 1.26.1, 1.26.2, 1.26.3 and 1.27 and other areas only where there
have been subsequent changes to the test.
1.23
1.24
Documentation provided by the distributor
as part of the test package
(select all that apply)
■
Methods of publication
(select all that apply)
■
User manual
Technical (psychometric) manual
■ Supplementary technical information
and updates (e.g. local norms, local
validation)
■ Books and articles of related interest
■ Combinations of the above (specify)
■
■
■
■
■
■
1.25.1
Paper
PC – Diskettes
PC – CD/ROM
Internet download
Live internet (instrument runs in a
web browser)
Other (specify)
Start-up costs
Price of a complete set of materials
(all manuals and other material sufficient
for at least one sample administration).
Specify how many candidates could be
assessed with the materials obtained for
start-up costs, where these costs include
materials for recurrent assessment This
item should try to identify the ‘set-up’ cost.
That is the costs involved in obtaining a full
reference set of materials, scoring keys and
so on. It only includes training costs if the
instrument is a ‘closed’ one – where there
will be an unavoidable specific training
31
Section 5: Supply, Condition and Costs continued
cost, regardless of the prior qualification
level of the user. In such cases, the
training element in the cost should be
made explicit. The initial costs do NOT
include costs of general-purpose
equipment (such as computers, cassette
tape recorders and so on). However, the
need for these should be mentioned. In
general, define: any special training costs;
costs of administrator’s manual; technical
manual(s); specimen or reference set of
materials; initial software costs, etc.
1.25.2
Recurrent costs
Specify, where appropriate, recurrent costs
of administration and scoring separately
from costs of interpretation. This item is
concerned with the ongoing cost of using
the instrument. It should give the cost of
the instrument materials (answer sheets,
non-reusable or reusable question booklets,
profile sheets, computer usage release
codes or ‘dongle’ units, etc.) per person
per administration. Note that in most
cases, for paper-based administration such
materials are not available singly but tend
to be supplied in packs of 10, 25 or 50.
Itemise any annual or per capita licence
fees (including software release codes
where relevant), costs of purchases or
leasing re-usable materials, and per
candidate costs of non-reusable materials
1.26.1
Prices for reports generated by user
installed software
1.26.2
Prices for reports generated by postal/
fax bureau service
1.26.3
Prices for reports by internet service
1.27
Prices for other bureau services:
Correcting or developing automatic reports
1.28
Test-related qualifications required by the
supplier of the test (select all that apply
and only indicate those that have changed –
if not please put in this box ‘no changes’)
■
■
■
■
■
■
32
None
Test specific accreditation
Accreditation in general achievement
testing: measures of maximum
performance in attainment
Accreditation in general ability and
aptitude testing: measures of maximum
performance in relation to potential for
attainment
Accreditation in general personality and
assessment: measures of typical
behaviour, attitudes and preferences
Other (specify)
Section 5: Supply, Condition and Costs continued
1.29
Professional qualifications required for use
of the instrument (select all that apply and
only indicate those that have changed – if
not please put in this box ‘no changes’)
■
■
■
■
■
■
■
■
■
■
None
Practitioner psychologist with
qualification in the relevant area
of application
Practitioner psychologist
Research psychologist
Non-psychologist academic researcher
Practitioner in relevant related
professions (therapy, medicine,
counselling, education, human
resources, etc.)
Holder of BPS Certificates of
Registration in Test User Occupational
Ability (former Level A)
Holder of BPS Certificates of
Registration in Educational Testing Ability/Attainment (formerly Level A)
Holder of BPS Certificate of Registration
in Test User Occupational Personality
(formerly Level B)
Other (please indicate)
In the sections that follow (Sections 6 and 7), if you have made changes, give the star rating allocated
for the item in the last test review, or the verbal judgement made, in the first column. For any
amendments/further information supplied/ or further studies that are relevant to the item, please indicate
what these are in the column on the right, with dates and page references to your manuals where
appropriate. Please do not indicate what you would like the new rating to be.
33
Section 6: Evaluation of Test Materials
Rating in last
test review
Quality of the explanation of the rationale,
the presentation and the quality of
information provided
2.1
Overall rating of the quality of the
explanation of the rationale
2.1.1
(i) Theoretical foundations of the
constructs
2.1.2
(ii) Test development procedure
2.1.3
(iii) Thoroughness of the item analyses
and item analysis model
2.1.4
(iv) Explanation of content validity
2.1.5
(v) Summary of relevant research
2.2
2.2.1
Rationale (see rating 2.1)
Well-argued and clearly presented
description of what is designed to measure
and why it was constructed as it was
2.2.2
Development
Full details of item sources, piloting, item
analyses, comparison studies and changes
made during development trials
2.2.3
Standardisation
Clear and detailed information provided
about sizes and sources of standardisation
sample and standardisation procedure
2.2.4
Norms
Clear and detailed information provided
about sizes and sources of norms groups,
conditions of assessment, etc.
2.2.5
Reliability
Good explanation of reliability and a
comprehensive range of internal
consistency and retest measures provided
with explanations of their relevance, and
the generalisability of the assessment
instrument
2.2.6
Validity
Good explanation of validity with a wide
range of studies clearly and fairly described
2.3
34
Adequacy of documentation available to
the user (user and technical manuals,
norm supplements, etc.)
Quality of the procedural instructions
provided for the user
This overall rating is obtained by using
judgment based on the ratings given for
items 2.3.1 to 2.3.7
Changes/new information
with date and document
and page reference
Section 6: Evaluation of Test Materials continued
2.3.1
For test administration
Clear and detailed explanations and stepby-step procedural guides provided, with
good detailed advice on dealing with
candidates’ questions and problem
situations
2.3.2
For test scoring, norming, etc.
Clear and detailed information provided,
with checks described to deal with possible
errors in scoring
2.3.3
For interpretation and reporting
Detailed advice on interpreting different
scores, understanding normative measures
and dealing with relationships between
different scales, with plenty of illustrative
examples and case studies
2.3.4
For providing feedback and debriefing test
takers and others
Detailed advice on how to present
feedback to candidates
2.3.5
For providing good practice issues on
fairness and bias
Detailed information reported about sex
and ethnic bias studies, with relevant
warnings about use and generalisation of
validities
2.3.6
Restrictions on use
Clear descriptions of who should and who
should not be assessed, with well-explained
justifications for restrictions (e.g. types of
disability, literacy levels required, etc.)
2.3.7
References and supporting materials
Detailed references to the relevant
supporting academic literature and crossreferences to other related assessment
instrument materials
Quality of the materials
This overall rating is obtained by using
judgment based on the ratings given for
items 2.4 to 2.8
2.4
General quality of test materials (test
booklets, answer sheets, test objects,
software, etc.)
2.5
Test quality of the local adaptation (if the
test has been translated and adapted into
the local language)
2.6
Ease with which the test taker can
understand the task
2.7
Ease with which responses or answers can
be made by the test taker
2.8
Quality of the items
35
Section 7: Evaluation
Rating in last
test review
Evaluation of technical information: Overall
adequacy
This overall rating is obtained by using
judgment based on the ratings given for
items 2.9 to 2.11
2.9
Norms or reference group information:
Overall adequacy
2.9.1
Appropriateness for local use, whether
local or international norms
2.9.2
Appropriateness for intended applications
2.9.3
Sample sizes
2.9.4
Procedures used in sample selection
(select one)
● No information is supplied
● Representative of population
● Incidental
● Random
2.9.5
Quality of information provided about
minority/protected group differences,
effects of age, gender, etc.
2.10
Validity: Overall adequacy
This overall rating is obtained by using
judgment based on the ratings given for
items 2.10.1 to 2.10.2.4
2.10.1
Construct validity: Overall adequacy
This overall rating is obtained by using
judgment based on the ratings given for
items 2.10.1.2 to 2.10.1.6
2.10.1.1 Designs used (tick as many as are applicable)
● No information is supplied
● Correlations with other instruments and
performance criteria
● Intra-scale (item-rest correlations)
● Differences between groups
● Matrix multi-trait – multi-method
● Exploratory factor analysis
● Confirmatory factor analysis
● Experimental designs
● Other (please indicate)
2.10.1.2 Sample sizes
2.10.1.3 Procedure of sample selection (select one)
● No information is supplied
● Representative of population
● Incidental
● Random
2.10.1.4 Median and range of the correlations
between the test and other similar tests
36
Changes/new information
with date and document
and page reference
Section 7: Evaluation continued
2.10.1.5 Quality of instruments as criteria or
markers
2.10.1.6 Differential Item Functioning (DIF)
analyses
2.10.2
Criterion-related validity: Overall adequacy
This overall rating is obtained by using
judgment based on the ratings given for
items 2.11.1 to 2.10.4.2
2.10.2.1 Description of the criteria used and
characteristics of the populations
(tick as many as are applicable)
● Concurrent
● Predictive
● Post-dictive
2.10.2.2 Sample sizes
2.10.2.3 Procedure of Sample selection
● No information is supplied
● Purposive or representative
(summarise criteria below)
● Incidental
● Random
2.10.2.4 Median and range of the correlations
between the test and criteria
2.11
Reliability: Overall adequacy
This overall rating is obtained by using
judgment based on the ratings given for
items 2.11.1 to 2.10.3.2
2.11.1
Data provided about reliability (select one)
● Only one reliability coefficient given
● Only one estimate of standard error of
measurement given
● Reliability coefficients for a number of
different groups
● Standard error of measurement given
for a number of different groups
2.11.1.
Internal consistency
2.11.1.1 Sample size
2.11.1.2 Median of coefficients
2.11.2
Test retest stability
2,11.2.1 Sample size
2.11.2.2 Median of coefficients
2.11.3
Equivalence reliability
Sample size
Median of coefficients
37
Appendix D: Schedule of Fees for Test Reviews
Upon receipt of your submission the PTC will contact you by email and advise you of fees payable.
Please inform us if you wish to provide us with a purchase order number. An invoice will subsequently
be mailed out to you. Please note that full payment is required by the BPS before commencement of
any activity in relation to test review. A receipt will be sent to you once payment has been collected.
Submission of a single test into the full test review process ....................................................... £622
Annual fee for an enhanced entry on the Directory of Test Publishers & Test Distributors .............. £153
(one entry covers all tests submitted for review)
Fee for an Editorial Panel Review Update (applicable to tests which already have ....................... £122
a published BPS test review, where the review is up to 15 years old and the review
is required as part of a five year renewal cycle for test registration)
Fee for renewal of a Certificate of Test Registration ................................................................... £255
Exceptional fees
Fee payable upon late submission of additional materials for consideration ................................. £510
in the test review process
Fee payable for re-submission of a test into the full test reviews process where ........................... £867
publication of a prior review has been withdrawn by the test publisher/distributor
Fee payable for an Editorial Panel Review Update where a test has been re-submitted ................. £622
for review and registration inside a standard five year cycle.
Repeat fee payable annually
Annual fee for enhanced entry on the Directory of Test Publishers/Test Distributors ...................... £153
38
Appendix E: Renewal of Certificate of Test
Registration
Affirmation from a test publisher/distributor that no substantive changes have
been made since the initial registration of the test (applicable to tests where
a renewal invoice has been issued from the BPS, where the accompanying
test review is up to 15 years old).
This should be completed by a Chartered Psychologist or a senior member of staff with responsibility
for test materials. This form should be completed and returned to the Society when an invoice for test
registration renewal has been issued.
I hereby confirm that the test ........................................ has not undergone any substantive changes
since application for initial registration. I understand that at the end of the next five year period if the
test has undergone any significant changes I (the publisher/distributor) must apply for an Editorial
Panel Review Update and that this would be the basis upon which a new Certificate of Test Registration
would be awarded.
Name of Chartered Psychologist/senior member of staff
....................................................................................................................................................
BPS membership number (where applicable)
....................................................................................................................................................
Signed.…………………………………………………. Date ......................................….…………………
Company or business name .............................................................................................................
Position.........................................................................................................................................
39
Psychological Testing Centre
The British Psychological Society
St Andrews House
48 Princess Road East
Leicester LE1 7DR
w: www.psychtesting.org.uk
t: +44 (0)116 252 9530
f: +44 (0)116 247 0787
e: [email protected]
PTC34/01.2015