Download Geocoding Rule Base Developer Guide

A rule of thumb, to determine if a record linkage application is
feasible, is to multiply the number of values in each field, and
compare this product with the number of records in both files. If
the product is much greater than the number of records, the
application is probably feasible.
For example, if the fields sex, age, and middle initial were the only
fields that could serve as matching fields, then the following
calculation could be made: sex has two possible values, age has
one hundred, and middle initial has twenty-six. When you
multiply the possible values in this field, you get 5,200
(2 x 100 x 26). Since there are only 5,200 possible values for the
fields, only very small datasets can be matched with any
confidence. The probability that more than one record is an exact
duplicate and does not represent the same individual is high with
a file size of 5,200. The actual probabilities would depend on the
distribution of the values in the fields.
Blocking
For any reasonably sized file, it is unreasonable to compare all
record pairs since the number of possible pairs is the product of
the number of records in each file. Even a case with two small files
of 1,000 records each has 1,000,000 possible pairs to examine. Of
this million, a maximum of 1,000 will be matches. The other 999,000
are unmatched pairs. If there were a way to look at pairs of
records having a high probability of being matches and ignoring
all pairs with low probabilities, then it would become
computationally feasible to conduct the linkage with large files.
Fortunately, the concept of blocking provides a method of
limiting the number of pairs being examined. If one were to
partition both files into mutually exclusive and exhaustive
subsets and only search for matches within a subset, then the
process of linkage would become manageable.
APPENDIX A
To understand the concept of blocking, consider a field such as
age. If there are 100 possible ages, then this variable partitions a
file into 100 subsets. The first subset is all people with an age of
zero, the next is those with an age of 1, and so on. These subsets
are called blocks (or pockets in some systems). Suppose, for the
sake of example, that the age values were uniformly distributed. In
this case, out of the 1,000-record file, there would be 10 records
for people of age 0 on each file, 10 records for people of age 1,
and so on.
The pairs of records to be compared are taken from records in the
same block. The first block would consist of all persons of age 0
in files A and B. This would be 10 x 10 or 100 record pairs. The
second block would consist of all persons in files A and B with an
age of 1. When the process is complete, you would have
compared 100 (blocks) x 100 (pairs in a block) = 10,000 pairs,
rather than the 1,000,000 record pairs required without blocking.
Blocking ensures that all records that have the same value in the
blocking variable are compared. One consequence of this is that
records that don’t match on the blocking variables will
automatically be classified as nonmatched. For example, if our
blocking variable were age, and age was in error in one of the
files, then the records involved are considered to be unmatched.
To get around this problem, multiple passes are used.
Suppose a match is run where age is the blocking variable. Any
records that do not match can be rematched using another
blocking scheme, for example, postal code of residence. If a
record did not match on age in pass 1, then it still has an
opportunity to match on postal code in pass 2. It is only those
cases that have errors in both the age and postal code fields that
will not be matched. If this is a major problem, then a third pass
can be run with different blocking variables. Errors on all three
blocking variables are unlikely.
89