Relaxing the conditional independence assumption?

[zmbc]

S4 of the appendix of the fastLink paper describes two methods for relaxing the conditional independence assumption. It looks like a simulation study of these was done, but I don't immediately see any documentation on how to do this in fastLink. Does the code exist somewhere?

[tedenamorado]

Hi,

Sure thing. To implement the second approach discussed in that Appendix, the following code should work:

## Load the package and data
library(fastLink)
data(samplematch)

matches.out <- fastLink(
  dfA = dfA, dfB = dfB, 
  varnames = c("firstname", "middlename", "lastname", "housenum", "streetname", "city", "birthyear"),
  stringdist.match = c("firstname", "middlename", "lastname", "streetname", "city"),
  partial.match = c("firstname", "lastname", "streetname"),
  cond.indep = TRUE
)

Since the first approach described in the Appendix produces similar results to the second, we focused on the second approach for implementation of fastLink. This is because it is a standard extension of the Fellegi-Sunter model.

Please, if anything, do not hesitate to let us know.

All my best,

Ted

[zmbc]

Thank you @tedenamorado!

I have a conceptual question about this that I'm wondering if you can shed some light on. It seems to me that S4.1 describes an approach that would modify both EM (parameter estimation) and prediction. It looks like the implementation of S4.2 affects only the parameter estimation. Is my understanding correct? Have you considered including interaction effects at prediction time?

[tedenamorado]

Hi @zmbc,

Both methods impact parameter estimation and, consequently, the predictions based on the Fellegi-Sunter model. The approach in S4.1 treats interactions as a linkage field comparison by combining the information from two different linkage fields. In contrast, the approach in S4.2 allows each linkage field and interaction to contribute to parameter estimation. While S4.2 complicates the EM algorithm because it is not based on closed-form solutions (though it is standard in the literature), S4.1 simplifies the EM algorithm but does not account for all possible two-way interactions.

Please, if anything, do not hesitate to let us know.

Ted

[zmbc]

@tedenamorado Apologies if I haven't been clear here. When I say "modify prediction" I mean the structure of the prediction model, not only the estimates of the parameters in it.

I'll give a concrete example. In S4.1 the agreement values of first name and date of birth are combined, so instead of having two agreement variables with 2 levels each, there is a single agreement variable with 4 levels (0-0, 0-1, 1-0, 1-1). If I understand correctly, this would result in m and u probabilities being independently estimated for each of the 4 levels. There is nothing forcing the match weight of 1-1 to equal the match weight of 1-0 + the match weight of 0-1. In fact it is probably quite a bit lower if the two agreements are correlated.

Then when predictions are made, match weights are applied according to the actual interaction pattern in each pair to predict. That is, a pair with 1-0 will get a fairly high match weight, as will a pair with 0-1, but a pair with 1-1 will get a match weight that is lower than the sum of those.

If I understand correctly, S4.2 does not do this. Instead, it would help when estimating the parameters for a match on first name or date of birth, likely lowering the match weight of both if they are correlated, but at prediction time the structure still dictates that the match weight of 1-1 = the match weight of 0-1 + the match weight of 1-0.

[leostimpfle]

I believe that fastlink's implementation of S4.2 uses the fully interacted model at prediction time.

This is based on the observation that when setting cond.indep = FALSE, the model's EM object only retains the match probabilities for each agreement pattern (stored in p.gamma.j.m and p.gamma.j.u) which are estimated using Poisson QMLE with a fully interacted model (but not the match probability of individual match fields stored in p.gamma.k.m and p.gamma.k.u, respectively):

fastLink/R/emlinklog.R

Lines 191 to 192 in da2e889

	non.matches <- glm(count ~ .*., data = data.frame(count = ((1 - zeta.j) * n.j + c), factors),
	family = "quasipoisson")

So, I don't think that with cond.indep = FALSE the model builds predictions from the probabilities of individual match fields (in your example: "match weight of 1-1 = the match weight of 0-1 + the match weight of 1-0") but predicts the match weight of 1-1 from the corresponding match pattern including the interactions between the two match fields.

Note that interactions are only enabled in the latent class of non-matches ($u$-probabilities) and that for matches ($m$-probabilities), no interactions are used:

fastLink/R/emlinklog.R

Lines 188 to 189 in da2e889

	matches <- glm(count ~ ., data = data.frame(count = ((zeta.j * n.j) + c), factors),
	family = "quasipoisson")

[zmbc]

Nice detective work @leostimpfle! When you say "fully-interacted", do you mean "all 2-way interactions"? I think that's what .*. is doing, but I could be wrong.

[tedenamorado]

Thanks @leostimpfle! All you mention is exactly correct. Note that, in terms of the log-linear model for a table of counts, we follow the work of Larsen and Rubin (2001), who claim (based on the work of Winkler) that: ``if a candidate pair is truly a match, it is possible that disagreements on fields are the result of random errors. If a pair is a nonmatch, it is more likely that disagreements occur in logical sets.'' We did not explore adding additional interactions, as the number of parameters would have grown larger.

@zmbc, you are correct, we include only two-way interactions.

Thanks for your interest in fastLink! If anything, do not hesitate to let us know.

All my best,

Ted