IBM® SPSS® Amos™ 28

Here is a portion of the output from the example.

With six observed variables, there are 720 possible permutations -- 719, if you don't count the permutation that leaves each observed variable in its original position. Of the 719 non-identity permutations, 15 made the discrepancy function smaller or left it unchanged. 617 of the permutations made the discrepancy function larger. 86 permutations resulted in a model for which Amos could not find a solution. As noted above, failures are to be expected in fitting a series of generally bad models. The question is, how do you classify the models for which no solution was found? Can it be assumed that each one of those models is worse than the original model? In other words, can you assume that, whenever Amos fails, it's the model's fault rather than Amos's?

Experience shows that Amos's failures to find solutions are almost always due to bad models (or samples that are too small). But not always. Therefore, there may be an objection to lumping the 86 permutations that produced an unfittable model together with the 617 permutations that produced a worse fitting model, on the grounds that doing so could result in an overcount of the number of permutations that make the model worse.

With these considerations in mind, Amos follows the convention that unfittable models are "worse than" the model being evaluated. Then out of 720 permutations (including the identity permutation), there are 16 permutations that produce a model that fits as well as or better than the original model. (The original model itself is one of those 16.). In other words, if you picked a model at random out of those generated by permuting the observed variables, there is a probability of 16/720 = .022 of getting a model as good as the one that Jöreskog and Sörbom proposed.

It is possible for an Amos solution to be inadmissible or to consist of an unstable linear system, although neither of these problems arose in the present example. There needs to be a policy on permutations that produce a model with a lower discrepancy function than was obtained for the original model, but for which an inadmissible solution or an unstable system occurs. Amos adheres to the following policy. First of all, if the original model results in an inadmissible solution, Amos disregards the admissibility status of estimates for models that are generated by permutations. Also, if the original model results in an unstable system, Amos ignores any instability that occurs in linear systems that result from permutations. If the original model yields an admissible solution with a stable system of linear equations, Amos reports the number of permutations that lower the discrepancy function while producing an inadmissible solution or an unstable system, and follows the convention that such permutations are harmful (i.e., that they make a model worse).

The frequency of inadmissible solutions and unstable systems is summarized as follows for the present example.

Of the 15 permutations that resulted in a discrepancy function that was as good as or better than that of the original model, all were in fact exactly as good - none were better. Examination of the output from the PermuteDetail method reveals that these 15 models are equivalent to the original model in the sense of Stelzl (1986), Lee and Hershberger (1990) and MacCallum, et al. (1993).

In principal, it would be possible to reduce the computational requirements of the permutation test by fitting one representative model from each set of equivalent models. Amos does not do this, however. More importantly, the fact that the "permuted" models come in clusters of equivalent models has a bearing on the interpretation of the permutation test. In the current example, for instance, the proportion of permuted models that fit as well as or better than the original model cannot take on just any of the values 1/720, 2/720, 3/720,.... Instead, the proportion is restricted to the values 16/720, 32/720, 48/720,.... The number of possible p values is still 720/16 = 45, and so it remains an interesting question what the value of p is. However, a serious problem arises when the number of permutations that leave the fit of the model invariant is very large, so that the number of distinct discrepancy function values that can occur is very small. To take an extreme case, consider the common factor model with one common factor, and no parameter constraints other than those required to make the model identified. No permutation of the observed variables will affect the fit of the model, and it will not be possible to apply the permutation test in a meaningful way.