Troubleshooting Experiments
Numbers on different scales
Check that the coding of variables is such that different attributes’ coefficients are comparable. For example, if dealing with the price of cars, using actual prices of, say, 100,000 and 150,000 will cause the coefficients to be very close to 0, making them easy to misunderstand. It can also cause computational problems. Better to represent these as, say, 1.0 and 1.5.
R-square is different to other programs
The r-square statistic that is presented is McFadden’s Rho, which is computed as 1 – Log-likelihood / Null log-likelihood. Q computes the Null log-likelihood under the assumption that there is an equal probability of choosing each alternative. Most other programs that estimate McFadden’s Rho compute the Null log-likelihood by fitting a model including alternative specific constants, which is more appropriate for revealed preference data. You can compute this Null log-likelihood by creating a model only containing an attribute for the alternative specific constants.
Collapsing ranks changes parameter estimates
Different parameter estimates are observed when ranks are collapsed. It is possible to collapse ranking data. For example, a ranking task can be analyzed as a choice task (Pick One) by treating the highest rank as the choice and ignoring the differences between the other ranks (this can be done by recoding the Value for each category of the ranking). Fundamentally different parameter estimates may be observed if this occurs. This is for two different reasons. First, the scale parameter may change during the ranking exercise, with people giving more precise rankings higher ranks.[1] Second, the respondents’ preferences may change during the task due to learning effects.[2] There is no obvious remedy for either of these problems.
Inconsistent standard deviations
The standard deviations shown in a table versus the shrinkage report differ. Q reports the sample standard deviation in all tables. This shrinkage computation uses the population standard deviation. To convert the population standard deviation to the sample standard deviation, multiply by (number_of_observations / (number_of_observations – 1)).
The question in the blue or brown drop-down has no data
The Outputs Tab contains the message: The question in the blue or brown drop-down has no data. Please check: the question(s)’ Values in the Variables and Questions tab, and any weights or filters.). This usually indicates a fundamental problem with the data, such as no observations, or large values (e.g., prices in thousands). If a visual inspection of the Data tab is insufficient for identifying the problem, it is often a good idea to delete attributes from the Experiment, one at a time, in the hope that this sheds light upon the cause of the problem.
References
- ↑ Hausman, Jerry A. and Paul A. Ruud (1987), "Specifying and testing econometric models for rank-ordered data," Journal of Econometrics, 34, 83-104.
- ↑ Allenby, Greg, Geraldine Fennell, Joel Huber, Thomas Eagle, Tim Gilbride, Dan Horsky, Jaehwan Kim, Peter Lenk, Rich Johnson, Elie Ofek, Bryan Orme, Thomas Otter, and Joan Walker (2005), "Adjusting Choice Models to Better Predict Market Behavior," Marketing Letters, 16 (3-4), 197-208.