Validity & Reliability and the control of confounding variables Module 6

Validity - relative accuracy/ correctness Internal Validity - extent to which a set of research findings provides compelling information about CAUSALITY External Validity - extent to which set of research finding provide an accurate description of what typically happens in the real world generalizability to people   generalizability to situations

Validity - relative accuracy/ correctness (continued) Construct Validity - extent to which IV and DV in study truly represent the abstract, hypothetical variables of interest Conceptual Validity - how well a specific research hypothesis maps onto the broader theory that it was designed to test

Reliability – repeatability quantified by a statistic Inter-rater Reliability – degree to which different judges independently agree on an observation. Internal Consistency Reliability – degree to which all observations in a multiple item measure behave the same way. Test-retest reliability – degree to which items on a score correlate posively with each other over time

Control of Confounds/Threats to Validity Cook and Campbell (1978) Confounds versus Noise People are Different Individual differences Selection bias Nonresponse bias

Control of Confounds/Threats to Validity (continued) People Change History Maturation Regression to the mean

Control of Confounds/Threats to Validity (continued) Your experiment changes people Sequence effects Experimental mortality

Control of Confounds/Threats to Validity (continued) Reactivity Biases Participant expectancies Participant reactance Evaluation apprehension

Control of Confounds/Threats to Validity (continued) Extraneous variables with your treatment Experimenter bias The placebo effect – subject beliefs Spontaneous remission

Design Critique Example A researcher wants to know if priming certain ideas or constructs would affect people’s behaviors to match the primed construct. In a previous study, they found that priming words related to rudeness made participants interrupt and experiment more often. They wanted to see if priming words related to old people (e.g., old, Florida, wrinkle, etc) made them act older. The first participants to arrive at the lab were assigned to the control group. The last participants to arrive at the lab were assigned to the treatment group. The control group was primed with neutral words, and the treatment group was primed with the old words. They were told that the experiment was over, but in fact a confederate was waiting down the hall to time how fast people walked down the hall. It turned out that the participants exposed to the old words walked more slowly than the participants exposed to the neutral words. Were there any confounds that could have contributed to these results?

Answer Significant results (the experimental group walked slower)– but where there any confounds Yes! There were potential confounds! All of the early arrivers were to the control group (neutral words). These people might have already been fast walkers – that’s why they got there first! The results don’t really mean anything because of this confound and lack of random assignment. Also, they could have had a third group that read neutral words to clarify further.