Quantitative Design Mostly about surveys

General Information Everything must be quantified Operationalization of measures = key Looking for relationships

Before you do anything else… What am I measuring? What are the relationships I want to understand? What do I think is influencing what? What is the unit of analysis?

From Seeing Disorder, Sampson & Raudenbush What was measured? Perceptions of disorder What are the relationships I want to understand? Race and perceptions of disorder Hypothesis: As % non-white increases, perceptions of disorder increases What is the unit of analysis? Individual level Census block group aggregated up to neighborhood cluster

Important concepts Reliability = consistency Validity = truthfulness or accuracy Great resource = Research Methods Knowledge Base GOAL = accuracy and the right amount of precision

Variables How do you measure what you want to measure? Do your variables do that? What form of data is it? Nominal (aka categorical) Ordinal Interval Ratio

How do you measure what you want to measure? Survey of neighborhood residents Six questions measuring perceptions of physical and social disorder “How much of a problem is…” Response alternatives: “a big problem”,“somewhat of a problem” ,“not a problem” What form of data is it? Ordinal From Seeing Disorder, Sampson & Raudenbush

Scales Scales measure the intensity, direction, or level of some concept (or construct) that exists on a continuum

Scale

Scales and Indexes Indexes are composites of multiple indicators Calculated by combing several distinct indicators (variables) of a concept into a single score

Index

Index

Surveys – what can they do for you? Behaviors How frequently do you brush your teeth? Attitudes/beliefs/opinions Are black people getting too much help from the government? Characteristics Are you married? How old are you? Do you wear an eye patch? Expectations Do you think the neighborhood population will change? Self-classification Do you consider yourself a good person or a jerk? Knowledge What percent of the federal budget goes to foreign aid?

Surveys – what they can’t do Answer “why” questions

Surveys – Key Principles Consider alternative explanations Planning Pilot testing

Surveys – alternative explanations Need to be able to prove yourself wrong Need to include control variables AVOID: How does the bus connection to the mall lead to higher crime? BETTER: Is there a relationship between the bus connections to the mall and crime in the mall? BEST: What are the characteristics of individuals who commit crimes at the mall? Crime is known to be related to: Poverty Age Sex Previous criminal activity

Surveys – Planning The instrument Mode of administration How to record data How Who QC Who to sample How many to sample How to access your sample How to obtain consent How data will be analyzed

Surveys – Pilot testing Test the instrument Test the mode of administration Support validity and reliability REMEMBER: Sloppy survey design and administration = bad results = embarrassing conference presentation

Surveys – Writing Questions Primary goals Avoid confusion Questions are constructed with your sample in mind

Surveys – Writing Questions Jargon, slang, acronyms FUN FACT: ~46% US adults have low rates of literacy. About 1/5 of US adults are functionally illiterate. What to avoid AVOID: Should women be allowed to seek a D&C without permission from her husband? BETTER: Dilation and curettage is a medical procedure that can be used to end a pregnancy. Should doctors obtain the husband’s permission for this procedure before performing it on women?

Surveys – Writing Questions Ambiguity and Vagueness Ambiguity = word or phrase has two or more meanings Vagueness = not clear what the word or phrase means What to avoid AVOID: Do you exercise regularly? BETTER: How many days last week did you do any form of exercise?

Surveys – Writing Questions Emotional language What to avoid

Surveys – Writing Questions Emotional language What to avoid

Surveys – Writing Questions Prestige bias What to avoid AVOID: Dr. Sanjay Gupta, the Surgeon General for the US, believes that meditation can help heal the body. Do you agree? BETTER: Tell me how much you agree with the following statement: Meditation can help heal the body.

Surveys – Writing Questions Double barreled questions Two or more topics in one question PRINCIPLE: each question is about one and only one topic What to avoid AVOID: Do you support or oppose raising social security benefits and Medicare benefits? BETTER: Do you support or oppose increasing social security benefits? Do you support or oppose increasing Medicare benefits?

Surveys – Writing Questions Leading or loaded questions Questions that are phrased or stated in a way that signals to the respondent that a certain answer is preferred or better. What to avoid AVOID: Did you do your patriotic duty and vote in the last election? BETTER: Some people were unable to vote in the last election. Did you vote in November?

Surveys – Writing Questions Questions beyond the respondent’s ability to answer Recall of past events Recall of past behavior PRINCIPLE: Ask questions your respondent’s can answer What to avoid AVOID: How much gas did you purchase last year? BETTER: How many gallons of gas did you buy the last time you went to the gas station? How many gallons of gas do you usually buy?

Surveys – Writing Questions False Premises PRINCIPLE: Don’t start a question with a premise that your respondent may disagree with. What to avoid AVOID: The city of Boston spends too much money on services for the elderly. Would you rather have more money go to public schools? BETTER: Please rank the following spending priorities: services for the elderly, police and fire departments, public schools. AVOID: When did you stop beating your spouse? BETTER: Have you ever slapped, punched, or hit your spouse?

Surveys – Writing Questions Double Negatives PRINCIPLE: Don’t confuse your respondents. What to avoid AVOID: Do you agree with those who do not want to extend the Greenway to Constitution Beach? BETTER: There is a proposal to extend the Greenway to Constitution Beach. Do you think this is a good idea, a bad idea, or do you have no opinion?

Overlapping or unbalanced responses Mutual exclusivity – responses do not overlap Exhaustive – all reasonable possibilities included Balanced – all alternatives available, good and bad Surveys – Writing Questions What to avoid

Surveys – Writing Questions Beliefs as Real If your goal is to measure an actual empirical relationship, do not ask questions about whether people believe there is a relationship. This does not apply if you are measuring people’s beliefs. What to avoid AVOID: Do you think more educated people smoke less? BETTER: What is your level of education? Do you smoke cigarettes?

Surveys – Sensitive and Threatening Questions People have a tendency to underestimate things that make them look bad, that they are ashamed of, are embarrassed about, or believe violate social norms People have a tendency to overestimate things that they thing make them look good or that support social norms. FUN FACT: Cross-national studies have found that men and women are shorter and heavier than they think they are.

Surveys – Sensitive and Threatening Questions Surveys have been found to be useful in obtaining information about sensitive subjects Most sensitive topics: Masturbation Sexual activity Drug use/ abuse (illicit and prescribed) Least sensitive topic: Sports activity

Surveys – Social Desirability Bias Giving socially acceptable answers that deviate from the truth How to deal with: neutralize the question Make the attitude or behavior less objectionable Provide “face saving alternatives”

Surveys – Open Vs. Close Questions OPEN: What is your current marital status? (Write your answer below)   CLOSED: What is your current marital status? (Select one.) Single Married Divorced Separated Widowed Each has trade-offs

Surveys – Open Vs. Close Questions How to decide? What will meet the goals and purpose of your study? (methodological congruence) What resources do you have?

Surveys – The Middle or Null Response This includes the neutral, not applicable, and don’t know responses If you do not provide these alternatives You do force people to choose Some will not answer the question at all A small number will write their own answer If you do Allows some people to evade making a choice

Surveys – The Middle or Null Response Argument in favor of including these: It’s important to know what people do not have an opinion about Argument against People are forced to make choices even when they don’t have opinions and it is important to know how they would choose What should you do? GOAL: methodological congruence

Surveys – Wording Structure questions to be clear Make sure that people know what the word means FUN FACT: some portion of the population does not know what the word “impartial” means. Some terms that evoke predictable reactions

EXAMPLE OF CBS/NYT POLL Source: CBS/NYT Poll. (2010). Gays in the Military, February 5-10, 2010. From http://www.cbsnews.com/htdocs/pdf/poll_021110_2pm.pdf?tag=contentMain;contentBody

Surveys – Sequence Surveys have a beginning, middle, and an end Start with easy and interesting questions Difficult, sensitive, or threatening questions in the middle Easy but boring questions at the end Group questions by topic or theme

Surveys – Ordering Effects Previous questions can influence the responses to later questions

Surveys – Design Layout is important Boxes or brackets to check Numbers to circle PRINCIPLE: Self-administered surveys should be easy to read and easy to complete

Surveys – Using surveyors Training is key FUN FACT: Skimping on training is wasteful and leads to bad results and embarrassing conference presentations Appearance of surveyor matters