Science Practice 2: Research Methods and Design

Original hypothetical retrieval experiment: researchers ask whether practicing recall improves one-week vocabulary memory relative to rereading. Eighty adult volunteers aged 18–24 respond to an advertisement at one university. A computer randomly assigns 40 to retrieval practice and 40 to rereading. Everyone first studies the same 24 unfamiliar word–meaning pairs for four minutes. For the next 12 minutes, retrieval participants see word cues, try to produce meanings, and then receive the correct answer; rereading participants see the complete pairs repeatedly. Both groups work in the same quiet room with identical materials and time. One week later, everyone writes meanings for all 24 cues without notes, with one point for each correct meaning (0–24). A scorer who does not know condition uses a predefined answer key. All 80 complete the test. Mean recall is 18/24 after retrieval and 14/24 after rereading. These are invented study summaries, not actual participants or a published finding.

This is an experiment under the AP framework because researchers manipulate the practice condition and randomly assign participants to conditions. Retrieval versus rereading is the independent variable; later recall is the measured outcome. The word ‘experiment’ in a title, the use of a laboratory, numerical scores or two groups would not establish those two defining design features by themselves.

Suppose volunteers instead choose whether to use retrieval or rereading. The practice conditions still differ, but random assignment is missing. Prior motivation or confidence could influence the choice and the score. Under the CED's instructional distinction, this is non-experimental rather than the experimental design described above. Identify the design actually used; proposing a better experiment is a separate task.

A study can combine methods. An experiment might also interview participants about their experience. The interview instrument supplies qualitative data; it does not erase the manipulated condition and random assignment. Conversely, a survey that compares self-selected groups remains non-experimental even if it reports a numerical difference. Classify the design from the procedures and classify the measurements from the information collected.

Non-experimental methods observe or synthesize information without the manipulation-plus-random-assignment combination. They are useful when a question concerns existing experiences, unusual cases, natural settings or patterns across prior studies. They do not automatically support a claim that one measured variable caused another.

A case study can give intensive detail about an unusual memory difficulty through interviews, task scores and history. That depth may suggest explanations to test later, but a single case is not a representative sample of everyone with similar difficulties. Naturalistic observation records behavior in its usual setting without an assigned intervention; it can reveal patterns that a laboratory task misses, but observers need clear coding rules and must consider reactivity and access to the setting.

A correlational study measures variables as they already occur and asks how they vary together. A meta-analysis combines results from multiple studies using explicit inclusion criteria and statistical synthesis; it does not mean researchers newly assigned all participants from those studies to one experiment. Its conclusion depends on the designs, quality, comparability and possible publication bias of the included studies. The examples below are invented study designs, not cited publications.

Return to the 80-person vocabulary experiment above. The hypothesis is that adults assigned to retrieval practice will recall more meanings one week later, on average, than adults assigned to rereading under the matched conditions. It is falsifiable because equal or lower average recall after retrieval would fail to support this directional prediction. ‘Retrieval works in an invisible way no outcome can detect’ is not falsifiable by this study.

An operational definition specifies exactly how a variable is manipulated or measured. Here ‘practice condition’ means the 12-minute cue–attempt–feedback procedure versus 12 minutes seeing full word–meaning pairs after common four-minute study. ‘Recall’ means correct meanings written for 24 word cues after one week, scored 0–24 with a fixed answer key. The definitions identify actions, timing and scoring well enough for another researcher to reproduce the procedures. Merely defining memory as retention is insufficient.

The independent variable is practice condition, with retrieval and rereading levels. The dependent variable is the one-week recall score. A confounding variable differs systematically with condition and provides an alternative explanation for its outcome difference. If retrieval participants practiced 20 minutes while rereading participants practiced 12, time and practice type would be tangled. Holding time constant fixes that design flaw; random assignment alone cannot untangle a procedural difference deliberately built into conditions.

Distinguish a confound from any uncontrolled variation. Some participants may have better prior vocabulary knowledge, but random assignment is intended to distribute such characteristics without systematic allocation to one condition; it does not guarantee perfect balance in every sample. A researcher should still evaluate whether the procedure created a systematic competing difference. Operational clarity also does not guarantee that a narrow word test captures every aspect of memory.

The sample is the 80 adults who volunteered. The population is the wider group to which a researcher wants to apply a conclusion—perhaps all adult learners. The university advertisement yields a convenience sample with volunteer self-selection; a computer’s later assignment to conditions does not change that recruitment method. People outside that university, outside ages 18–24 or unable to attend the session are absent.

Random sampling would select people by chance from a defined population frame. It aims to reduce selection bias and improve population representation; it is different from chance assignment within a sample. A sample may be unrepresentative because the frame excludes people, selected people do not respond, or access barriers prevent participation. Large size alone does not fix systematic exclusion. Convenience sampling is easier to conduct, but broad generalization then needs particular caution.

A worked evaluation: the results are most directly relevant to volunteers aged 18–24 from the sampled university doing this word-pair task. They cannot by themselves establish an equal benefit for children, older adults, people with different educational experiences or learners using other languages. Age, university context and willingness to volunteer are specific participant characteristics supporting this limitation; simply writing ‘only 80 people’ would not explain who is underrepresented.

Representation also affects what research discovers. If accessible recruitment, language options or varied ages are absent, researchers may miss differences in how the task is experienced or how the intervention works. Including people appropriately requires usable materials, fair participation and analyses suited to the question, not merely a diverse-looking photograph. Increasing representation strengthens the relevance of outcomes; it does not guarantee that every subgroup responds identically or that a convenience sample becomes random.

In the vocabulary experiment, retrieval is the experimental condition and rereading is an active comparison or control condition. The control group receives an actual study activity, allowing comparison of retrieval with rereading rather than with doing nothing. Both groups receive the same initial study, words, time and test. The random assignment process reduces systematic preexisting differences; the control procedure supplies a benchmark for interpreting the outcome.

There is no placebo in this vocabulary experiment. A placebo is an inactive treatment used to help separate an intervention's specific effects from participants’ expectations of treatment. In a separate hypothetical adult memory-pill trial, a committee-approved study randomly assigns consenting volunteers to an active pill or a visually identical inactive pill, then compares performance on the same task. An improvement after an inactive pill can reflect expectations or other nonspecific processes, so comparison with a placebo helps evaluate an active treatment. This teaching example recommends no treatment and reports no actual result.

In conventional single-blind treatment procedures, participants do not know which treatment they receive. In double-blind procedures, participants and the relevant researchers interacting with or assessing them do not know allocation. This reduces expectation-related behavior and experimenter bias, such as unintentionally giving extra encouragement to an expected-to-improve group. State exactly who is unaware rather than assuming a label fits every design.

Vocabulary participants can tell whether they are recalling or rereading, so this experiment is not participant-blind or double-blind. Only the scorer is blind to condition. Blind scoring helps prevent differential credit for answers; it cannot erase participants’ expectations. Standard instructions and equal contact help further. Social desirability bias is a participant tendency to give a socially approved answer: if a researcher praises retrieval as smart, participants may overreport using it on a survey. Blinding alone does not solve every self-report distortion.

The vocabulary experiment’s primary outcome is quantitative: the number of correct meanings, 0–24. A supplemental question—‘How confident are you that you will remember the words?’ with responses 1 = not at all to 5 = extremely—produces a numerical ordered rating, commonly called a Likert-type item. It measures reported confidence rather than objective memory. Calling it quantitative does not make each step an equal amount of felt confidence.

Researchers could also conduct a structured interview, asking every participant the same open questions in the same order, such as ‘Describe what you did when you could not remember a meaning.’ The spoken descriptions provide qualitative data. They can explain strategy use or frustration that a score misses. If researchers later code descriptions into category counts, they should show the coding rules; qualitative descriptions and derived quantitative counts answer related but different questions.

Peer review asks other specialists to scrutinize methods, analyses and conclusions. Reviewers might question whether the scoring key accepts equivalent meanings or whether the active control receives equal time. Revision can improve the report, but peer review does not prove a result true. Replication repeats the procedures with new observations to test whether the finding recurs; systematic extensions can examine other ages or languages.

If later experiments reproduce the effect only with difficult vocabulary, the conclusion should become more conditional rather than remain ‘retrieval always works.’ Researchers compare results and inspect design differences, precision, measurement and participant contexts. A single failed replication does not automatically erase all prior evidence; a single successful one does not remove every limit. Experimental conclusions evolve through this accumulated scrutiny.

Original hypothetical sleep–recall study: researchers ask whether usual sleep duration is associated with memory. They recruit 160 adults aged 18–65 through a community-library mailing list; participation is voluntary, not random selection. For seven mornings each person reports the previous night's sleep in hours, and researchers average those seven entries. At the end of the week, everyone reads the same 24 word–meaning pairs for four minutes and completes a cued-recall test 20 minutes later, scored 0–24 with an answer key. The survey also asks how often participants studied that week (1 = never, 5 = every day) and asks them to describe factors interfering with sleep. No sleep amount is assigned or manipulated. The reported correlation between mean sleep hours and recall is r = +0.45. The study does not report a p-value. Consent, withdrawal without penalty and coded confidential records are described.

The hypothesis is that higher mean reported sleep hours will be associated with higher recall scores. It is falsifiable: no positive association, or a negative one, would fail to support this directional prediction. Variables of interest are mean reported hours per night and the recall score. Neither is a manipulated independent variable; do not call one an experimental treatment merely because it appears on the horizontal axis.

The operational definitions specify the seven-night averaging window, self-report method, common learning period, 20-minute delay and 24-point scoring key. A replication needs the same reporting prompts and task conditions, not the vague instructions ‘measure sleep and memory.’ Self-reported sleep may differ from objectively recorded sleep; the defined procedure is replicable but its accuracy still needs evaluation.

The positive r = +0.45 indicates that people reporting more sleep tend to recall more meanings in this sample. Directionality remains unresolved: sleep could influence learning, or difficulties with learning and related worry could affect sleep. A third variable such as stress, work schedule or health could influence both. The design does not randomly assign sleep or isolate it from those influences, so the association alone cannot establish that adding an hour of sleep causes a specific increase in recall.

A justified conclusion states the measured association and population limits. A stronger causal test would require a suitable ethical manipulation and random assignment, but redesigning the study is not the same as evaluating the design already supplied. Not every question about real-world sleep can safely or appropriately be turned into an experiment. Non-experimental research can still contribute useful description and prediction.

In the sleep–recall study, hours per night, 0–24 recall and the 1–5 study-frequency item are quantitative. The open question about sleep interference supplies qualitative descriptions. A survey is a technique for eliciting reports, not automatically an experiment or automatically qualitative. Read the response format and content of each measure.

Compare ‘How often did you irresponsibly stay awake instead of sleeping properly?’ with ‘During the past seven nights, how many hours did you sleep each night?’ The first wording is loaded: it supplies a negative judgment, invites a socially acceptable answer and does not define ‘properly.’ The second states a time window and a concrete unit, although remembering hours may still be difficult. Asking ‘How satisfied are you with sleep and studying?’ also combines two topics; separate them to know which judgment a rating represents.

Self-report bias can arise through inaccurate recall, selective reporting or expectations, even without deliberate dishonesty. Social desirability is one reason for distortion: someone may report more sleep or studying to appear responsible. A confidential, neutral survey and shorter reporting intervals may reduce these pressures, but cannot guarantee accuracy. An anonymous survey can reduce identification concerns, yet the recurring study uses coded confidential records to link the seven days and test; it is not anonymous to researchers holding that key.

Qualitative responses may reveal work shifts, caregiving or noise that fixed-choice items miss. Use consistent prompts and a transparent coding process if summarizing themes. Do not assign invented numerical meaning to every description or generalize a vivid quote to all participants. The format chosen should fit the question, and its limitations must travel with the conclusion.

The sleep study’s library mailing-list volunteers may differ from adults who lack library access, cannot read the questionnaire language or work during testing hours. Recruitment needs to fit the intended community and provide appropriate access. Representation changes which sleep circumstances become visible; omitting shift workers, for example, can hide an important source of variation. It also affects how reasonably findings transfer to adults outside the sample.

Peer reviewers could ask whether missing diary days were handled consistently, whether the survey wording led respondents, and whether the correlation was interpreted causally. Those are non-experimental methodological questions, not a demand that every study contain treatment groups. Reporting the procedures and limitations lets others evaluate the result.

A replication could recruit another community using the same diary and recall procedures. If the association differs, researchers should examine sample context, measurement, range of sleep values and other plausible explanations before changing the conclusion. A broader replication with varied language access or shift schedules may reveal that the original relationship was context-dependent. Peer review and replication refine non-experimental conclusions as well as experimental ones; neither turns the original association into a randomized causal test.

Ethical evaluation asks whether the procedures described protect participants and support informed voluntary participation. A worthwhile question and an interesting result do not excuse preventable harm. Institutional review provides an independent examination of proposed risks, procedures and protections before a study begins, with continuing oversight where appropriate. Human research review and the appropriate institutional animal-care review serve related protective purposes but examine different needs.

Ethics extension to the hypothetical vocabulary experiment: before recruiting, an institutional review board evaluates the mild learning task and data safeguards. Adults receive a plain-language description of activities, time, possible frustration, privacy protections and the right to stop without losing the promised participation compensation. They ask questions and agree before beginning. Researchers offer breaks and stop a session when a participant requests it. This supplies concrete evidence of informed consent and harm protection rather than merely asserting that the study is ethical.

For children or others unable to provide legally valid consent, appropriate guardian permission and the participant’s informed assent may be required. Assent is an understandable affirmative agreement, not silence or failure to resist; the exact protections depend on the population and circumstances. A child who is told to participate by a teacher has not thereby freely assented. Vulnerability calls for accessible explanations, careful recruitment and protection from pressure, not automatic exclusion from every study.

For a hypothetical animal-learning study, the institutional animal-care process should evaluate necessity, alternatives, humane housing and handling, appropriate species-related care and minimized distress. Human informed consent is not something nonhuman animals can provide. An ethical review must match the population and procedures; human consent paperwork is not a substitute for animal welfare protections. This is curricular analysis, not a legal protocol for conducting research.

In the vocabulary experiment’s ethics extension, the research team uses participant codes rather than names on answer sheets, stores the code–name key separately with restricted access and reports only group summaries. This is confidentiality: researchers may know identity but protect it. Anonymous information cannot be linked to identity by the researchers. Because the team retains a key to arrange follow-up testing, calling these records anonymous would be inaccurate.

There is no deception or confederate in the vocabulary study. In a separate hypothetical social-influence study, consenting adults are told they will discuss neutral choices in a group. With prior institutional approval for limited deception, one apparent participant is a trained confederate who gives standardized answers; the true focus on conformity is initially withheld. A confederate is a person working with researchers while appearing to be an ordinary participant. Deception concerns misleading information or withholding the true purpose, not simply failing to disclose individual scores publicly.

Approval for that separate scenario depends on justification and safeguards; it is not permission to expose people to arbitrary distress or deny withdrawal. The researcher must consider whether less deceptive methods could answer the question. At the end, an appropriate debriefing explains the real purpose, identifies the confederate’s role, corrects misconceptions, answers questions and addresses participants’ reactions and applicable data-withdrawal options. A generic ‘thank you’ does not explain the deception or its meaning.

A worked ethical judgment: publishing identifiable interview excerpts without the agreed privacy protections violates confidentiality even if consent to participate was obtained. Fix the disclosure procedure and honor the consent terms; do not assume the original consent covers any later use. Conversely, coded confidential records are not evidence of a violation merely because they are not anonymous. Evaluate the safeguard that was promised and actually applied.

The Article Analysis Question integrates several practices: identifying a method, stating an operational definition, interpreting a statistic, examining ethics and generalizability, and using results in argumentation. The examples in this guide demonstrate individual skills; they are not a complete six-part AAQ response or a promise of a score.

The current Part D asks you to identify an ethical guideline described in the study and explain how the researchers applied it. The October 2025 clarification and updated CED scoring examples require both an accurate guideline and an accurate application from the supplied summary. Choosing a safeguard researchers ought to have used but that the source never describes does not answer this task. Broader Skill 2.D can evaluate missing safeguards or violations; AAQ Part D has this more specific prompt.

For the complete vocabulary study plus ethics extension above, a justified Part D demonstration is: ‘The researchers used informed consent by explaining the activities, possible frustration and withdrawal without loss of compensation before the adults agreed to participate.’ The guideline is named, and the procedure demonstrates its application. An alternative is confidentiality, explained through the coded answer sheets, restricted identity key and group reporting. You need one accurate described guideline for this task, not an invented list of every guideline.

A weak response says ‘They were ethical’ and supplies no guideline or procedure. Another says ‘They should have used anonymous data,’ which proposes a change rather than describing an applied guideline and also ignores the confidential follow-up design. Use the study’s actual details. AAQ generalizability likewise needs relevant participant information; age and recruitment context explain a population limit more clearly than a bare statement that the sample is small.