Every generation of social scientists inherits a set of methodological anxieties from the generation before it, and then discovers a new set of its own. The generation trained in the 1990s inherited debates about the validity of case studies, the meaning of statistical significance, and whether the distinction between quantitative and qualitative research was methodologically fundamental or merely stylistic. The generation working today has inherited all of those debates, largely unresolved, and added a new one: what happens to social inquiry when the data available exceeds by orders of magnitude any individual researcher’s capacity to read, process, or even meaningfully survey it.
The honest answer, drawn from sustained engagement with four landmark methodological works — Gary King, Robert O. Keohane, and Sidney Verba’s Designing Social Inquiry; W. Phillips Shively’s The Craft of Political Research; Kosuke Imai and Lori Bougher’s Quantitative Social Science; and Martha Finnemore’s National Interests in International Society — is that the expansion of available data has not solved the fundamental problems of social scientific inference. In important ways, it has made them harder to see.
The greatest crisis facing the modern researcher is not a shortage of data. It is the illusion that abundance of data substitutes for rigor in the logic of inference.
I. The Shared Foundation: One Logic Beneath Two Traditions
The most consequential argument in King, Keohane, and Verba’s 1994 work was not a claim about any particular method. It was a claim about the unity of scientific reasoning beneath the apparent diversity of methodological practice. Quantitative and qualitative research, KKV argued, however different they appear in terms of style, vocabulary, and technique, must both be evaluated against a single underlying standard: the quality of the inferences they support. The ultimate goal of social science — whether you are running a cross-national regression on seventy countries or conducting six months of fieldwork in a single village — is to use observed evidence to make reliable inferences about unobserved processes and structures in the world.
This is not an obvious or uncontroversial claim, and it was not received without resistance. Many qualitative researchers heard it as an attempt to colonize their methods with the logic of regression analysis. But the more careful reading is different. KKV were not demanding that everyone count things. They were insisting that everyone be explicit about the inferential moves they are making — about what their observations actually establish, what they do not establish, and what assumptions are required to bridge the gap. That standard applies with equal force to a case study and to a time-series cross-sectional analysis.
Shively’s contribution to this shared foundation is the concept of theoretical elegance. A good theory is not the most comprehensive theory — it is the simplest theory that achieves meaningful predictive accuracy on questions that matter. The fewer variables required to explain a phenomenon, the more leverage the theory provides, and the more clearly falsifiable it becomes. In an era when machine learning models can achieve near-perfect fit on training data by multiplying parameters without limit, Shively’s insistence on the value of parsimony is not nostalgia for simpler times. It is a principled defense of the distinction between prediction and explanation — between a model that tracks variance in the data and a theory that identifies the mechanism generating that variance.
II. The Productive Contradiction: Where Positivism Meets Constructivism
There is a category of intellectual disagreement that is generative rather than merely contentious — disagreements where neither side is simply wrong, where the tension between them is itself productive of better thinking. The encounter between the positivist framework of KKV and the constructivist framework of Martha Finnemore belongs in this category.
Finnemore’s intervention targets a deep assumption embedded in most mainstream international relations theory, whether realist or liberal: the assumption that state interests — in power, security, and prosperity — are fixed, pre-social, and exogenous to the system of international interaction. States have preferences, the standard story goes, and they pursue those preferences through strategic interaction. Finnemore’s argument, developed through careful historical case studies of how international organizations shaped state behavior, is that this story has the causation backwards. States do not arrive at the international system with fully formed interests. International social structures and norms are what define, communicate, and legitimize certain interests as appropriate for states to hold. Interests are not inputs to the international system; they are outputs of it.
The methodological implications are sharper than they might initially appear. If interests are socially constructed rather than given, then treating them as exogenous variables in a model — as the vast majority of quantitative international relations research does — is not merely a simplification. It is an ontological error that will systematically produce misleading estimates of the causal relationships the researcher is ostensibly studying.
The sharpest point of tension between KKV and Finnemore concerns what KKV call selection on the dependent variable. KKV issue a stern warning: if you select your research cases based on the value of the outcome you are trying to explain, you guarantee that your inferences about causes will be biased. Study only successful revolutions to understand what causes revolutions, and you have silently excluded from your sample all the cases where the putative causal conditions were present but revolution did not occur. The effect of the supposed cause is therefore systematically overstated.
Finnemore does not deny the force of this warning in the context of theory-testing. But she identifies an important exception. When confronted with what she calls anomalies of similarity — why do all states, regardless of culture, history, or domestic politics, suddenly build bureaucracies organized on identical Western scientific principles? — studying precisely that similarity is the necessary first move. The constant outcome is not a selection bias problem; it is the puzzle. The task is not to test a theory about variation, but to generate a theory about conformity. Method must serve the theoretical problem, not foreclose it.
Theory is not what you build after you have collected enough data. Theory is what determines where you look, what counts as evidence, and what you are capable of seeing.
III. Four Fatal Traps in Research Design
There is a certain kind of methodological error that looks entirely reasonable at the stage of design and only reveals its consequences at the stage of analysis, when it is too late to do much about it. The four errors documented across these four works are of exactly this type.
The first is the indeterminate research design — when the number of inferences the researcher wants to make exceeds the information available in the data. A dissertation that proposes to explain foreign policy behavior across fifteen countries using twelve independent variables faces a fundamental problem of degrees of freedom that no amount of sophisticated analysis can resolve. The researcher is asking more questions of the data than the data can answer. The result is a study that can gesture at correlations but cannot establish causal weights, producing conclusions in the conditional tense — might suggest, appears to indicate, is consistent with — that sound academic but represent an admission that the design was insufficient to the question.
The second error is the shallow operationalization of multidimensional concepts. Shively’s point is uncomfortable but true: ordinary language is a constant source of analytical confusion, because words that appear to name single things actually refer to family resemblances among several distinct phenomena. Power, legitimacy, democracy, development — each of these concepts carries multiple theoretically distinct dimensions that may have different causes, different effects, and different relationships to other variables. Compressing multiple dimensions into a single measurement scale does not capture the concept more completely; it creates a measurement that conflates conceptually distinct things and produces estimates that are a weighted average of relationships that may run in opposite directions. The solution is not to measure everything; it is to measure precisely the dimension that the theoretical mechanism requires.
The third error — the paired dangers of endogeneity and omitted variable bias — is perhaps the most technically discussed but also the most systematically underestimated in practice. Endogeneity occurs when the variable you have designated as your cause is itself affected by your outcome, creating a reverse-causation problem that makes it impossible to determine the direction of the relationship from observational data alone. Omitted variable bias occurs when a third factor, correlated with both your cause and your outcome, is excluded from the model — silently absorbing variance that is attributed to the included variables and producing systematically incorrect estimates. The troubling characteristic of both problems is their invisibility: a model with omitted variable bias often looks clean and well-fit, and nothing in the standard output flags the presence of the problem.
The fourth error, selection bias in data collection, is one that becomes more rather than less serious in the age of Big Data. Large administrative datasets, social media archives, and digitized historical records share a common characteristic: they were not collected for the purpose of answering your research question, and the mechanisms that determine which observations end up in them are frequently correlated with the phenomena you are studying. Data from declassified diplomatic archives is data that governments chose to release; it is not a random sample of diplomatic activity. Twitter data is data from people who chose to use Twitter; it is not a random sample of political opinion. The larger the dataset, the easier it is to mistake the systematic patterns introduced by the collection mechanism for the substantive patterns you set out to study.
IV. Artificial Intelligence, Big Data, and the Questions Machines Cannot Answer
Imai and Bougher’s contribution to this conversation is to demonstrate, persuasively and in concrete technical detail, that the modern social scientist’s toolkit has expanded to include capabilities that were not available to the authors of Designing Social Inquiry. Network analysis, natural language processing, spatial data methods, causal inference with observational data through instrumental variables, regression discontinuity, and difference-in-differences designs — these represent genuine advances in what social scientists can credibly establish from the data available to them.
The expansion of technical capacity, however, creates a new form of the old problem. When a machine learning algorithm can identify thousands of patterns in a large dataset in seconds, the researcher faces a different version of the inference challenge: not how to find patterns, but how to evaluate which patterns are substantively meaningful, which are statistical artifacts of the data collection process, and which represent genuine causal mechanisms that would replicate in new data and new contexts. An algorithm optimized for predictive accuracy will find the combination of variables that minimizes prediction error in the training data. It will not tell you why those variables predict the outcome, whether the relationship would hold under different conditions, or whether an intervention that changes one of the predictors would change the outcome in the way the model implies.
KKV’s insistence that complex events require simplification through theory — through the identification of observable implications that a specific theoretical mechanism would generate, rather than through atheoretical pattern-matching — becomes more rather than less important as the scale of available data grows. The theoretical framework is the filter that allows the researcher to distinguish signal from noise, to know in advance what patterns would count as evidence for or against a particular account of how the world works. Without that filter, Big Data generates not knowledge but an impressively large collection of correlations, most of which will not survive contact with new data.
Finnemore’s constructivist critique adds a dimension that is easy to overlook in technically-oriented methodology discussions. Machine learning models trained on historical data embed within their parameters all the systematic biases and structural constraints of the historical period that generated that data. If state interests are socially constructed — if what counts as a legitimate national interest has changed substantially over the past two centuries, and will continue to change — then a predictive model trained on past behavior is not merely forecasting the future; it is potentially forecasting a future conditioned on a normative structure that may itself be in the process of transformation. The epistemological challenge this poses is not one that better algorithms can resolve.
V. Five Principles for Research Design That Survives Peer Review
The practical application of these methodological principles is not a checklist to be completed once and set aside. It is a habit of mind that shapes every decision in the research process, from the initial framing of the question to the final presentation of results. The following five principles capture the most consequential of those decisions.
The first principle is to pursue anomaly rather than confirmation. The studies that make lasting contributions to a field are rarely those that test — yet again — a well-established theoretical prediction. They are studies that begin from an observation that does not fit: a country that developed without the institutional preconditions that the dominant theory identifies as necessary, a conflict that broke out between states whose trade interdependence should have made it prohibitively costly, a social movement that mobilized without the resource base that resource mobilization theory requires. The anomaly is not evidence that the existing theory is wrong. It is an invitation to think about what the existing theory is missing.
The second principle is to operationalize narrowly and precisely, not broadly and comprehensively. Identify the single dimension of your central concept that is directly implicated in your causal mechanism, and measure that dimension as directly as possible. Resist the temptation to build composite indices that aggregate multiple dimensions — they typically confound theoretically distinct processes and make it impossible to determine which dimension is driving the results.
The third principle is to multiply observations within the constraint of your design. When adding new cases is not feasible, disaggregate existing cases temporally, geographically, or by unit of analysis. A study of ten countries observed annually over thirty years, disaggregated to the subnational level, offers thousands of observations rather than ten — along with the statistical power to separate effects that would be hopelessly confounded in a ten-observation design.
The fourth principle is to design for falsification, not confirmation. Before collecting any data, identify the three strongest alternative explanations for your hypothesized result, and build your research design so that your data can discriminate between your hypothesis and those alternatives. A rigorous peer reviewer will do this exercise anyway; doing it first allows you to preemptively address the challenges rather than encounter them in revision.
The fifth principle is to report uncertainty honestly. Every estimate carries error. Every conclusion is conditional on assumptions. The practice of reporting confidence intervals, sensitivity analyses, and null results not only makes individual studies more credible; it contributes to the collective project of accumulating reliable knowledge by resisting the publication bias that systematically overstates effect sizes and overrepresents positive results in the published literature.
Science at its best is a social enterprise — not because we agree with each other, but because we share a common commitment to evidence and to arguments that are genuinely open to being shown wrong.
Conclusion: What Methodology Actually Is
There is a persistent misunderstanding about what methodology is for. It is not a set of rules imposed by journal editors to make research more tedious. It is not a competition between quantitative and qualitative approaches to determine which counts as real science. It is the accumulated practical wisdom of a discipline about how to avoid fooling itself — about how to build studies that can actually produce the inferences they claim to produce, rather than studies that generate the appearance of evidence while quietly failing to establish what they purport to establish.
KKV, Shively, Imai, and Finnemore disagree with each other on important points. They offer different emphases, different cautionary examples, different visions of what the most important methodological challenges are. What they share is a conviction that the quality of social scientific knowledge depends on the quality of the reasoning that connects evidence to conclusions — and that this quality is not automatic, not guaranteed by the size of the dataset or the sophistication of the software, and not something that can be outsourced to an algorithm.
Let the data speak. But first, make sure you have learned its language well enough to tell what it is actually saying.
◉ 0 Thảo luận Khoa học