Producing Procedural Inequality Through the Empirical Turn

Open PDF in Browser: Danya Shoicair Reda,* Producing Procedural Inequality Through the Empirical Turn

---

Procedural rulemaking and scholarship have taken an empirical turn in the past three decades. This empirical turn reflects a surprising consensus in what is otherwise a highly divided field and an inherently adversarial system. Because procedural rules distribute legal power in society, they invariably raise questions about who should have access to courts, information, and the means to defend one’s legal rights. While debate rages about these normative commitments, procedure has developed a surprising epistemic agreement on empiricism, with its promise of rising above these competing interests with data. In procedure, the turn toward empiricism has become a strategy for avoiding contentious political questions, or at least obscuring them.

However, the dodge fails. Procedural reform and debates remain as polarized as ever. To understand why, this Article examines the use of empirical data in debates over the recent amendments to the Federal Rules of Civil Procedure on discovery and how that reliance on data served to mask rather than reveal the reality of contemporary discovery and litigation in the federal courts. This examination illuminates that procedural reformers’ insistence on neutrality, and the empiricism that accompanies it, shields rulemakers from understanding the unequal effects of their amendments and the ways that broader social inequality are reproduced and heightened in the courts.

This Article argues that proceduralists should place the fact of conflicting interests front and center in its debates, both because our system is designed for adversarial engagement, and because the attempt to ignore conflicting interests under the guise of empiricism leaves procedural reform privileging some interests over others without public deliberation.

What democracy requires is public debate, not information. Of course, it needs information, too, but the kind of information it needs can be generated only by vigorous popular debate. We do not know what we need to know until we ask the right questions, and we can identify the right questions only by subjecting our own ideas about the world to the test of public controversy. Information, usually seen as the precondition of debate, is better understood as its by-product.

– Christopher Lasch^[1]

Introduction

This Article relies on a premise that should be obvious but will come across today as counterintuitive. Ours is an adversarial system. We invite parties to bring adversarial controversies to the courts, to wield civil process self-interestedly in order to vindicate their rights in a satisfactory settlement or judicial disposition. Yet when it comes time to think about the healthy functioning of the rules governing the adversarial system, the role of self-interest is ignored, denied, or decried. This suppression of even the fact of self-interest happens at the hands not only of policymakers—the judges, professors, and attorneys charged with drafting amendments to procedural rules—but also of legal scholars devoted to studying and understanding the interaction of civil litigation with the rules.

Rulemakers and scholars have, over the past three decades, increasingly turned to data to drive their debates about procedural reform. While empirical analysis can unearth inequities, it is no guarantee of “neutrality,” and its use does not alone ensure sound and equitable decision-making. It can also mask or deny the roles of subjectivity and interests. This Article reveals how reverence for empiricism obscures the inherently subjective aspects of procedural rules in an adversarial system. Rather than deny subjectivity under the cloak of empirical data, procedural reform in the information age should acknowledge and embrace both. Doing so will increase the legitimacy of the process and improve its substance.

Procedural rules regulate the playing field on which legal rights are contested. A simple change such as requiring a more stringent standard before a party can obtain discovery will have material effects on the public—hindering some parties’ ability to assert their rights more than others, and conversely, protecting some parties from legal consequences. Individuals who might come to court seeking redress against a company for workplace discrimination, or who might sue the government for civil rights violations, are unlikely to have in their possession the evidence to prove unlawful discrimination or disparate impact of a governmental policy. They will need discovery in order to prove their claim and protect their rights. On the other hand, the government or large companies are more likely to possess information, and so stand to benefit from any heightened barrier to discovery. The rules can serve to even the playing field. But they just as easily can reinforce inequality, making it harder for those with fewer resources to vindicate their legal rights.

This Article argues that procedural reform, and its attendant literature, should place the fact of conflicting interests front and center in its debates. Without recognition of this conflict, procedural reform will privilege some interests over others without public deliberation. There is no way to avoid privileging some sets of interests in any system designed to resolve adversarial conflict in an unequal society, and therefore such privileging must be done deliberately and deliberatively. That we do not do so, and instead generally insist on neutrality, today contributes to and paradoxically increases the contentiousness of debates over procedural reform. The assertion of neutrality heightens the sense that improper interest is dictating the direction of procedural change.

Today, procedural policy aims to use empirical data to guide policy change. The use of empirical data in procedural rulemaking, including by the Advisory Committee on Civil Rules (“Advisory Committee” or “Committee”) charged with reviewing and amending the Federal Rules of Civil Procedure (the “Rules”), is itself a response to criticism that procedural rulemaking was occurring in a vacuum, indifferent to the empirical realities of civil litigation.^[2] For years, rulemakers were criticized for perceived bias. Scholars demanded that procedural reform take a more empirically minded approach.^[3] The Advisory Committee has responded in kind. Today, the Rules Committee of the Judicial Conference (“Judicial Conference”) justifies the rulemaking process in significant part based on its unique ability to amass empirical data about the litigation system.^[4]

To bring this empirical epistemic approach into focus, this Article will examine three aspects of the Advisory Committee’s method in its recent major reforms to discovery: the miscategorization of quantitative data, the misinterpretation of that data, and the avoidance of systematic collection of qualitative data. Before turning to the Advisory Committee’s approach to discovery data, Part I introduces the broader project, of which this Article is the capstone. That project explores the relationship between empiricism and inequality in civil procedure reform and explores the ways that procedural commentators have, sometimes inadvertently, contributed to the dynamic of inequality. Part II explores the empirical turn in procedure and suggests ways that this turn has been insufficiently considered. It is not only policymakers and the Advisory Committee that are operating with an under-theorized conception of the “empirical” in procedure, but scholarship also suffers from similar limitations. This Part investigates, more broadly, what it means to develop knowledge of the civil justice system.

Part III examines the Advisory Committee’s approach to empirical data. It contends that the Advisory Committee miscategorizes and misinterprets empirical data in systematic ways, not because of a conscious desire to craft facts to match preexisting policy commitments, but rather due to an unwillingness to confront their active role in shaping particular outcomes. The miscategorization is a central feature of the epistemic error; the Advisory Committee recognizes only two possible forms of information about civil litigation: empirical and other.^[5] The Advisory Committee expresses great interest in a reform effort informed by empirical research, but its 2015 amendment process suffers from category error and misinterpretation of data. Part III describes the Advisory Committee’s struggles to identify research that is truly empirical and its further difficulties in properly interpreting the empirical data it does have.

Part IV examines how the commitment to neutrality manifests itself in a skepticism of, or even hostility to, visible interests as manifested in strategic behavior by attorneys or parties. Failure to acknowledge the systemic quality of such behavior—indeed, its expectation and necessity in an adversarial system—leads the Advisory Committee to identify strategic behavior inconsistently. The Committee identifies only some strategic behavior—that which is engaged on behalf of non-normative interests or those interests that are deemed sectarian in our current society and economy. The Committee remains blind, however, to other strategic behavior operating in plain sight. Consequently, the Committee avoids any direct examination of or effort to classify which types of strategic behavior are bad (if any) and how best to limit them.

Part V turns to the ways that epistemic commitments to empirical research have actually limited the Advisory Committee from gathering valuable qualitative research, as it is not generally recognized as a category of empirical research at all. Because qualitative assessments typically take the form of impressionistic observation, the Advisory Committee misses out on an opportunity to tap a crucial resource for understanding the operation of rules. To do so, however, would require the Committee to take control over the information gathering process, to structure its inquiry, and to carefully consider how to obtain the information it seeks. Part V identifies sociological factors of litigation, derived from existing empirical research, to suggest methods for developing qualitative data on civil litigation.

The analysis in Part VI suggests that rulemaking desires for procedural neutrality have shaped the Advisory Committee’s very formulation of the litigation system and how information can be developed to understand how the system is operating. Unfortunately for civil rulemakers, for the federal courts, and for the litigants that appear before them, it seems that the more firmly the civil rulemaking committees cling to an assertion of neutrality, the more they are viewed as politically motivated, ideologically biased, and unfair.

I. The Empirical Turn: How We Got Here

Starting in the 1980s, a key criticism of federal judicial rulemaking that gained traction was the lack of empirical foundation to Advisory Committee action.^[6] With each successive package of amendments, scholars lamented the Advisory Committee’s lack of interest in empirical support for its suppositions. Commentators demanded empirically grounded procedural reform.^[7] With each subsequent amendment, concerns about insufficient empirical investigation increased. Stephen Burbank’s 1989 article connected the lack of interest in empirical investigation to normative commitments of the rulemakers then comprising the Advisory Committee, particularly to the efforts to manage the tensions inherent in trans-substantive rules and “the gap between formal equality and inequality in fact.”^[8] Under intense scrutiny for the better part of the 1980s and continuing into the 1990s, federal rulemakers came to embrace the calls to empiricism.^[9] Today, it would not be an exaggeration to say that the judicial rulemaking apparatus understands itself to be not only an empirically grounded institution, but also a uniquely qualified institution to control civil rulemaking through its capacity to gather and analyze empirical data about civil litigation.

To appreciate the important role that empirical inquiry plays in the legitimacy of the civil rulemaking process, one need look no further than the rules committees’ own defense of their rulemaking authority to Congress in the aftermath of the Twombly and Iqbal pleading decisions.^[10] Chairs of both the Advisory Committee and Standing Committee on Rules of Practice & Procedure (“Standing Committee”) defended the Rules Enabling Act process as the proper division of labor for ensuring “the best rules possible” by highlighting the Committees’

dedicat[ion] to obtaining the type of reliable empirical information needed to enact rules that will serve the American justice system well and will not produce unintended harmful consequences . . . . [The committees] are at this moment deeply involved in precisely the type of work Congress required . . . gathering and studying the information needed both to understand how [the rules] . . . have in fact worked . . . and to consider changes to the text of these rules . . . . Major empirical work on discovery costs and burdens . . . is underway in preparation for a May 2010 conference at the Duke Law School . . . .^[11]

This “empirical turn” in procedure in both rulemaking and scholarship is well recognized. As much as two decades ago, legal scholars Bryant Garth and Michael Heise identified the trajectory, with optimism for the benefits that might attend our understanding of civil justice and of the potential for improved quality in policymaking debates.^[12] Since then, there have been dizzying developments in the sophistication and techniques available to study the civil justice system empirically. It would not be unreasonable to speculate that we are entering a golden age of procedural study. Yet more recently, some scholars have drawn our attention to the limitations of empirical inquiry.^[13]

The empirical turn reflects a surprising consensus, within what is otherwise a highly contentious field, on how to best understand the civil justice system and the functioning of the courts; though there is disagreement as to almost all procedural questions, there is a shared epistemic outlook. This point of agreement comes about not only because our empirical capabilities have expanded at a remarkable pace (as indeed, they have), but also because empirical investigation suggests the promise of rising above interest. Indeed, as the most prominent advocate for empirical legal studies, Heise has argued this explicitly.^[14] That is, policymakers and scholars alike seem to hold out hope that empirical approaches will rise above normative disagreements and answer questions without resorting to the messy interests that flood the adversary system. Looking for neutral ground, procedural rulemakers are all too happy to seize on the terrain of empirics.

Even as the Advisory Committee has incorporated the demand for empirically grounded procedural reform, scholars have identified a pronounced trajectory to procedural reform, what Benjamin Spencer has called a “restrictive ethos” and Stephen Subrin and Thomas Main have categorized as the “Fourth Era” of civil procedure.^[15] Based on these accounts, the Advisory Committee’s responsiveness to empirics has not resulted in any shift in either interpretation of federal rules or in the trajectory of procedural reform.^[16] Despite the integration of empirical study into procedural rulemaking, rulemakers appear to make the same types of choices unsupported by the very same empirical research.

The restrictive ethos in procedural reform has been explained as a product of the particular ideological commitments of individuals involved in procedural reform—often centered on the individuals who comprise the rulemaking committees.^[17] Scholars have documented the intense resources that major industries devote to lobbying and reform efforts.^[18] Some scholars have connected the consistent trajectory of reform to broader deregulatory trends in U.S. politics.^[19] Indeed, the Advisory Committee’s use of empirical data has itself been the subject of scholarly criticism. The Advisory Committee has been criticized for ignoring existing empirical data^[20] and for drawing conclusions unsupported by the data they do solicit.^[21] The most common assessment is that, while rules committees solicit empirical data and announce their empirical interest with great fanfare, this data appears to have little to no effect on their decision-making.^[22]

A. The Methodology Produced by Neutrality Epistemology

This Article takes no issue with any of this. Certainly, lobbying is substantial and concerted; there is no question that the professional background, training, and political ties of those who sit on rules committees are similar and disproportionately reflect elite, well-heeled sectors of the profession.^[23] And indeed, there is no denying that we live in a deregulatory moment.^[24] The focus in this Article, however, is on methodology. That is, how is empirical data gathered and deployed in ways that are predictably misinterpreted? And why are realities of the system that seem apparent to many shielded from rulemakers’ purview? The existing literature may leave us with the impression that it does so because of conscious ideological commitments of the membership—that is, some kind of bad faith. This Article suggests a broader explanation—an underlying epistemic problem that would likely be expressed even with changes to the membership of rules committees.^[25] That is to say, the consistent and predictable outcomes of the committees’ work do fall in line with the deregulatory commitments of mainstream politics, but are reflective less of the conscious commitments of individual rulemakers and more of how those deregulatory commitments mesh with the dominant epistemic commitments of the day.

Clearly, it is not enough to demand “more empirics!” Data can misguide just as surely as insider-group impressions. We have to understand what it is we are trying to understand and be mindful of what such data can tell us. In other words, our purpose guides our search. To do so, we have to think hard about the different things we need to know, recognize that there are different methods to serve different purposes, and understand that the ultimate quality of our information—whether it translates into real knowledge—depends on our ability to gather the right information for our purposes and to interpret that information effectively. It is the contention of this Article that the prominence of data breakdowns in recent years is not only a result of the increased availability of sophisticated data analysis, but also a product of our contemporary posture toward knowledge (or, perhaps, to the exercise of judgment and its interaction with information).^[26] The use—and misuse—of data in the recent amendments to the Rules on discovery (the “Discovery Amendments”) provides an opportunity to examine how contemporary proclivities to deify data and denigrate judgment operate in this specific context and help to shape the trajectory of rulemaking.

The Advisory Committee’s approach to developing and processing information relevant to the project of discovery-cost reform helped to dictate the outcome of the project. The Advisory Committee treated such information in two ways: either as empirical data or as impressionistic observation. Rulemakers have rightfully incorporated calls to gather and be guided by empirical data and are also anxious to solicit and utilize input from the bar and other stakeholders in the procedural system. All procedural information is treated as falling into one of these two buckets. The proportionality discussions illustrate how this binary approach to the information has prevented the Advisory Committee from gaining the full benefits of the information it does gather and prevents it from seeking out all, or even perhaps the most pertinent, information it needs.

Ultimately, the desire for neutrality is broader than mere efforts to mask existing political commitments.^[27] It is a product of the milieu in which rulemaking operates, with a desire for neutrality and for objective measures manifested in an eagerness to apply empirical research. But skepticism of judgment leaves the empirical research carelessly accumulated and frequently misapplied or misinterpreted. Worse still, judgment-skepticism leaves other methods of assessing litigation and knowing how it operates untapped. Since finding good information depends fundamentally on asking the right questions and utilizing methods appropriate to the question, the objectivity and neutrality-reliant rulemakers sink further into the quagmire.

This Article aims to challenge the insistence on procedural neutrality as both ineffective at changing the trajectory of procedural reform and as missing an important piece of the context that shapes our procedural policymaking. It argues that procedural discussions must be explicit in grappling with how process rules structure inequality that comes before the courts. This Article suggests concrete improvements that can be made in the rulemaking process to tackle inequality head-on: careful and attentive categorization of litigation information and the use of sociological factors of litigation to develop more accurate, concrete, and fully drawn models of how rules operate and the effects that any proposed amendments might have.

II. Neutrality and Inequality

This Article is the culmination of a long-developing project exploring an interlocking set of concerns: the increase of inequality in civil litigation;^[28] the dissonance between empirical data about civil litigation and the dominant conceptions of civil litigation; and the express commitment to neutrality in civil procedure. This final chapter of the project aims to tie together the relationship between the commitment to neutrality and inequality.^[29]

A. The Cost-and-Delay Narrative

I first tackled these themes in The Cost-and-Delay Narrative in Civil Justice Reform, an article that sought to explain the resilience of what I termed the “cost-and-delay narrative.” The cost-and-delay narrative is fundamentally a myth, a story of disfunction that (primarily) elite lawyers and judges tell about our courts and the civil justice system. Its hallmarks are a conviction that the civil justice system is beset by crippling costs that produce delay, backlog, and miscarriages of justice, typically through the filing of cases that abuse the system and harass defendants. The Cost-and-Delay Narrative revisited these claims, relying on an excellent body of scholarship stretching back decades. That scholarship conducted and examined empirical research on costs and delays in the civil courts and sounded the alarm that false claims of crisis were serving as the foundation for deregulatory reforms, limiting the public’s power to get into, and stay in, court.^[30]

The Cost-and-Delay Narrative traced this thread straight into the second decade of the twenty-first century, where once again the best and brightest of the bench and bar warned that the system was staggering under the weight of runaway costs and unbearable delay.^[31] The Advisory Committee was yet again considering whether amendments to the Rules could help to tackle the purported crisis, and the Supreme Court was upending a half-century of settled case law on the Rule 8 pleading standard in the Twiqbal decisions.^[32] New cries of a cost-and-delay crisis echoed the worries of preceding decades.

The throughline from the twentieth century straight into the last two decades was also apparent in the lack of empirical support for the cost-and-delay narrative. To guide rulemaking efforts to reduce the costs of discovery in the last decade, the Federal Judicial Center (FJC) conducted studies of discovery costs in the federal courts.^[33] Consistent with several decades worth of study of the procedural system, the FJC findings confirmed that discovery costs were, on average, a small fraction of attorneys’ estimates of the stakes of the litigation.^[34] Perhaps even more significant, the single best predictor of discovery costs in a case were attorneys’ estimates of the stakes in the matter.^[35] That is, even as the Advisory Committee’s reform efforts were focused on the fear of a discovery system out of control and demanding proportionality, the best data we had indicated that discovery was best characterized as proportional to the legal claim.

The FJC study confirmed what prior data had demonstrated, yet it nonetheless “rang false. It did not support anything that attorneys ‘knew’ about the civil justice system.”^[36] Despite the fact that decades of empirical work supported a view of the system as “highly effective in most cases, that total costs develop in line with stakes, and that discovery volume and cost is proportional to the amount at stake,”^[37] elite lawyers and judges repeatedly returned to the “crisis” of cost-and-delay in the courts.^[38] Why is a myth that lacks supportive empirical evidence so enduring? Why aren’t the careful studies of legal scholars or the publicly commissioned studies by the judiciary itself successful at puncturing the myth and putting the narrative to rest? Part of the puzzle The Cost-and-Delay Narrative aimed to solve was that, despite the endless debunking, the narrative was, in the twenty-first century, more salient than ever. Indeed, it ultimately underwrote arguably the largest changes in federal civil procedure in this century, playing an important role in the Discovery Amendments to the Rules, the Twiqbal decisions, and the continued doctrinal expansion of arbitration.^[39]

One understandable response to the resilience of the cost-and-delay narrative has been to suggest that we have insufficient data about the civil justice system.^[40] However, since The Cost-and-Delay Narrative built on a wealth of scholarship carefully demonstrating a lack of empirical foundation to cost-and-delay claims, the article argued that the lack of empirical data could not be the reason for the resilience of the cost-and-delay narrative.^[41] We already know a lot about the costs of the civil justice system but struggle to trust what we know, since the data does not confirm our common impressions. A related argument, still influential today, is that the myth persists because decision-makers do not follow empirically based processes or are not required to ground their decisions in data.^[42]

Examining the existing scholarly literature on the cost-and-delay narrative, I identified three clusters of explanations for the power of the cost-and-delay narrative, while laying a foundation for future investigation. The clusters included (1) media distortion and cognitive heuristics; (2) policymaking bias due to the political priors of rulemakers and extensive lobbying by defense interests; and (3) the dominance of a neoliberal economic and political movement toward deregulation in the United States.

First, among the most common factors cited to explain the strength of the narrative has been the role of media coverage that demonstrably privileges reporting verdicts in favor of plaintiffs over those in favor of defendants, large jury verdicts over smaller ones, and particular types of cases, such as medical malpractice and product liability, rather than all other types of torts trials.^[43] These distortions work in conjunction with reliable cognitive heuristics in the general public to leave the U.S. population susceptible to acceptance of the cost-and-delay narrative.^[44] Even more than the working system of media distortion and cognitive bias, the most common cause identified in perpetuating the cost-and-delay narrative is the role of interest-group influence and rulemaker bias.^[45] The perpetuation of torts “horror stories” by pro-business organizations couples with more traditional lobbying for pro-business reforms of the procedural rules.^[46] Scholars believe that procedural rulemakers are prone to persuasion on these points. Separate from lobbying efforts, scholars are also focused on the biases that the rulemakers themselves bring to the task of procedural reform.^[47] The composition of the Judicial Conference itself has been the subject of numerous studies exploring the race and gender of its members, their defense-oriented experience, and their lack of experience in trial courts.^[48]

However, accounts primarily focused on lobbying and bias are unsatisfying because they are prone to a conspiratorial interpretation in which misinformation is disseminated to influence committee members, or by the committee itself, in order to carry out an agenda by obscuring facts that are already known. The power of the cost-and-delay narrative stems from its ability to describe a certain subset of practice that reflects the experience of the top strata of the profession, whose positionality and desire to see that positionality as universal leads them to understand the cost-and-delay narrative as true. In The Cost-and-Delay Narrative, I explained how the hierarchy of the legal profession itself contributes to the gap between the realities of typical civil litigation and procedural policymakers’ own experiences of the system. Because opinion makers in the legal profession disproportionately represent large entities in complex litigation, they are most likely developing their impressions of the “normal” litigation experience in the most extraordinary contexts. That is, they are most likely to spend their time handling high-volume discovery in high-stakes litigation. Hence the 2009 FJC study’s median numbers—whether they be for total cost, amount of discovery, or proportionality of discovery—are unlikely to look anything like the practice experience of these attorneys.^[49]

Further, these explanations do not appear to resolve a fundamental contradiction that Cost-and-Delay identifies. It is the very aspect of the system that we do have data about, and that rulemakers have repeatedly commissioned data-collection studies on, that seems to be entirely impervious to the findings of that data. This counterintuitive dynamic continues to trouble me. The issue seems not to be a rulemaking process devoid of empirics, nor one of lack of data.^[50] On the question of cost-and-delay, the rulemakers appear eager to ask for data and claim a reliance on empirics.^[51]

It is perhaps this contradiction that leads many commentators to explain the stubborn commitment to a cost-and-delay crisis through bias on the part of federal rulemakers and the outsized influence achieved through the concerted lobbying and public relations efforts of industry.^[52] Yet bias, or prior political commitments of rulemakers hostile to litigation, does not help make sense of the repeated turns to the empirical or the calls by opponents of cost-and-delay reform for further empirically grounded rulemaking and procedural reform.^[53] Calls of bias have tended to lead to calls for more data and empirically based rulemaking as the antidote. The greater the calls for empirics, the more the Advisory Committee has itself called for empirical data and claimed the position of data-steward—going only where the data take them and drafting proposals that, they claim, address the realities suggested by the data.

The Cost-and-Delay Narrative has proved a useful way of understanding the enduring rationales for access-limiting, deregulatory reforms to the civil justice system.^[54] Yet the fundamental puzzle that animated the article has continued unabated. Indeed, the narrative has continued to serve as conceptual support for the most significant changes to civil procedure this century, including the Twiqbal cases that rewrote Rule 8 to introduce a plausibility pleading standard to the federal courts and the Discovery Amendments to the Rules that serve as the focus of this Article.^[55]

B. What Does It Mean to Say Procedure Is Political?

Reforms continue to rely on the cost-and-delay narrative even while commentators and rulemakers alike call for a commitment to empirically based procedural reform. The move to seemingly objective and neutral bases for decision-making is not an isolated development of procedural rulemaking and is best understood as a style of judicial argumentation. In What Does It Mean to Say Procedure Is Political?, I placed the retreat to the ostensibly neutral and objective grounds of empirics in the context of the historic turn—that is, to the neutral and objective in judicial reasoning—on highly politicized questions.

For decades now, commentators on procedural reform have lamented the polarized discourse and the bitter tenor of reform debates. What cure should we prescribe for a highly politicized rulemaking context? This question is at the core of my inquiry in What Does It Mean to Say Procedure Is Political? and in this Article. My concern is that the dominant position—that politics has no place in procedure, that strategic behavior is an indication of attorneys behaving badly, or of corrupting the rights-based legal system—may not only be inaccurate, but may in fact exacerbate the polarization, the bias, and the inequality of procedural operation and reform.

Despite platitudes about procedural neutrality, the effect that procedural rules have on legal outcomes is widely acknowledged. How the procedural defaults are set will ultimately determine who can bring claims to court, whose claims can stay once there, and whose cases may ultimately prevail.^[56] They help determine what our courts are and will be used for, and what the relationship between paper rights and actual legal power will be.^[57] The reality of civil process is that it is, in many ways, the body of law that is most clearly a collection of policy choices, and a set of “background rules that create . . . inequality or leverage that parties exploit to reach agreements . . . . The background rules, procedure among them, create the inequality with which litigants arrive in court, and on which their relative bargaining power as to a civil settlement depends.”^[58]

Our recognition of the substantive importance of procedural policy may actually heighten polarization and hinder productive debate, as all players scramble for facts that can place their positions beyond debate. To confront the politics of procedure has been challenging because, for the most part, we have agreed that procedure should be above politics. Thus, the terrain for debate becomes constrained, taking on a personalized character. The real stakes of the debate—that is, the ways that procedural rules will distribute legal power and resources—remains outside the bounds of discussion.

As I have previously examined, however, procedure is not the first field of law to struggle with the realities of its political nature. In What Does It Mean to Say Procedure Is Political?, I drew on the work of early twentieth-century institutionalists to sketch out what it means to understand procedure as political and to place contemporary procedural reasoning in the context of legal reasoning by courts when faced with controversial political questions.^[59] By examining procedural discourse as a species of legal discourse, we are able to connect the types of arguments made, and the seemingly curious analytical choices of rulemakers, to trends in judicial reasoning deployed at other times and in other substantive areas, particularly when the substance of the rulemaking is contested (whether judicial or, in this case, by a rulemaking body).

This comparative exercise makes clear that judicial bodies resort to familiar methods when confronted with the need to engage in controversial policy determinations. In procedural rulemaking, as in the constitutional property cases of the past, a common method relied upon is to anchor those controversial assessments in what are hoped-to-be neutral and objective concepts.^[60] That the policy choices are contested is precisely what makes them political. Even where a choice might distribute resources of legal power differentially across litigants or society, if there is a high degree of consensus as to that distribution, the legal field will not face difficulty in its operation as a neutral body.^[61] That most commentators agree that procedural policymaking is “political,” or politicized, today does not mean that rulemaking reflects the political commitments or biases of decisionmakers; it is rather a reflection that procedural decisions serve some interests and stymie others, interests that are at odds outside of the courthouses and that wish to impact the field of play inside the courthouse.

Furthermore, relying on Hale’s concept of coercion, I concluded that there is no way to write a neutral procedural law. As we mostly recognize, in 1938 the drafters of the Rules had a high degree of consensus as to what kind of courthouses they wished to build—courthouses that enabled the contestation of competing rights in their public venues—and drafted rules to facilitate civil litigation.^[62] However, our 1938 Rules nonetheless rely on their own myth, a myth of procedural neutrality.^[63] In so doing, they ignore the insight of Halean coercion, which reminds us that the rules intervene in already existing social relationships, that are themselves unequally structured by law.^[64] When we see social relations as bargaining, and legal rules as intervening in those bargaining relationships, the impossibility of neutral rules is clear and the centrality of legal rules to inequality becomes manifest.^[65]

C. Bargaining over Procedure

As procedural scholars have recognized for decades, it is not just the substantive laws of property and contract that are structuring bargaining, but the law of procedure itself involves bargaining over procedure, the development of the case, and its settlement.^[66] On the secondary level, parties bargain over the controversy or relationship that gives rise to the suit—their bargain is structured by law outside of the courthouse. That is, parties’ bargaining positions in the litigation itself are structured by their relative bargaining position external to the litigation. It seems apparent that procedure should recognize the primary level of bargaining and engage directly in understanding how the Rules affect relative bargaining position with respect to that primary level and, normatively, what effect it ought to have.^[67] It is less apparent, but no less relevant, to wonder what attention, if any, procedure should pay to the secondary level. That level—the social inequality with which parties walk into court—plainly influences the effect that procedural rules will have.

If procedure is choosing the distribution of power—both legal and material—and there is no way for it to refrain from doing so, then it becomes incumbent on procedural rulemakers to explore what impact on power and resources their rules have, and what impact any amendments may have. They must, of course, also make some transparent commitments about how the procedural rules ought to intervene in these legally structured, hierarchical, social relations.

A number of scholars have recently begun to tackle the inequality exacerbated by procedure—efforts that do not address questions of procedural neutrality directly but look instead to uncover the system’s effects on sets of litigants and cases often overlooked by federal rulemaking. In so doing, this burgeoning scholarship offers some concrete examples of how we might leave the neutrality imperative behind, even as we strengthen the coherence, functionality, and justice of civil process. Professor Andrew Hammond has placed a focus on poor litigants before the federal courts, examining existing in forma pauperis procedures throughout the district courts, in an aim to study procedure “from the bottom up.”^[68] Hammond’s study provides an excellent model for how attending to a known category of litigants, particularly one unlikely to be presently represented in discussions of procedural rules and reforms, produces knowledge about the civil system that is otherwise obscured from proceduralists’ view and suggests procedural revisions that we would not otherwise identify.^[69]

Even when we step back from consideration of the poorest litigants in the federal courts, we see that the relative resources of litigants bringing particular types of cases increases the likelihood that such cases can adapt to new procedural developments. Professor Brooke Coleman documents what she terms the “adaptation, migration, or extinction of claims” in response to significant procedural changes in the last several decades.^[70] Concluding that “despite the procedural rules or doctrines that are adopted, the most powerful litigants will almost always find a way to survive,” Coleman suggests that procedural policymakers should structure their reforms around the aim of conserving meritorious claims.^[71] Professor Coleman proposes that policymakers look at the claims with an eye to recalibrating attention from the “powerful” to, presumably, those who are not powerful or wealthy. This does not sound like the dominant demand for procedural neutrality upon which procedural debates have, until now, depended. While not a call to abandon procedural neutrality, Professor Coleman’s proposal is a specific example of how we might break free from the neutrality consensus to direct our procedural reform efforts, test our assumptions, and inform our data collection to serve selected ends, such as, in Professor Coleman’s proposal, the preservation of meritorious claims.^[72]

Of course, examining the substantive claims that might be impacted by a procedural change points to the challenges of our commitment, perhaps foolhardy, to trans-substantive procedure. On this count, Professor David Marcus has recently called the system’s bluff. In a tour de force, setting forth a persuasive argument that the “Federal Rules System” has collapsed,^[73] Professor Marcus argues that procedural commentators must develop new principles to undergird the civil justice system, ones that confront directly the political conflict at the heart of procedure.^[74] The retrenchment of the last forty years is best understood, he suggests, as a symptom of the collapse of a commitment to procedural neutrality and the concerted advocacy for procedural reform in line with the principle of neoliberalism.^[75] Thus, it cannot be challenged through appeals to procedural neutrality and must instead confront the principle of neoliberalism directly.^[76] I agree. This Article aims to challenge the insistence on procedural neutrality as both ineffective at changing the trajectory of procedural reform, and as missing an important piece of the context that shapes our procedural policymaking. It argues that procedural discussions must be explicit in grappling with how process rules structure the bargain for typical parties before the courts.

Recognizing that the Advisory Committee’s commitment to empirical work is part of a long tradition in U.S. law to avoid challenging and conflicting questions in policymaking helps to make sense of the Committee’s choices, not as nefarious bias, but as a well-worn move when faced with the charge of making neutral rules on unequal terrain. I return to this question of objectivity and empirically based rulemaking here, to trouble our understanding of the concept of the empirical and our aspirations for what it can help us to achieve.^[77] I contend that we ask too much of the empirical, guided partly by shared aspirations with judicial decisionmakers, for neutral and objective waters in which to anchor our procedural ships. To understand why this effort has failed, and is doomed to continue doing so, it helps to consider what we mean by “empirical” in procedure.^[78]

In the next Part, I explore the ways that empirical scholarship in procedure has been limited by a commitment to procedure’s neutrality. I then turn to the example of the Discovery Amendments process, describing the commitment to empirically grounded reform and the problems that commitment encountered. Finally, I consider the ways in which we can arrive at more accurate and honest discussions of the workings of our Rules through frank consideration of the competing interests at stake and recognition of the adversarial context in which the Rules operate.^[79]

III. Understanding the Empirical Turn in Procedure

The increasing attention to empirical methods in both the study of procedure and in procedural policymaking is an important point of consensus in a field often described by its rancor and discord. To recognize the broad level of agreement as to how to understand the procedural system, even as we disagree over what new directions to take, suggests a common ground that shapes the procedural moment we are in. Ultimately, this moment, including its empirical turn, operates to maintain and produce procedural inequality.

Robert Bone, who has, perhaps more than anyone else, attempted to explore the philosophical foundations of good procedural design, provides a useful periodization for the eras of procedural rulemaking since the passage of the Rules.^[80] The current procedural era, he suggests, is characterized by a lack of consensus, and this inability to think about first principles, to explore and agree upon a normative direction for procedure, helps explain the rulemaking milieu in which we exist today.

Recognizing that procedural change today is contentious and lacking a strong sense of what must be done, Bone has argued that part of the problem lies in a lack of normative foundation for procedure, an absence that leaves rulemakers adrift as to what changes to make and how to justify those changes. As Bone pithily describes it: “[I]t is not possible to choose good Federal Rules without knowing what makes a Federal Rule good.”^[81] He calls for a theory of procedure that “derives broad normative principles from core features of litigation practice.”^[82] Doing so would allow an evaluation of procedural rules with reference to the core principles, enabling the accounting of “all the costs and benefits.”^[83]

Bone’s identification of normative contestation is both incisive and incomplete. He points us to a key defining feature of our procedural moment—collapse of normative consensus—but leaves out the remaining areas for consensus, those that allow us to think we are merely engaging neutral and objective claims (or numbers). This is where quantitative methodologies and numbers fill the void left by uncontested disagreement. Garth explains that, in today’s world, one way to make a legitimate argument is to marshal data to support your position, and, in the absence of clear consensus, particularly in civil procedure, actors turn to social science to help generate agreement or answers.^[84] Garth too hoped that reliable empirical data would constrain and guide politicized debates.^[85]

If this contention is true as to the substance of procedure, we should not lose sight of the corresponding epistemic consensus. Agreement on this level, coupled with the otherwise contentious field of procedural study and reform, helps to form the contemporary procedural moment. Counterintuitively, it contributes to the production of procedural inequality. This is a feature of our procedural inquiry today, though it is of course not an inherent characteristic of empirical scholarship or empirically informed policymaking. What makes inequality a feature of procedure’s epistemic turn? Our epistemic consensus is characterized by a particular conception of the “empirical,” which has a number of limitations: (1) this conception serves to limit the methods of inquiry that are available to us in our study of our civil justice system; (2) it reinforces a commitment to objectivity that is likely inappropriate to a social system dependent at every turn on individual decision-making and judgment; and (3) it tends to suppress categories and voices central to understanding the operation and effect of the system.

While literature addressing the “empirical turn” has presented thoughtful examinations of both the reasons for the increase in empirical scholarship, as well as the quality of that research and suggestions for the future, there has been surprisingly little attention paid in these works to the concept of “empirical” study itself. This is, at once, understandable and surprising. Rigorous engagement with the notion of the empirical is challenging and threatens to take one far afield from the concrete questions that might animate a scholar’s empirical research in procedure. Indeed, debates about what exactly constitutes empirical study and what type of knowledge it allows us occupied scholars for much of the last century.^[86]

At the same time, any accounting of empirical approaches to procedural law, to say nothing of efforts to assess its value to the field, requires some engagement with what we mean when we refer to the “empirical.”^[87] Policy debates in procedure do not pay much attention to this question.^[88] The rulemakers use the concept polemically to suggest the special role they are keenly suited to play or to preemptively legitimate the process they engage in and conclusions they will ultimately reach. Moreover, while the question of data or empirics and procedure has been lurking for decades in procedural scholarship, little definitional or epistemological work has been done to clarify what is meant by the concept of “empirical data” or “empirical scholarship” on procedure or on the civil justice system.^[89] There is no way to dispense of this question quickly or simply. Indeed, what constitutes “empirical” study of social phenomena is one of the great and unresolved questions of the twentieth century. The answer will tend to turn on the underlying commitments of the person asking the question.

In law, these epistemological battles played themselves out along a number of theoretical battlefields: realist, critical legal studies, and perhaps most potently feminist legal theory and critical race theory. A number of scholars in both the critical legal studies tradition and those employing feminist legal theory took very seriously the challenge of what constitutes empirical study and what our deployment of it suggests about our epistemological commitments.^[90] These scholars argued that a lot of empirical research in law is structured by a commitment to positivism, where positivism is marked by “empirical validation of theories about the conduct of actors affected by the legal system”^[91] and requires “a bright line between the realms of doctrine and behavior and between values and facts.”^[92] This approach is undergirded by, and indeed depends upon, a “belief in [the] objective knowledge of causal relationships.”^[93] Knowledge about society is objective “because it is knowledge about facts that are neutral and external to the observer.”^[94]

The focus on empirically based procedural reform fits in with a broader movement in legal scholarship—empirical legal studies—that has been in the ascendancy for much of this century. Indeed, these trends in law (as is often the case) mirror trends across the academy and economy. Part of what the Advisory Committee and Judicial Conference pick up on when they assert their empirical bona fides as a basis for their civil rulemaking authority is this broader quantitative empirical zeitgeist. The way to know anything in today’s world is through quantitative metrics, and in the world of procedure, the members of the Advisory Committee are making a pitch for that role.

A. A Quantitative Turn

There is robust literature on quantitative empiricism and its consequences—both positive and negative—in other fields outside the law.^[95] The law itself has been slow to develop its own critiques or theory of empiricism, perhaps because it has been slower to enter the quantitative empirical era, or at least, its quintessential institution—the court—still claims to eschew quantitative methodologies.^[96] Many of the explanations for the turn to quantitative empirical methodologies and to metrics are helpful in understanding the developments at the Advisory Committee as well.

Fundamentally, the quantitative turn can be understood as a strategy for avoiding contentious political questions.^[97] That is, in a context in which there is no consensus as to the right outcomes and goals, but also a desire or need to avoid contentious political questions, a quantitative approach can seem like a safe alternative on which to build broad agreement despite political differences and a neutral methodology from which to obtain objective information and, ultimately, solutions.^[98]

These decades-old critiques by feminist legal theorists, critical race theorists, and those from other disciplines are bolstered by more recent assessments of how empirical study of procedure has fared. Specifically, the dominance of a positivist conception of empirical study limits the terrain that is visible to proceduralists and rulemakers. The assumption has tended to be that empirical questions can be answered by empirical analysis of data sets. In turn, qualitative study, institutional knowledge, and normative questions are constrained.

Bryant Garth concludes that

[t]he history of the relationship between empirical research and the reform of civil procedure has been one of alternating enthusiasm and disappointment. The natural tendency of the legal community is to demand solutions from empirical research and then to blame the research for failing to resolve the debates that it was—at least in the eyes of the legal community—supposed to resolve.^[99]

Noting the rise of empirical research on civil justice, Garth identifies two rationales for its recent growth.^[100] Each of these accurately describes the empirical efforts of the Advisory Committee. One rationale is to use data to, in Garth’s words, “civilize” unsettling debates.^[101] The demand for empirical research is a response to the “difficulty of finding a common language or authority.”^[102] He concludes that this demand for the legitimacy of social science measures unfortunately does not necessarily produce “useful and competent research.”^[103] A second rationale, and the one that Garth suggests is the only universally accepted rationale for empirical research, is the deployment of social scientific methods to determine “what works” and “what does not work.”^[104] Garth believes such an approach “has much to offer” but that it also has limitations due in part to the effect of ideological battles.^[105]

The limitations of such an approach are several. First, it is difficult to make any meaningful evaluation of quality, so an empirical assessment of “effectiveness” cannot take quality into account in any reliable manner. Garth also identifies common difficulties for empirical research on procedure and civil justice, such as difficulties avoiding selection bias, establishing control groups, and developing data before and after reform measures.^[106] Perhaps closest to the argument explored in this Article is what Garth calls “the implicit hope of the ‘what works’ approach.”^[107] Empirical research is not best used instrumentally to bolster or debunk a polemical position: “[T]he utility of empirical research does not depend on its ability to resolve debates . . . . Empirical data should be collected for other purposes as well, and those purposes may over the long term prove to be more important.”^[108]

More recently, Jonah Gelbach has presented a more constrained challenge to empirical research by suggesting that even empirical questions may not always be susceptible to empirically derived answers. In considering the question of the effect of plausibility pleading, Gelbach presents two competing positions: one supportive of Twiqbal’s development of a plausibility pleading standard and the other critical of that development. Supporters claim that Twiqbal finally allows for the filtering out of strike suits at the motion to dismiss stage before those cases are allowed to impose sufficient costs to make them valuable despite their meritless nature. Critics, on the other hand, claim that Twiqbal’s heightened standard functions to filter out cases with true merit that do not have access to facts to prove those merits without discovery (a catch-22 problem). Gelbach explains that both of these positions are logical on their face; either could be correct. And indeed, it is certainly possible that there is both a sizeable number of strike suits that can be filtered out under plausibility pleading and also a sizeable number of meritorious cases that are improperly filtered out. “Therefore, the question as to which effect predominates ultimately is an empirical one.”^[109]

Ultimately, Gelbach concludes that “these results underscore a disappointing but unavoidable fact: there are some empirical questions that cannot be clearly answered by using feasible data.”^[110] Indeed, he describes as the article’s “basic message” that “data are unlikely to settle the debate over the case-quality effects of the new pleading regime ushered in by Twombly and Iqbal.”^[111] The fundamental contradiction here, the “disappointment,” is that an empirical question (what are the case quality effects that result from the institution of a plausibility pleading regime) cannot be answered using the data available (that is feasible to obtain).^[112]

David Engstrom also identifies significant limitations of empirical scholarship on procedure. He notes the “democratization” of empirical legal methods, no longer limited to those with doctoral degrees in related social science fields, but today utilized by a broad array of legal scholars without specialized training in empirical methods. Engstrom considers whether this is a salutary development or whether it would be better to place empirical study in technocratic control.^[113] Engstrom’s answers raise important limitations facing empirical procedural scholarship: “[T]he dispiriting reality is that existing Twiqbal empirical efforts offer precious little guidance to a Congress or an Advisory Committee considering revisions to the Twiqbal pleading standard.”^[114] Much of the reason is methodological problems of the type we might anticipate: sampling bias, lack of covariate controls, and inattention to the statistical significance of results. Another common limitation is in study design not carefully aligned with study questions. Engstrom points out that although Twiqbal studies were often interested in the effect of the change in pleading standards on plaintiffs, they took as the unit of analysis the judicial order or claim, rather than the party.

One important result of a large volume of poorly executed studies is that it diminishes the value of empirical study to sharpen the terms of procedural debate, in contrast to Garth’s hopes two decades ago. Engstrom points out that

[a]n important part of [empirical legal studies’] promise at the dawn of the movement was that empirical research—even relatively simple descriptive work—could discipline public debate over litigation by deterring the more overheated claims made by the Chamber of Commerce, the American Tort Reform Association, or the plaintiffs’ bar.^[115]

But because the literature has instead produced widely divergent findings, they have little disciplining effect and may even serve to muddy the waters.

Engstrom also points out that the studies themselves may have an effect on their subject. They “may well shape litigant and judge perceptions as much as (or even more than) they reveal them.”^[116] In a related vein, he suggests that the increasing technical sophistication among empirical scholars, while overall a promising development, runs the risk of crowding out qualitative institution insight

and, more specifically, lawyerly understanding and judgment—in the formation of hypotheses, the construction of data samples, and the coding of variables. The result may be a “naïve” empiricism that is prone to basic interpretive errors and no more likely to generate valid inferences about complex interactions of procedure and substantive justice than qualitative surveys of doctrinal developments or practitioners’ ground-level, gestalt sense of things.^[117]

Moreover, such limitations mean that very few studies “permit . . . a social-welfare or other policy-analytic judgment about the decisions’ on-the-ground effects.”^[118] Engstrom tells us that “the most devastating” problem with these studies is that even a well-executed study

would . . . not tell us whether judges are using their newfound case-screening powers in ways that increase or decrease social welfare . . . . Here, then, is the most defeating observation of all, for it suggests that even the best . . . empirical efforts can offer only limited guidance to a Congress or Advisory Committee considering revising the Twiqbal pleading standard.^[119]

That is, to the extent the hope for empirical studies of procedure allows for empirically informed policy change, there is reason to believe such an aim is unattainable.

In procedure, the neutral goals to which end we can put quantitative methods to use are, most often, efficiency, cost, and delay.^[120] That is, if we think back to Bone’s typology, it suggests our moment is characterized by a lack of consensus, which is true to some extent, but it is also characterized by an effort to look away from that fact of conflict, to suppress it, and to trust that there is a technical—often, quantitative—approach that can help avoid addressing the conflict.^[121] That is, it is an effort to re-present an actually conflictual and political question that fundamentally concerns the division of resources, into a technical one upon which everyone can agree so long as we use our admittedly sophisticated and impressive quantitative methods. Thus, the turn to quantitative methods is part and parcel of an effort to mask the distributional character of the inquiry—to flatten what are, in actuality, quite sharp differences in interest.

Individual actors engaged in the procedural reform project are often not motivated to do so—to gain victory at the expense of other factors in society—but rather, they are looking to move forward in an irresolvable contradiction. How do we make procedural rules that do not “expand or contract a substantive right”—everyone agrees that this is an impossible task—and how do we do so when there is sharp divergence of interest over what type of rules we ought to have, who ought to be allowed into the courts, and for what purposes?

The only way to press on is to focus on things that we can all agree on. And so, all are complicit in this dodge. The drive to find objective terrain and to work with measurable facts focuses the debate on those features that we assume will lend themselves to quantitative analysis. It therefore tends to focus procedural debate on categories such as cost, delay, and efficiency, and to crowd out the qualitative study as well as the “lawyerly understanding and judgment” that Engstrom calls our attention to. The primacy that quantitative empirical assessment has achieved in the last twenty-five years means that all sides clamor for more measuring, more quantitative analysis, and better metrics—while leaving the real conflict, that is, who is this procedural system for and what do we as a democratic society want it to achieve, marginal and on the edges.

B. What It Means to Be “Empirical”

With this theoretical background in mind, can we make sense of what procedural rulemakers mean by “empirical?” It seems they are operating under a positivist conception of empirical method. We can see this systematically in the way that the rulemakers seem to oppose empirical work to bias, and to hold out neutrality as a gold standard in behavior, even for actors in a purportedly adversarial system. Below we will consider the ways that rulemakers address neutrality throughout the process and the pejorative view they hold of advocacy, or any explicit articulation of interest. Accordingly, the role of the concept of “empirical” in procedural reform is open to two categories of critique. First, as previous analyses of empirical studies of procedure have highlighted, empirical research often suffers from poor methods, by the criteria and disciplinary requirements of positivist social science. Part IV discusses how these poor methods negatively impacted both the reception and outcome of the 2015 amendment process. Second is a challenge to this positivist approach to understanding empirically the relationship of procedural rules and civil litigation.^[122]

We might best understand the Advisory Committee’s approach to procedural inquiry as a positivist one, and, on this point, they are in line with the most common approach of commentators. Our scholarship has yet to explore the implications of empirical indeterminacy, although perhaps support for that possibility is beginning to take shape.^[123]

The lopsided attention that empirical concerns have received from proceduralists has been criticized by Bone.^[124] He warns that “[b]etter empirics alone cannot produce better rules[;] . . . it is not possible to choose good Federal Rules without knowing what makes a Federal Rule good.”^[125] Bone’s work calls for the development of a normative procedure that can justify procedural rules. He recognizes the impasse, that is, the uncertainty, plaguing contemporary rulemaking. The failure of procedure theory, Bone argues, is unique to our present procedural moment.^[126] In the two earlier periods of procedural reform, Bone contends, reformers were moved by a “strong sense of shared mission.”^[127] This normative consensus was marked by a confidence that reformers “knew what was wrong and what should be done about it.”^[128]

Let us examine for a moment the contention that we operate in a moment of peculiar or acute uncertainty as to aims, mission, or normative consensus. Is there really no such broad normative consensus today? It is the contention of this Article that there is such a normative agreement or shared milieu. Its norms and conventions are different perhaps from earlier moments, and, for example, it would be difficult to describe them as amounting to a “mission.” Nonetheless, there exists today a procedural orientation, at once epistemological and political, committed to an empirical accounting of procedural activity, value, and success, and a marketized conception of civil adjudication as a service for resolution of individual disputes.^[129] Counterintuitively, the turn to consensus in empirics may heighten the sense of politicization and reduce the legitimacy of the rulemaking process.

IV. Use and Misuse of Empirical Data

As described in the introduction, procedural rulemakers have been fatally imprecise when thinking about the information they need to understand civil litigation in the federal courts. Over the years, civil rulemakers have faced pressure to engage in reforms based on empirical data. Beginning in the 1980s, many critics argued that Advisory Committee members acted based on their suppositions about the world of litigation, suppositions that would not be supported by the facts. Rulemakers were not deaf to these pleas.^[130] In subsequent years, the Advisory Committee has actively sought out empirical research to inform its rulemaking process. Indeed, in the 2015 discovery reform process, the Committee solicited empirical research and presented empirical data with great fanfare.^[131] Although the Advisory Committee was plainly responsive to prior external criticisms of the rulemaking process, its understanding of “empirical” research was hopelessly muddled. Committed as the rulemakers are to neutrality, the members seem reluctant to take control of the reform inquiry. This has resulted in a number of related problems. The rulemakers over-rely on limited empirical data, they misinterpret much of that same data, and they fail to solicit high-quality information about the operation of discovery in the system—whether that information is quantitative or based on alternative sources of civil justice knowledge.

The conceptual problems stem from basic identification of “empirical data” itself. The structure of the Duke Conference agenda illustrates this problem. The Duke Conference was designed to throw open the doors of the Advisory Committee to the broad swath of constituencies involved in federal civil litigation, to gather helpful information about the system from them, and to use that information as a basis for procedural reform and improvement of the civil system.^[132]

To structure this broad and ambitious undertaking, the Advisory Committee sought out empirical research and aimed to make it the basis of Duke Conference discussion. As will be discussed further below, some useful data was obtained under this “empirical” label, but that data was generated by the FJC in coordination with the Advisory Committee itself. In other words, the Advisory Committee did gather empirical data that helped shed light on the actual operation of discovery within the civil system, but this useful data was obtained with the Advisory Committee’s questions centered in the study design.^[133] All other feedback, including that labeled and treated as “empirical,” was left unstructured by the Advisory Committee.^[134]

Moreover, the focus on generating “empirical data” has diverted attention from other means of understanding the litigation system. The Committee appears to recognize only two categories of information about the litigation system: “empirical” and other.^[135] Problems abound with treatment of the information deemed not to be empirical, but the category of empirical itself is rife with problems, both of mislabeling and misunderstanding.^[136]

A. Categorical and Interpretive Errors

The Advisory Committee’s treatment of empirical data is characterized by several errors. We can group these into different forms. First, the word “empirical” is pretty loosely applied in a manner that makes it both over- and under-inclusive. It is difficult to figure out from what characteristics the Advisory Committee believes something to be an empirical study or to reflect empirical data or not. As a result, we have the categorization of some materials as empirical that are not for the purposes that the Committee would like to use them, and the treatment of similar information as non-empirical, impressionistic observation. The second difficulty arises when the Committee turns its attention to the empirical data it has gathered. On more than one occasion, the Advisory Committee has fundamentally misunderstood the information before it—despite clear guidance from investigators at the FJC. Finally, the Advisory Committee leaves some existing empirical data untapped and fails to generate others.

Put differently: First, there is category error taking place. The committee labels as “empirical” material that is not empirical for its purposes. Second, there is misinterpretation, a misunderstanding of the empirical data before them and to what it refers. Third, perhaps because of category error, there is a failure to mobilize or marshal the abundant empirical data that could be helpful to the inquiry at hand.

A clear indication of rules committees’ emphasis on use of empirical data as a mark of legitimate process and the development of the “best rules” is the letters sent to Congress by then-chairs of the Advisory Committee and Standing Committee, Kravitz and Rosenthal respectively.^[137] According to these, the capabilities that make the rules committees a superior venue for evaluating procedural change hinge upon their ability to gather and study empirical information.^[138]

To gain a feel for how the Advisory Committee thinks about discovery-pertinent information, it makes sense to examine their starting point.^[139] In preparing to tackle, once again, the perennial question of discovery, the Advisory Committee sought out information. Perhaps it was responding to sustained scholarly critiques arguing that earlier civil rulemaking was based on faulty assumptions unsupported by empirical data. Or perhaps an evidence-based rulemaking approach seemed sensible to Committee members. Whatever the specific motivations, the Committee plainly sought to engage data to guide its reform efforts.

Civil rulemakers were interested in the impact electronic discovery was having on civil litigation practice. In the fall of 2008, the Committee’s Associate Reporter, Professor Richard Marcus, reported on electronic discovery, reassuring the Committee that “there are no signs” that its recent e-discovery amendments “ha[ve] added to the problems that continue to be reported.”^[140] Nonetheless, Professor Marcus reported: “[T]here continues to be ‘a lot of anguish’ about e-discovery.”^[141] That assessment relied on his review of the American College of Trial Lawyers’ (ACTL) survey of its members on e-discovery issues.^[142] The Committee thought it would be good to organize a conference that would build on “the foundation” laid by this survey, and another by the Institute for Advancement of the American Legal System (IAALS) to examine discovery issues. Immediately, then-chair of the Committee Judge Kravitz proposed that “[e]mpirical data on the cost of discovery would be important”^[143] and asked whether the FJC could assist in “building foundations for the conference.”^[144]

Once the Advisory Committee decided to arrange a conference to explore civil litigation needs, it focused on developing what it called an “empirical foundation.”^[145] The Committee solicited a study from the FJC, which designed a “closed case study.” That study selected attorneys involved in cases in federal court that had closed during the prior calendar year. In answering survey questions, an attorney was prompted to provide information about the specific recent case that had closed.^[146] The ACTL surveys, which were conducted independently and prior to conference planning, comprised an important part of this foundation as well. Unlike the 2009 FJC Study, other surveys captured only attorney impressions.

As the date for the civil rules conference neared, it became clear that the issue of pleadings ought to be addressed as well.^[147] Congress was threatening action to turn the clock back on pleading to pre-Twombly times, and this discussion, as the chair of the Standing Committee articulated, was “intensely political.” As a result, the Advisory Committee believed data would be necessary to help sort out whether a rules change was warranted. In particular, the FJC took responsibility to gather data, available through the Administrative Office, on filing of motions to dismiss before and after Twombly and Iqbal and to prepare a report for the conference.^[148]

B. Duke Conference Efforts to Understand Civil Litigation

Ultimately, the Advisory Committee conceived of the Duke Conference as a way to “step back and take a hard look at civil litigation in the federal courts generally and to ask the bench and bar how well it is working and how it might be improved.”^[149] If the Committee was going to dialogue with the bar, it “agreed that the most productive way” to do so was through this conference, to which it would invite “a broad, representative range of lawyers, litigants, law professors, and judges.”^[150]

At all times, however, the original vision of centering the Duke Conference on empirical data persisted.^[151] The Advisory Committee highlighted the FJC Closed-Case Study, though interestingly it initially presented this as an item “in addition” to the empirical data, a tool for “elicit[ing] the practical insights of the bar.” In its planning, the Committee emphasized that data would be produced for it by the ACTL, the IAALS, RAND, Fortune 200 companies, and the National Employment Lawyers Association (NELA). The Committee also garnered input from other jurisdictions (two state court systems and two foreign systems) on their civil litigation reform efforts.

C. Category Error

The “empirical data” presented at the Duke Conference was largely surveys that solicited attorneys’ impressions of a variety of aspects of civil litigation. This was the case for the featured ACTL/IAALS attorney survey, the same survey was administered to NELA, and the ABA Litigation Section survey.^[152] The major exception to this pattern was the FJC’s Closed-Case Study, which surveyed attorneys as to one specific case in which they were involved that had closed in the prior calendar year.^[153]

The Committee, which began its inquiry wondering whether discovery was really “out of control,” gradually redirected its conference to consider very broadly how to improve civil litigation. The Duke Conference design reflected this undirected inquiry. The event began with two panels on empirical research. The first presented the FJC Closed-Case Study as well as a presentation on the FJC’s follow-up interviews with lawyers, alongside data from the ABA Litigation Section survey.^[154] The second presented data from the ACTL/IAALS attorney survey, RAND survey data, and Cornell data.^[155]

The Duke Conference agenda was understood to be in two parts—on the one hand, the empirical data (“The conference will begin with the empirical research”^[156]), and on the other, all other information that could be provided about the system presented through a patchwork of papers by academics, judges, elite lawyers, and elite bar associations. These panels covered (in order): pleadings and dispositive motions, discovery, judicial management, discovery of ESI, settlement, perspective from users of the system, and perspectives from state practice.^[157] The panel on “users of the system” involved presentations from corporate counsel, who the Committee organizers understood would provide “a perspective different from the lawyers who provide services to them.”^[158] As with presenters from state courts, conference organizers explained that corporate counsel “similarly present views not often heard in these discussions.”^[159] Bar association proposals and rulemakers from the past were included as well.

D. Interpretive Error

The Advisory Committee’s reluctance to take control of the data gathering and interpretation process contributed not only to poor categorization of information, but also led to interpretive errors. In an early assessment prior to the Duke Conference, Judge Kravitz presented FJC study findings to the Standing Committee.^[160] Judge Kravitz noted that the Twombly decision cites Judge Easterbrook’s assertions that discovery is out of control and not susceptible to restraint through judicial management.^[161] However, Kravitz suggested, the FJC’s preliminary survey results don’t back that up. In fact, preliminary results show “little discovery occurs in the great majority of federal civil cases, and the discovery in those cases does not appear to be excessively costly, with the exception of 5% to 10% of the cases. That result, he said, is surprising to lawyers, but not to judges.”^[162] But the minority of cases with significant discovery have “caused serious discovery problems.”^[163] Judge Kravitz reiterated that “[t]he empirical data will be available well in advance of the conference, at least for the most part, enabling all panelists to draw on it.”^[164] The notion that this empirical research would serve as the basis for discussion and analysis at the Duke Conference remained strong.

By the time Committee members were inundated with survey data and impressions from large, elite bar associations, however, Judge Kravitz’s understanding of the data had shifted somewhat. He explained that “[t]he general consensus . . . is that the civil rules are generally working well. At the same time, frustration experienced by certain litigants leads them to believe that the system is not in fact working.”^[165] However, “[t]he two competing perceptions . . . are reconcilable.”^[166] Judge Kravitz explained these differing perceptions as two different sets of cases—a process that “works well in most cases but not in certain kinds of cases, particularly complex cases with high stakes.”^[167] What the “various empirical studies” showed, according to Judge Kravitz, was that the stakes in cases “clearly matter” with higher discovery costs and greater numbers of discovery problems in complex cases.^[168]

Immediately after Judge Kravitz’s “reconciliation” of the divergent perceptions of the system, Dr. Emery Lee, who had conducted the FJC Closed-Case Study, explained the “various studies presented at the conference.” Dr. Lee addressed the same divergence but explained the differences as products of the nature of the studies conducted. Lee explained that two different kinds of surveys had been conducted—general perception surveys “and those that were empirically based on actual experiences in specific cases.”^[169] “The two approaches . . . produce different results.”^[170] Dr. Lee explained that these “different results” actually reflect methodological differences, that is, two separate types of surveys, rather than system failure in certain types of cases. Although Dr. Lee did not overtly describe them as such, the lawyer association surveys were poorly designed, or what Lee called “perception-studies.” The claims that the system was “not in fact working” were predictable and the product of well-understood cognitive biases.^[171] “Respondents focus naturally on extreme cases and cases that stand out in their memory, and not on all their other cases. Perceptions, understandably, are not always accurate.”^[172] Judge Kravitz, perhaps seeking to rescue his earlier analysis of the diverging perceptions of the system, reiterated that, although the “vast majority of civil cases in the federal courts actually have little discovery,” in complex civil cases, discovery “can be enormous and extremely costly.”^[173]

This interpretive error—understanding the FJC Closed-Case Study to be a sample of a different set of data than the attorney opinion surveys—was repeated several times, not simply by then-Chair Kravitz.^[174] Meanwhile, the principal investigator on the FJC Closed-Case Study patiently explained the methodological differences that would account for the different findings. Several times, Lee explained that, based on this data, it seemed likely that rules changes would not have an impact on cost.^[175]

Committee deliberations over sanctions and preservation were similarly stilted, dominated largely by the large, corporate entities that typically face significant preservation obligations. Particularly, the Committee concluded from the Duke Conference that lawyers “were in agreement on two points.”^[176] Attorneys recommended a rules amendment “specify[ing] with greater precision what materials must be preserved at the outset of a case, and even before a federal case is filed,” as well as revision of sanctions under Rule 37(e).^[177] The Conference panel concluded unanimously that there was a need for further empirical study as to preservation and sanctions.

The results of such further research, however, might have called the Duke Conference consensus into question. The Advisory Committee solicited research into spoliation sanctions cases from Emery Lee and the FJC. The conclusion of that research was that sanctions were vanishingly rare and that the cases in which courts imposed severe sanctions were cases involving highly unusual facts.^[178] The Advisory Committee eventually moved forward with rules amendments addressing preservation and sanctions concerns, despite the truly rare occurrence of spoliation motions. The Advisory Committee’s engagement with and interpretation of the empirical research helps us to understand how this decision was reached.^[179]

V. Not All Impressionistic Observations Are Created Equal

Lack of care in framing questions started right from the beginning with the 2015 amendment process. The framing questions for panels at the Duke Conference are revealing. The discovery panel was titled “Issues with the Current State of Discovery: Is There Really Excessive Discovery, and If So, What Are the Possible Solutions?”^[180] Although the Committee had gone to some effort to gather data on precisely this question—how much discovery was occurring in cases across the federal docket and to what extent such discovery was insufficient or excessive—the Conference planners nonetheless re-presented this question as a purely conjectural one, separating the discovery panel from the presentation of data on discovery by lunch and nearly three hours, and making no effort to ensure that the observational assessments from the bench and bar engaged in any way with the empirical data gathered. This was also true of the following panel on judicial management, which was pitched purely as a cost-control mechanism. The panel title asked, “Judicial Management of the Litigation Process: Is the Solution to Excessive Cost and Delay Greater Judicial Involvement?”^[181] Obviously, this panel assumed the fact of excessive cost and delay in its very title, despite years and volumes of study debunking the cost-and-delay narrative.^[182]

The following morning, the Conference considered “Cost Benefit Analysis of E-Discovery and the Degree to Which the New Rules Are Working or Not,” yet again divorced from discussion and presentation of the study designed to examine precisely these questions—both of the actual state of e-discovery at the time and what, if anything, could be determined as to how the recently enacted e-discovery rules were operating.^[183] Most of the second day provided an opportunity for “perspectives” from various sets of lawyers. The first set were “users of the system,” identified as corporate general counsel, outside lawyers, public, and governmental lawyers. They were followed by impressions from counsel in states that had experimented with alternative procedural rules structures. Second to last were presentations from “The Bar Association Proposals,” which consisted of five defense bar associations and one plaintiffs bar organization.^[184] Finally, observations of former rules committees members were presented. Contemporary binary conceptions of available information were apparent in this conference structure. The Conference led off with what it called “The Empirical Data.” Once that “foundational” information was out of the way, it re-solicited information touching on many of the same questions investigated in the empirical studies, through constituent “observation” or “perspectives,” or through prepared proposals reflecting the deliberations of various lawyers groups. These proposals were entirely the work of the lawyer groups, without guided inquiry by the Committee or Conference organizers.^[185]

The best example we have of how the Advisory Committee approaches qualitative study of litigation is the 2013 Dallas Mini-Conference (“Mini-Conference”), organized to solicit early feedback on the Advisory Committee’s discovery reforms.

It is important to state at the outset of the Mini-Conference investigation that fear of partiality and the equation of their mission with neutrality guided, or rather misguided, their investigative approach. Additionally, this Article suggests that there is information that is evidently lacking from the 2015 information-gathering process that would be possible to obtain through a Mini-Conference-type approach. To do so, however, would require recognition that we want such information that is central to understanding the issue at hand and being up-front that this is what the Advisory Committee is seeking to do.

The Mini-Conference used experiential knowledge to inform the rulemakers’ understanding of the issues and the likely consequences of their proposals. But as with all information, experiential knowledge will only be as good as the methods for obtaining it. So, the idea would be to recognize that to devise “good” rules (or rule changes) requires a commitment to a vision of the “good” that must be explicit. Even with that explicit normative commitment, however, relying on experiential knowledge requires some effort to understand systematically how the rule, general and universal as it is in the federal procedural system, does (or can be predicted to) interact with the most common situations that arise in the federal courts. What “situations” are relevant for consideration? That may depend somewhat on the procedural question at issue, however, there are a few evident rules of thumb (or factors to draw on from the existing empirical information that we have).

To be clear, the instinct to obtain input from practitioners as the reform process is taking place makes sense. To the extent it reflects a recognition that practitioners develop valuable insights from using procedural rules in their specific litigation practices or, perhaps more accurately, have a strong understanding of how the rules operate within their specific litigation contexts, rulemaker efforts to tap into that experience and to use it to develop litigation information beneficial to procedural rulemaking may be highly valuable. Indeed, from the perspective of this Article, practicioners’ inputs are a necessary component to broadening our understanding of what types of information are necessary to forming a foundation for procedural understanding. However, as with empirical information, the quality and utility of such information is contingent on the quality of the inquiry. In the case of the Mini-Conference, the rulemakers’ refusal to self-consciously and expressly control the information-gathering process, and to define their questions and search methods, resulted in very poor results.

Consider the goals of the Mini-Conference. Certainly, a professional committee charged with reviewing and amending rules is not going to approach its information gathering in the same way as a social scientist conducting research that must be subjected to peer review. It is neither likely nor particularly desirable that the Committee select a single methodology and remain faithful to it; just as it is unlikely that committee questions or goals be articulated with the precision of a social scientific study. These are different processes, and one is more informal than the other with good reason. Nonetheless, a committee’s approach is still informed by epistemologies, the identification of a problem, an understanding of what information is required in order to fashion a solution, and a theory of how that information may best be obtained.

The materials organizing the Mini-Conference reflect the procedural epistemology of the 2015 amendments. These materials reflect some confusion as to what such a Mini-Conference might accomplish, what information this format was understood to provide, and how that endeavor was facilitated by the Mini-Conference. These materials make clear that the rulemakers have a very different understanding of the utility of non-Committee input than this Article would advocate and that this epistemological commitment is actually counterproductive to the Committee’s information-gathering process.

There are three aspects to the organization of the Mini-Conference that are available to us. First, we have formulation of the plan for a mini-conference, as reflected in the minutes of the meetings. Second, we have the selection of participants for the Mini-Conference. Third, we have the materials provided to the participants to help frame the discussion and feedback collected.

A. Developing the Mini-Conference

After several months of working to draft rule amendments responsive to problems identified coming out of the Duke Conference, the Advisory Committee resolved to obtain early feedback from users of the system. As valuable as that effort might have been, limitations are apparent from the approach the Advisory Committee took in setting up the Mini-Conference.

The most obvious problem with the Mini-Conference framing was that the Advisory Committee asked questions designed to obtain general impressions, yet again, of a set of lawyers, legal scholars, and judges. This approach squandered the unique capacities of those before them—particularly current practitioners whose utility is their ability to inform Advisory Committee members of their experience with particular applications of rules in specific case contexts.

The idea behind the Mini-Conference was to obtain early feedback from individuals in a good position to comment on the value and utility of proposed rule amendments and on the current functioning of the Rules. This instinct—that there is some litigation knowledge to cull from those practicing in the system who know it best—is hard to argue with.^[186] But, unfortunately, little thought is put into why we might benefit from the comments of attorneys. This ad hoc approach manifests itself in two ways. First, the conception of who the Committee should hear from is ill-thought-out; and second, what the Committee should hear from them appears to be given even less attention.

I contend that practitioners are useful to the Committee to the extent they can contextualize the operation of universal rules, articulated and conceived of at a general level, but that will apply to particular cases and contexts. That is, practitioners can help rulemakers develop a strong sense of the typical operation of rules in a set of likely contexts in which the rules will tend to operate. Thus, reaching out to practitioners to hear from them about the operation of rules in particular cases representative of those frequently encountered on the federal docket, with which those practitioners have substantial experience, is an invaluable tool for developing a credible, granular understanding of the Rules’ contemporary operation and of generating reliable predictions of the effect of any proposed changes.

In contrast to such granular information that practice experience can provide, general impressions about the system as a whole, current rule operation, or even proposed amendments are unlikely to provide rulemakers with litigation information that practitioners are well-suited to provide. To put a finer point on it, general impressions of practitioners, no matter how carefully assembled, are no more valuable than the general impressions of any observers of the system. These general impressions are liable to suffer from the cognitive biases that attend any general opinions and will be accordingly limited in their reliability.^[187]

With these principles in mind, the benefits of practitioner input would be best where (1) practitioners were asked for feedback concerning their particular experiences of litigation in their particular substantive areas, and discussion was guided to maximize this particularistic, advocacy-oriented information; (2) participants were specifically developed to provide a cross section of such experiences and substantive areas; and (3) existing information about the federal docket and about features of litigation that directly affect the question at hand (here, the cost and delay of discovery) were identified in order to better engage in both (1) and (2) above.

Judge Koeltl introduced the Mini-Conference^[188] and provided an immediate framing that highlighted common goals and interests, de-emphasizing differences in perspectives or client interests: “The shared goal is to improve the workings of the judicial system.”^[189] Unsurprisingly, the problem to be addressed by the Mini-Conference—and the rules proposals it would review—were vague and equally pitched to minimize a perception of conflict. Judge Koeltl explained that “some” believe there is “need for substantial change,” even that “we have reached a crisis that requires fundamental changes. Others think the situation is not so dire.”^[190] What precisely these concerns were about was left unstated, but the judge optimistically reported that “[a]ll recognize that we can always find ways to do things better.”^[191] Thus everyone joins in the effort to make improvements, though problems remain undefined. Judge Campbell reiterated this sense that everyone present could and should participate to provide the Advisory Committee “full criticism” to aid the committee in finding answers. His articulation of the goal was no less vague: “The Advisory Committee knows that it does not have all the answers. But it does have one goal, to improve the civil justice system.”^[192]

Eventually, Judge Koeltl explained that the specific proposals put before this Mini-Conference “seek to advance the just, speedy, and less costly disposition of litigation.”^[193] Of course, this constitutes the most meager identification of goals, as this is simply the language of Rule 1 restated.^[194]

These vague articulations of the Advisory Committee’s mission are at once reflective of the flawed process and constitutive of its flaws. Committed to neutrality and conflating policy positions with abridging, enlarging, or modifying a substantive right, the rulemakers are comfortable engaging in a reform process expressed only in the most banal platitude—improving the system—or in the language of the Rules themselves, pursuing just, inexpensive, and speedy disposition of civil matters.^[195] Unfortunately, this vagueness hamstrings the information-gathering process that flows from it and masks the inevitable policymaking the Advisory Committee is charged with undertaking.

We will return to the expressed aim here—access to court—when we examine the framing questions of the Mini-Conference, where it will become clear that court access does not feature as a framing question or concern for deliberation.

Next, Judge Koeltl clarified that the Advisory Committee aims to achieve something that will make a difference through its Rules Amendments. “The hope is that adopting a number of changes all at once, as part of an integrated package, can have an important impact.”^[196] The implication here seems to be that, because there is disagreement as to whether or not the system is in crisis and warrants substantial changes, the Advisory Committee was hoping to make a number of smaller changes that would nonetheless have a substantial impact.^[197] Accordingly, Mini-Conference participants were instructed: “Each individual proposal must be scrutinized, both for its intrinsic quality standing alone and for its place in the overall package.”^[198]

Perhaps the most significant problem with the Advisory Committee’s approach in gathering information from attorneys and practitioners was its failure to recognize what particular information these participants possess that might be helpful to the Committee’s inquiry. After presenting the broad goals of the new rulemaking effort, Judge Koeltl explained that participants gathered together to share their wisdom with the Advisory Committee ought to leave the source of that wisdom outside the door.

This directive is important. What theory of expertise, of knowledge, or of information underlies this command? The most obvious basis for participant knowledge would be experience. But if we aim to benefit from the experience of participants, this would require discussion of that experience in representation of particular clients with specific interests and needs, served either well or, perhaps, not so well by the litigation system. The understanding of the Advisory Committee, on the other hand, appears to be that good information about litigation must be free from the taint of private interest, from the suspect miasma of adversarialism. Thus, the primary focus is on leaving clients at the door.

This focus, unfortunately, is a fatal flaw, as it denies as improper precisely the information that litigants are best placed to furnish the Committee. Indeed, the information that the Advisory Committee—dominated as it is by judges who are non-adversarial actors—can obtain is likely to be devoid of any insights that come from understanding the specific interests of a litigant in an adversary process.

Hence, it was important to emphasize at the outset that “participants have been invited knowing that they will leave their clients at the door.”^[199] This instruction makes clear that the Advisory Committee did not operate from the assumption that attorneys’ understanding of their clients’ interests and experience of the rules from that particularized perspective was useful information that practitioners could provide the committee and that it otherwise would be unlikely to understand.^[200]

In fact, quite clearly, based on this preamble, the Advisory Committee deemed an interest-informed perspective improper. The phrasing that participants “have been invited knowing” suggests that perhaps the Advisory Committee selected out anyone it deemed too partisan or unable to participate within the polite parameters the Committee wished to establish. Nonetheless, what information the Committee did think it could obtain from the Mini-Conference remains unclear, as their approach reflected ambivalence. Immediately following the directive to leave clients outside the discussion, participants were told that “advice is particularly important on how things will work in practice.”^[201]

The Mini-Conference proceeded after these opening remarks with an introductory discussion in which all participants were invited to introduce themselves and state their initial thoughts briefly. These initial statements were divergent and touched on a number of issues. The divergence seemed to highlight the difficult task before rulemakers charged with maintaining general trans-substantive rules of procedure. Judge Kravitz summarized the initial remarks by describing (again with a high degree of generality and vagueness) the expansiveness of the federal docket from “a small fraction that involve large amounts of money . . . rang[ing] down to cases that involve only a few dollars. They also include cases that affect interests and rights that are not easily measured by dollar values.”^[202] One thing that became obvious from simply the initial observations of this range of attorneys is not only that the docket is varied, but the variance matters for operation of the rules. Judge Kravitz picked up on this and immediately suggested that not everything can be handled through rules changes. But this reasoning ultimately dodges the actual decision that faces the Committee, which is the question of which cases the Committee is going to prioritize with its rules. It is reasonable to try other methods aside from rule amendments—though those should also be thought through for varying effects on different types of cases—but it doesn’t resolve the Committee’s problem, which is that it needs to find some way to get a handle on all these differences.

Without a clear set of aims, the Mini-Conference produced a predictable opinion soup. Participant after participant simply provided their impressions of what works, what does not work, and how proposed changes might be good or bad. Some attorneys provided some sparse context as to their practice area and how it informs their experiences of the system and assessment of the proposals.^[203] Many did not even provide this minimal level of context.

Even those who did explain their practice area, however, did not provide enough detail to be really useful. For example, after explaining generally the practice areas in which he works—mass disaster, catastrophic injury, and commercial cases—Wayne Mason, president of Lawyers for Civil Justice, proceeded to speak generally about the “huge problem” created by electronic discovery:^[204] “There is no economic incentive to be reasonable.”^[205] Then he shared a sparse anecdote that no doubt felt like a specific example—precisely the type of case-based information the Committee would like to hear—but the example lacked vital information. Mr. Mason explained that in a recent case, “e-discovery cost my client $5,000,000, including $66,000 a month just for storage and access.”^[206] There is no reason to have expected more from Mr. Mason, given the guidance provided by the organizers of the Mini-Conference, but, unfortunately, this made it difficult to learn anything from the example provided.

When it came to the Mini-Conference discussion of Rule 26(b)(1) and the scope of discovery, Judge Koeltl was much less vague. It appears the subcommittee chair was seeking to explain the Committee’s work to that point and its reasoning in providing the sketches presented. In this vein, he explained that the Mini-Conference had “devoted a lot of time to these issues.” He declared that lawyers and courts talk a lot about the broad scope of discovery without raising the “proportionality limitation” found in then-26(b)(2)(c). On this basis, “the [Mini-Conference] thought it would be useful to incorporate the concept of proportionality into the 26(b)(1) definition of scope.”^[207]

In addition to describing the changes rulemakers wished to make, Koeltl added: “The cross-reference in the final sentence of present (b)(1) is not only redundant, but apparently is ineffectual as well.”^[208] After this, Koeltl leapt right into the different proposal sketches. At no point was the Mini-Conference’s initial analysis qualified as simply that—an analysis that might be wrong—much less any explicit request for clarification as to whether their descriptive account was accurate. For example, participants might have been asked whether they ever made reference to proportionality and when the most recent time was that they could remember.^[209] Whether parties truly failed to notice the proportionality language, or forgot about it because it was too far down in the rule, seems like the type of information that one could glean from asking participants of their own experiences with recent cases in which they had faced a discovery dispute.

The structure of the framing questions and the presentation of the rules’ sketches containing the proposals did not focus participants either on examining the accuracy of such assumptions or on providing feedback specifically informed by participants’ concrete experiences. The proposal concerning scope of discovery had three elements: (1) incorporating proportionality in the Rule 26(b)(1) definition of the scope; (2) considering removal of any opportunity for discovery relevant to the subject matter of the action, then permissible where good cause is shown; and (3) considering narrowing the reference to discovery “reasonably calculated to lead to the discovery of admissible evidence.”^[210] Just as with its impressions that the “proportionality” limitations of 26(b)(2)(C) are neglected, ignored, or invisible to parties, the Advisory Committee did not focus its Mini-Conference on interrogating its own assumptions as to party behavior.

It is no surprise, then, that the ensuing discussion of the proposed amendments to Rules 26(b) proceeded in generalities and familiar chestnuts as to litigation as a whole, discovery specifically, and lawyer behavior.^[211] The first comment, by a judge, observed that lawyers do not consider limitations and instead focus on whether discovery may lead to discoverable evidence as the standard for its permissibility. “It is a mindset,” he continued, “They even ignore the requirement of relevance.”^[212] Mini-Conference minutes record that “many feel that the 2000 Amendments have not significantly narrowed discovery practice.”^[213] A judge observed that “everyone seemed agreed that (b)(2)(C) is not doing the job.”^[214] In addition to these generalities that the present rule is “not working” and that “everyone agrees” to this, there were further assertions of what “must be done.” A few examples will suffice: “It is critical to narrow the scope of discovery”; “[w]e need a ‘noisy change’ in the scope of discovery”; “[w]e should publicize that this is a change”; “[t]he concept [of proportionality] is already there in 26(b)(2)(C), but litigants do not follow it and judges do not cite it”; “one issue is there are many court decisions on the scope of discovery that do not cite to (b)(2)(c), and lawyers do not raise it.”^[215]

The Duke Subcommittee, the Advisory Committee subcommittee tasked with marshaling all the efforts from the Duke Conference into concrete proposals, selected participants for the Mini-Conference, which provides some indication of (1) the type of feedback they hoped to obtain and (2) the sectors of the profession that the Duke Subcommittee understood have information relevant to the rulemaking question.^[216] These choices reveal certain expectations about what we need to know concerning civil litigation and who we need to hear from in order to know it. What conclusions can we draw from the list of invitees? One thing that is apparent is that the Duke Subcommittee values the input of former rulemakers. The selections also seem to be dominated by a desire for balance between plaintiffs and defense bar, which is unsurprising as this is a factor of litigation sociology that has tended to be understood as central to understanding the dynamics of litigation and debates about procedural reform.

B.Mini-Conference Materials

The materials provided to participants in advance of the Mini-Conference are also revealing. The Duke Subcommittee provided attendees with sketches of possible rules amendments and an accompanying set of questions.

The questions concerning Rule 26 and proportionality were as follows:

Does the concept of “proportionality” in civil discovery have meaning to civil litigants and judges?^[217] Do you find that it is regularly considered in the cases that you handle? Would adding proportionality as an explicit limit on the scope of discovery in the first sentence of Rule 26(b)(1) increase the effectiveness of civil litigation? Is this more effective than the cross reference to Rule 26(b)(2)(C) in the final sentence of Rule 26(b)(1) which does not explicitly use the term “proportionality” and should that sentence be deleted?^[218]

Perhaps even more telling is the next paragraph. After focusing on the concept of proportionality, the materials then backed out more broadly and asked about the amount of discovery. They did so in this fashion:

Some are of the view that civil litigators—even those who are efficient civil litigators—conduct more discovery than is necessary to try a case, settle it, or brief a motion for summary judgment. The amount of such discovery is plainly far in excess of what occurs in a criminal case. In your view, do civil litigators conduct more discovery than is necessary and what, if anything, should be done to curtail unnecessary discovery?^[219]

This is a poorly constructed prompt. Indeed, the 2009 FJC report that the Advisory Committee itself commissioned had been designed precisely to avoid general questions such as this—ones that request the participant to draw on their own generalizations about the behavior of others, precisely because such generalizations are prone to predictable errors in cognition (specifically, availability bias and confirmation bias).^[220] Indeed, the FJC study design and the problem of cognitive biases in general opinion surveys was explained directly to the Advisory Committee by the study’s authors.^[221] Nonetheless, after having a survey designed expressly for its purposes in determining to what extent a cost-and-delay problem exists in civil discovery, and having that study’s design explained and indeed the pitfalls of general opinion surveys outlined, the Mini-Conference returned precisely to the general opinion question for its follow-up meeting with the bench and bar.

Yet the problems with this prompt did not end at the general opinion design. Here, the framing of the prompt itself generated responses reflecting predictable bias. The prompt purported to leave open the question as to whether over-discovery is common and a problem, but it was framed by lead-up prompts focusing on proportionality, thus preparing the participant to center proportionality, and therefore the natural possibility of excessive discovery in its considerations of whether there is too much discovery. With this lead-in, the prompt began by stating the opinion that excessive discovery is a problem. It then suggested that this assertion is reasonable since “such discovery is plainly far in excess of what occurs in a criminal case.”^[222] By juxtaposing these two thoughts, (1) some believe even those with good intentions, whom we might label “efficient civil litigators,” nonetheless engage in over-discovery, and (2) discovery is “plainly far in excess” of discovery in criminal cases. The prompt communicates that civil discovery is far in excess of what it needs to be or at the very least that the amount of discovery taken in criminal discovery is relevant and comparable to civil discovery.

This, of course, is nonsense. Anyone who considers the question for even a short while will recognize for themselves that in finding information to put forward a case, the prosecution in a criminal case does not rely on or, for that matter, even use discovery. It instead has available to it an entirely funded investigative arm with legal authority to question any relevant actors and to use the vast resources of the state in that effort.^[223] Moreover, limitations on discovery provided to criminal defendants are a constant source of controversy in the area of criminal procedure.^[224] But the juxtaposition here, and indeed its cursory, seemingly off-hand reference, does not invite further reflection but rather simple comparison.

Finally, the assumption that we are being invited to make in the prior two sentences, is made explicit in the last sentence. Here, any pretense of testing a common opinion is dropped when the participant is asked simply: “What, if anything, should be done to curtail unnecessary discovery?” Perhaps, the question suggests, it will be best to do nothing, but there is no doubt that we have “unnecessary discovery” that we should consider curtailing.

The Mini-Conference prompt concerning discovery cost allocation exhibits similar framing problems. First, a normative assessment of discovery cost allocation is presented (there is too much discovery, and this is in significant part the result of the American rule’s application to discovery—that is, the discovery-producer bears the costs). That normative assessment is placed in the mouths of “some,” as in: “Some believe that documents requests are overused because they are essentially free and impose the costs and burdens on the producing party.”^[225] The Mini-Conference studiously avoided assessing the truth of this claim—which may be precisely the correct approach if the aim is to interrogate the truth of the claim by seeking input from participants. However, the question bulldozes forward, after articulating this belief held by “some,” to assume its truth and consider ways to address what is now no longer a belief but a fact of the discovery process: “Are there any reasonable ways of limiting the amount of free discovery, and then imposing on the requester the cost of the additional discovery, subject to reasonable exceptions for parties who could not afford to pay for the opponents’ production costs?”^[226]

Now some might deem it unfair to subject to such scrutiny the Mini-Conference materials organized by a committee of mostly judges, with the odd practicing attorney or scholar thrown in. This is a service committee of busy and hardworking legal professionals, making their best effort to engage in the task of rule-reflection and rulemaking in a timely fashion and to run a process that will have a foreseeable end. Surely their efforts to reach out to the public for feedback need not meet social science standards for data collection and survey construction? That is, of course, not what this Article intends to suggest. The close attention paid to these questions and the framing of the Mini-Conference is not an argument for turning the whole effort over to sociologists, but rather is aimed at highlighting the connection between institutional and professional needs of the Committee and the procedural information that it is able to identify or believe is pertinent to the inquiry at hand, in this instance, discovery.

The problems that arise with respect to the proportionality question are repeated with respect to cooperation.^[227] In this question too, the Mini-Conference re-solicits generalized opinions from its participants, opinions already surveyed in advance of the Duke Conference. The form of question invites unreliable reflection.^[228] The lurking problem with the approach to proportionality, cooperation, and the remainder of discovery-related reforms is rulemaker efforts to avoid any explicit commitments to what the procedural system should look like, what it might try to accomplish, and how it ought to change. Nonetheless, rulemakers are embarking on a reform project, something that is difficult to do without any conception of what is good and what can or ought to be better.

The normative commitments here bubble up to the surface in the final questions provided to Mini-Conference participants. The second to last question asked participants to provide their additional thoughts as to how rulemakers can make change in addition to, or aside from, rules amendments. But here, finally, the change they envisioned was, perhaps inadvertently, spelled out: “What can be done to educate the Bench and Bar about the current means of reducing the cost and expense of litigation and the implementation of any changes?”^[229] The purpose, it seems, of all this discussion was cost reduction. This was not as well thought out as it might be—whose costs are rulemakers hoping to reduce, for example? However, it is at least a goal that, coupled with the constant exhortations to consider whether proposals will make the rules “efficient and effective,” we can develop a picture of what the rulemakers’ aims are. The final question sums this up: “What other suggestions do you have for improving the efficiency and reducing the cost of civil litigation in federal court?”^[230]

The questions framed the substantive issues to be discussed at the Mini-Conference: proposed amendments to the Rules that were presented to participants in the form of “sketches.”^[231] The Advisory Committee provided a little more clarity as to its normative concerns when presenting its sketches of the Rules amendments. As its name indicates, the charge of the Mini-Conference was to consider how best to convert the enormous undertaking of the Duke Conference into actionable change to the conduct of civil litigation. Thus, the sketches led off by explaining the “most prominent themes developed” at that conference, “frequently summarized in two words and a phrase: cooperation, proportionality, and ‘early, hands-on case management.’”^[232] Pilot projects and bench and bar trainings were already underway so all that remained for consideration were amendments to the Rules. The introduction to the sketches explained that, although some proposals were small on their own, altogether these would “encourage significant reductions in cost and delay.”^[233] In explaining the set of changes proposed to the scope of discovery, the materials explained that

[t]hey begin with alternative means of emphasizing the principles of proportionality already built into the rules. More specific means of encouraging proportionality are illustrated by models that reduce the presumptive number of depositions and interrogatories, and for the first time incorporate presumptive limitations on the number of requests to produce and requests for admissions.^[234]

The Duke Subcommittee expressed interest in a mini-conference as a form of “informal outreach to bar groups . . . to gather perspectives on how the proposals are likely to play out in the trenches of adversary litigation.”^[235] The information, then, that rulemakers understood could be gleaned from a mini-conference was of a particular type. The Mini-Conference sought to feature members of the bar (“bar groups including a mini-conference”) and its purpose was to provide a “view from the trenches,” that is, to assist the Duke Subcommittee in assessing how these would play out in practice and specifically in “adversary litigation.”^[236] This articulation is important because we will see shortly that the information-gathering process did not aid in actualizing this view.

If anything, the design of the Mini-Conference served to suppress the adversarial nature of litigation and to limit the insights “from the trenches.” To the extent the Mini-Conference questions invited generalized impressions and opinions as to what other attorneys must feel about particular concepts or rules, their conference design solicited the opposite of what it intended in detailed observation of the adversary context.

The opposite of such generalizations would be particular representative factual scenarios. Not only would this remove the likelihood of unreliable generalizations (or solicitation of anecdata), but it would also provide valuable information. The federal civil litigation system carries a knowable and limited docket, with reliable information concerning the subject matter of disputes. Moreover, there is useful information as to case characteristics aside from the substantive legal area that is likely to affect the application of procedural rules. Thus, understanding representative cases does not require rulemakers to make a decision as to whether to privilege the median case, or an average case, or the most “significant” cases, whether weighted to cost (often used as a litmus test for significance) or a judgment as to subject matter in issue. Rather, the docket already has clear categories of cases that can be used to consider and assess likely applications of existing procedural rules.

Rulemakers are already engaged in a process in which they ask themselves and interested parties to consider procedural mechanisms and to engage in an inquiry as to how those mechanisms are operating in practice. Concomitantly, they are asking themselves and others to engage in an imaginative exercise in which they consider how proposed future rules will likely affect litigation. Indeed, the Rules sketches presented to the Advisory Committee and later to Mini-Conference participants provided exactly such an attempt to structure this imaginative exercise. The questions shared with participants in advance of the Mini-Conference provided framing and guidance on how to imagine. Often this imaginative exercise is highly unstructured. The inquiry is simply “affect litigation” in an unbounded manner, rather than to imagine how it will affect litigation in specific contexts that we can reasonably predict will arise. The important question, and the one the rulemakers are well-situated to identify, is what contexts one can reasonably predict will arise.

Rather than the unbounded imaginings or “non-empirical” information gathering that rulemakers deploy today, we might consider the sociology of the litigation that are likely to impact the issues that the Advisory Committee is attempting to address. To understand how such a sociological approach might be utilized we should begin with the categories of substantive law treated in the federal courts. Here, docket information is our guide.

C. On Neutrality

Perhaps there is nothing more indicative of the Advisory Committee’s desire to cloak its role in and responsibility for the 2015 amendments than its public declarations about the purpose and effects of the changes. Much of the members’ communications were at pains to assert that these amendments were minor and made substantively little change to the structure of the rules.^[237] This seemed an odd defense since, if the changes to the Rules were so minor and the effects predicted to be marginal, why had the Committee invested so much time and energy into making them—and why go forward with them over substantial, vocal opposition? Perhaps the best indicator of this dodge—this effort to erase its own role in making some structural changes to the discovery apparatus, for what are apparently policy reasons—is the 2015 Committee note (“Note”) following Rule 26.^[238] The Note is at pains to suggest that the rulemakers here are doing nothing more than restoring the “original purpose” of the 1983 amendments to Rule 26.

The controversy over amendments to Rule 26(b)(1) revolves, in substantial part, around the introduction of a proportionality requirement into the definition of discovery scope. Much of the language setting out the proportionality standard was taken from Rule 26(b)(2)(C)(iii), language adopted in 1983 (limitations on discovery). The Note states that: “The 1983 provision was explicitly adopted as part of the scope of discovery defined by Rule 26(b)(1).”^[239] The clear suggestion here is that the 2015 amendments, incorporating proportionality into the scope of discovery, do nothing more than honor the 1983 change. This is consistent with the company line: “these are minor.” But it goes further. The Committee suggested it was only giving effect to the desires of the 1983 amendments, but also, the (policy) choice was not the Committee’s—it was the determination of rulemakers from thirty-two years ago:

The present amendment restores the proportionality factors to their original place in defining the scope of discovery. This change reinforces the Rule 26(g) obligation of the parties to consider these factors in making discovery requests, responses, or objections . . . Restoring the proportionality calculation to Rule 26(b)(1) does not change the existing responsibilities of the court and the parties to consider proportionality.^[240]

The Committee’s explanation of its amendment “restoring proportionality” relies yet again on the cost-and-delay narrative, reproducing the 1993 reference to “the information explosion of recent decades greatly increas[ing] both the potential cost of wide-ranging discovery and the potential for discovery to be used as an instrument for delay or oppression.”^[241] Although the Committee should have been well-informed that such cost-and-delay assertions have never been shown to be more than that—mere assertions repeatedly uncorroborated by empirical investigation—nonetheless the Committee implied that what was concerning in 1993 must be downright terrifying in 2015: “What seemed an explosion in 1993 has been exacerbated by the advent of e-discovery.”^[242]

After this lengthy effort to hitch itself to the wagon of the 1983 rulemakers, asserting no difference between its just-completed reform effort and those of three decades earlier, the Note serves as a tell of just how policy-oriented a concept the new proportionality standard is. The Note expends a full page explaining how different proportionality factors might consider “philosophic, social or institutional terms.”^[243] The precise terms of this explanation are less relevant for our purposes, although it is clear that the Note seeks to address voluminous comments raising concerns with the concept of proportionality as a standard. The Committee emphasized that its proportionality concept was not meant to lead to an exclusive or myopic focus on how financially large a case is or how much the discovery costs in monetary terms.^[244] The aim appears to have been to reassure in Note language what could not be accommodated in rule language—that the policy aims of the Committee were not intended to tilt the balance of the procedural system entirely in favor of high-dollar monetary claims, nor to eviscerate any commitment to “public policy” litigation. Indeed, these amendments are not to be deemed an introduction of crass assessment of wealth differentials into the procedural standards: “[C]onsideration of the parties’ resources does not foreclose discovery requests addressed to an impecunious party, nor justify unlimited discovery requests to a wealthy party.”^[245] In the face of public outcry that rules amendments would produce material effects, ones that exacerbate procedural inequality, committee members aimed largely to assure their neutrality rather than to defend the authoritative exercise of their power. Under the current epistemic consensus, the rulemakers understand that neutrality determines whether the exercise of power is neutral rather than the substantive ends of reform. As we will see in the next Part, the fixation on neutrality extends to how the rulemakers understand adversarial attorney behavior as well.

VI. Strategic Decision-Making

The need for neutrality has led the Advisory Committee to insist that right answers exist procedurally without requiring the system to pick among distributive outcomes. That requirement demands a conception of procedural good devoid of interest. There are consequences to this. For one, it seems that a commitment to neutral procedural reasoning entails ignoring a fundamental feature of the procedural system: its adversarial character. In an adversarial system, interest-based decision-making, including strategic behavior, is a feature, not a bug. To utilize rules in furtherance of a client’s interests is the role attorneys are required to play. This fact of U.S. litigation often disappears from Committee view. Consequently, only some strategic behavior is problematized and only some of the time.^[246]

The core rationale for the 2015 amendments (and perhaps especially the proportionality concept in Rule 26 and sanctions in Rule 37(e)) was rooted in a concern that litigating parties may engage in strategic decision-making when litigating. This was an odd problem to identify for two related reasons. First, our procedural system is predicated on the notion of adversarialism—zealous advocacy by attorneys on behalf of parties is a cornerstone of the system. Second, as has been well noted, cases that come before the courts are a highly structured negotiation between the parties.^[247] Accordingly, the rulemakers’ focus on stamping out opportunities for strategic decision-making appears to convert a feature of the legal system into a bug.

The pathologizing of adversariness was made worse by rulemakers’ inconsistent recognition of strategic behavior. As a result, the problems that might have been resolved through procedural change were identified through a distorted lens. This distortion, resulting as it does from inconsistent attentiveness to strategic behavior, also results in a substantial appearance of bias, damaging the legitimacy of the rulemaking process and of the federal courts themselves.^[248]

The effort to root out strategic litigation behavior stems from the same imperative as we saw in the empirical research context. Procedural rulemakers operate under the need to deny their own policymaking role and to uphold neutrality. The preferred decision-making method for performing such neutrality is: (1) embarking on reform initiatives with poorly articulated hypotheses, preferably as problems raised to them by others; (2) grounding those initiatives in “empirical data” that, by virtue of an unreflective engagement with the category “empirical,” is understood to provide neutral and objective information by definition; and (3) generally ignoring (at times even denying) the non-neutral, distributional effects generated by procedural choices. The Advisory Committee’s tendency to treat strategic litigant behavior as an aberration arises out of the impetus to deny strategic interest in litigation itself, or rather to ignore the yawning inequality with which the facially neutral procedural rules must intersect. This posture towards strategic behavior both contributes to and is exacerbated by the rulemakers’ approach to procedural information. It denies the rulemakers the benefit of using outreach to litigants as a means for understanding their material interests and using that knowledge to evaluate the operation of current and proposed rules. Rulemakers simultaneously overestimate what is objectively knowable or identifiable about litigation and underestimate or underspecify what type of knowledge can be gleaned from litigants and contextual analysis.

A. Factoring in Interest-Based Decision-Making

Certainly, we do not have enough empirical study of the functioning of our civil system and would benefit greatly from the substantial strides being made in that vein.^[249] Rulemakers’ recognition that litigants are a useful source in thinking about the needs of the procedural system is also a helpful starting point. However, the imprecise understanding and categorization of procedural knowledge impedes the utility of litigant sources. It is important to be precise about what empirical information is and to be realistic about what it can tell us.^[250] There is also much to be gained from practitioners, but most of these gains are unrealizable if rulemakers continue to engage practitioners in an ad hoc manner that elicits impressionistic accounts and underspecified generalities.

The Committee’s inconsistent attention to the centrality of strategic decision-making in litigation is well-illustrated by its deliberations over proportionality and its proposed amendment to Rule 26(b)(1).

Recall that the Committee began its inquiry into discovery cost and delay with a recognition that the federal courts’ liberal discovery rules carry with them the potential for high costs created by broad discovery. The expansive ability to request discovery opens up the danger of strategic manipulation of discovery requests to drive up the costs of pretrial litigation, thereby enhancing the party’s settlement position.^[251] One participant in Committee deliberations described this dynamic as the “impositional benefits of discovery.”^[252] Here, then, there was an open recognition that parties to a litigation are likely to deploy rules strategically to their tactical advantage in furtherance of party goals. The Duke Subcommittee understood its task to be adjusting the Rules to make “discovery quicker, less expensive, and more efficient.”^[253] To do so, the Duke Subcommittee believed it would help to make the concept of proportionality “more prominent.”^[254]

Early consideration of proportionality indicated Advisory Committee members’ understanding of the importance of the particulars of any given litigation context in evaluating the validity of the proposal. As we would expect of highly experienced and sophisticated Committee members, discussion indicated an appreciation of the difficulties in generalizing about costs in the variegated dockets of the federal courts. Members raised concerns about differences in the size and complexity of cases and how those might affect the very question of whether proportionality needs greater emphasis. As one member explained, the results of the FJC Closed-Case Study show that “a large number of the lawyers said the cost of the case actually involved in the survey was appropriate.”^[255] Another followed up to suggest that empirical data indicates “very expensive litigation at the high end. Perhaps there is need to study and remedy the most expensive cases. But attempted reform should not mess up things that people are satisfied with. Additional requirements, if needed, should not be imposed on simpler cases working well currently.”^[256]

Size and complexity of cases were instinctively recognized as important factors to consider when pondering changes to discovery. This instinct appeared to be corroborated by the differing reports of discovery cost in the data collected for the Duke Conference.^[257]

Compared to the ACTL and ABA proposals is the interesting FJC finding that there is little discovery in most cases, and that most lawyers think the level of discovery is appropriate to the circumstances of the particular cases in the closed-case survey. The problems tend to concentrate in high-stakes cases, where lawyers tend to be more assertive.^[258]

In addition to the Committee’s interest in aspects of a litigation that might affect cost, in subcommittee discussions immediately following the Duke Conference, Committee members also recognized a need to understand the nature of the costs that proportionality would aim to alleviate. Some discussion suggested that the proportionality principle might help minimize problems of access to courts.^[259] Others pointed out that even if litigation cost is largely proportional to the stakes involved, even the median cost is still unaffordable for many litigants. “Absent public subsidy, it does not seem possible to design procedures that will bring costs down to a level that can be managed by most potential litigants.”^[260] Nonetheless, “[i]t remains important to attempt to control costs as far as can be done.”^[261] Another member suggested that the high-end costs and the median costs are not unrelated. When considering the relationship between cost and access to the ordinary citizen, it is important to recognize that in some cases the stakes are painfully high: “[In foreclosure cases] many defendants can’t afford a lawyer. It can be the difference in losing your home.”^[262] Yet, the member pointed out, “[C]ommittee discussions and lawyer dissatisfaction, regularly focus on the top of the pyramid. Federal court will always be a luxury court to the ordinary citizen. Revising the rules will not affect that problem.”^[263] The connection between the top of the pyramid and the needs of the median system is that “aggregate litigation will bring many ordinary people to court.”^[264]

What is notable about these observations is their attentiveness to how dynamics of particular litigations will affect the workings of discovery, cost, and the very concept of proportionality. In the above comments, we find important complications to the customary generalities. Committee members identify that that there is a real question to be asked whether discovery cost control has any bearing on problems of access to courts. In fact, by tying ordinary access to aggregate litigation, one Committee member appears to suggest that limiting discovery in high-end cases may actually serve to limit access in aggregate (hence, “high-end”) cases. Furthermore, member observations, for example in raising foreclosure cases, also complicate the plaintiff-defendant binary that is frequently used as a shorthand for understanding which litigants will support or oppose broad discovery and discovery cost control, among other issues.

Based on these preliminary impressions, some comments in the initial Committee meetings suggested a need for more granular information.^[265] Perhaps it would have benefited the Committee to identify system factors versus case factors that influence litigation cost, as well as factors that make settlement more likely.^[266] “We don’t know much about the impact of procedure on litigation of complex commercial versus priced out at $15,000-types of cases. There is a lot we do not know about the operation of the rules, and a lot to be learned.”^[267]

But this attention to the contextual dynamics of litigation, including both the strategic interests of the parties and the details of litigation, dropped out of the discussion much of the time. Instead, we see Committee deliberations reverting to a formalistic and static understanding of the underlying issues. By the time of the November 2011 Advisory Committee meeting, the proportionality issue was being discussed in these terms: “[T]he FJC survey for the Duke Conference suggests that for a great many cases, discovery is held within appropriate limits proportional to the needs of the case. But it also seems clear that discovery can run beyond what is reasonable.”^[268]

Whereas initial deliberations touch upon some particular litigation contexts, such as foreclosure described above, and aggregate versus median-case litigation, Committee discussions very quickly cease to identify particularities of a suit. Parties are flattened to plaintiffs and defendants. The aim of proposals is to achieve the “right” amount or only “necessary” discovery—treating discovery volume as an objectively knowable fact. Of course, the proposal’s multi-factor test is an effort to address the contextual nature of a proportionality inquiry, or of an inquiry into discovery needs, but commenters raise problems particularly with the multi-factor test’s inability to account for the subjective and context-dependent character of appropriate, or necessary, discovery.^[269] Ultimately, the Committee’s own assessment of its proposals fails to grapple with the impact of these contextual factors in any systematic or rigorous sense.

The Advisory Committee has been consistently unable to acknowledge strategic behavior of litigants. Rather, much of the time Committee deliberations identify strategic behavior itself as the problem:

The importance of best practice guides and other forms of encouragement was noted by recalling a theme that sounded periodically throughout the conference. If we simply followed present rules in good faith, there would be no need for new rules that attempt to coerce the behavior that the rules now contemplate.^[270]

To resolve the problem thus conceived, the Committee looked to develop amendments that remove the strategic element.

Inattentiveness to the inherently strategic engagement of parties in litigation left important questions unexplored. For example, the Mini-Conference’s proportionality deliberations began from the premise that proportionality is “already in the Rules,” but it is underutilized. As the Committee chair explained, there are “grounds to suspect that judges often fail to enforce proportionality, in part because parties do not raise it.”^[271]

The Advisory Committee should have given pause to the recognition that parties do not raise proportionality as an objection to burdensome discovery. In a system that we understand is driven by self-interested parties with counsel paid to fulfill client goals, why would parties not raise it? What are the situations in which we would expect it to be raised? What are the reasons that would deter such behavior? One explanation might be that there is not a lot of disproportionate discovery. This would be confirmed, in part, by the FJC Closed-Case Study. However, what about the cases in which attorneys and/or clients are experiencing discovery as disproportionate? Why are counsel not raising discovery proportionality in these cases?

Another explanation, which might support a rule amendment, could be that the proportionality argument is likely to be shot down and that the costs of making the argument, whether monetary or paid via the repercussions of annoyance caused to the judge, might make it a poor tactical choice. Are there other explanations? These types of considerations would be beneficial for the Mini-Conference to think through and might help guide not just the analysis but any information-gathering commissioned, whether through informal mini-conferences or formalized surveys and studies, in the process.^[272]

The inability of the Committee to incorporate its own complicating observations is not surprising, but rather a product of the Committee’s approach to developing and processing information relevant to the discovery-cost reform project. The proportionality discussions illustrated how its binary approach to information prevented the Advisory Committee from gaining the full benefits of the information it does gather, and perhaps prevents it from seeking out all the information it needs. At the outset, the Committee was concerned about understanding the differences between large and small, or complex and simple, cases and how the discovery rules may affect these differently. Similarly, they were interested in recognizing the difference between costs that prohibit access and those that may not affect access but may otherwise impose excessive or avoidable cost on litigants. These observations, at times, also pointed to how such questions might connect to the structure of the litigation—who the parties are and by whom they are represented. Once the process was really underway, however, the Advisory Committee proceeded with open-ended input or feedback from legal professionals soliciting general and impressionistic observation yet again.

B. Possibilities for Future Rulemaking

While perfect foresight, or the assessment of every iteration of how a rule might play out, is clearly impossible, methods can be developed to understand the particular through more than simply impressionistic observation. Approaching contextual dynamics with greater rigor will lead to a more robust analysis that takes into account a much fuller picture of civil process. How is it possible to do so? Though there are undoubtedly many analytical tools that may aid in this endeavor, here I propose three tools.

First, the Committee should seek to understand the implications of cognitive biases when considering the operation of an existing rule and any subsequent proposed amendments.^[273]

Second, everyone knows that context matters to the process of a litigation and its ultimate outcome, but how might we structure our thinking about this context in order to make more rigorous and reliable assessments? It will be helpful to look to what I call the “sociology of the litigation.” The variables to attend to can continue to be developed and may well evolve as the litigation landscape evolves. For now, however, we can identify factors that appear to clearly affect the relative negotiating position of the parties with respect to discovery and cost.^[274] Sociological factors that should be systematically considered by the Committee when it addresses discovery and cost are: information asymmetry; size of the parties; nature of legal representation (fee structure); nature of legal representation (size of legal organization); and non-monetary interests involved.^[275] One question that these factors raise is the extent to which a plaintiff-defendant lens is useful. To the extent that the Committee has consistently attended to sociological factors, it has done so almost exclusively through the plaintiff-defendant lens.^[276]

Third, not unrelated to the sociology of the litigation, is the need to grapple forthrightly and consistently with the strategic interests and behavior of litigants. How the above sociologic factors will affect engagement with particular rules can be understood by considering their strategic interests vis-à-vis the sociology of the particular litigation. Recognizing strategic interest is an asset for analyzing how rules work and how they operate to distribute and redistribute power between the parties in different substantive contexts and case structures. Here I discuss those most pertinent to discovery and cost.

By structuring consideration of the particulars and the potential application effect of proposed rules, the Committee will not only sustain a more robust discussion of application but will also develop new avenues for empirical collection and improve the information it obtains from engagements with the bar, such as the Mini-Conferences that it has relied on in the past. Guiding questions for such events can be much more structured and therefore help avoid soliciting mere opinion reflective of availability and confirmation bias, while gaining access to specific, likely scenarios drawing from concrete participant experiences.^[277] Ideally, it should also help the Committee ask incisive follow-up questions when examples or anecdotes are raised.

Conclusion

Procedural scholarship has sought to understand how rulemakers arrive at decisions to implement reform. This interest has only increased as the trajectory of procedural reform has continued to rein in litigation in what has come to be known as the restrictive ethos of procedural reform. Efforts to understand this consistent trajectory have taken two general approaches. The first is as a product of preexisting ideological commitments. The second is as a part of broader political trends. This Article has taken a different approach, one that complements and complicates existing explanations. It examines the modes of reasoning and conceptualization of procedural information deployed by rulemakers to identify and evaluate proposals for procedural reform. Specifically, this Article identifies the categories of knowledge that rulemakers utilize to investigate the functioning of civil litigation. This exploration reveals several trends.

First, the informational categories the civil rules committees utilize are crudely drawn, arranging information as either “empirical or not” and then approaching what it deems non-empirical information in an ad hoc, impressionistic manner. Second, because rule reform relies on broad categories of empirical and observational information, little work is done to specify these categories—neither to understand what empirical research is needed and what questions it can answer nor to consider these same questions as to non-empirical research. Third, information deemed empirical research takes on an outsized importance (relative to the quantity and quality of extant empirical data) as non-empirical or qualitative study flounders.

The dynamics at work are not contingent or coincidental. In fact, emphasis on empirical research is responsive to specific demands made of the rules committees for empirically grounded decision-making. But it is also reflective of broader pressures on and desire of rulemakers to play a neutral, non-policymaking role. The flattening of non-empirical information is part of this. Harnessing qualitative assessments requires firm stewardship and articulation of aims up front, which will then determine appropriate information-gathering design. This up-front control and articulation of aims is contrary to the Advisory Committee’s current approach and, it appears, feels to rulemakers to be a violation of their neutral, custodial role.

The analysis here suggests that rulemaking desires for procedural neutrality have shaped the Advisory Committee’s very formulation of the litigation system and how information can be developed to understand how it is operating. Unfortunately for civil rulemakers, for the federal courts, and for the litigants that appear before them, it seems that the more firmly the civil rulemaking committees cling to an assertion of neutrality, the more they are viewed as politically motivated, ideologically biased, and unfair. This Article suggests some concrete improvements that can be made in rulemaking process: careful and attentive categorization of litigation information; the use of sociological factors of litigation to develop more accurate, concrete, and fully drawn models of how rules operate; and the effects that any proposed amendments might have.

The value of these technical suggestions, however, is contingent upon the civil rules committees’ willingness to engage in explicit discussion of the policy choices that it inevitably makes. To better develop information of any kind (whether it be quantitative or qualitative, empirical or not) about civil litigation requires clarity at the outset as to what proceduralists are seeking to determine. It requires clearly articulated hypotheses, explicitly reasoned procedural goals, and the development of appropriate methods capable of answering those questions.

 

---

*Assistant Professor, UMass Law School. I am grateful to Robert Bone, Brooke Coleman, Richard Freer, Suzette Malveaux, David Marcus, Richard Marcus, Elizabeth McCuskey, Troy McKenzie, Arthur Miller, Aziz Rana, Judith Resnik, Briana Rosenbaum, Mila Sohoni, Norman Spaulding, Diego Zambrano, and Adam Zimmerman. Many thanks also to the participants of the Annual Civil Procedure Workshop at Stanford Law School and the Junior Faculty Federal Courts Workshop at Emory Law School.

1. Christopher Lasch, Journalism, Publicity, and the Lost Art of Argument, Gannet Ctr. J. 1, 1 (1990). ↑
2. See Stephen B. Burbank, Ignorance and Procedural Law Reform: A Call for a Moratorium, 59 Brook. L. Rev. 841, 844 (1993) [hereinafter Burbank, Ignorance]; Bryant G. Garth, Observations on an Uncomfortable Relationship: Civil Procedure and Empirical Research, 49 Ala. L. Rev. 103, 103–13 (1997). ↑
3. See Burbank, Ignorance, supra note 2, at 103–04; Stephen B. Burbank, The Transformation of American Civil Procedure: The Example of Rule 11, 137 U. Pa. L. Rev. 1925, at 1927–28, 1934–41, 1957–59 (1989) [hereinafter Burbank, Transformation of American Civil Procedure]; Marc Galanter, News from Nowhere: The Debased Debate on Civil Justice, 71 Den. U. L. Rev. 77 (1993); Linda S. Mullenix, Hope over Experience: Mandatory Informal Discovery and the Politics of Rulemaking, 69 N.C. L. Rev. 795, 855–57 (1991). ↑
4. See discussion infra notes 5–10. ↑
5. The “other” category is unnamed by the Advisory Committee but can generally be described as impressionistic observation. ↑
6. See Burbank, Transformation of American Civil Procedure, supra note 3, at 1927–28, 1935–36; Burbank, Ignorance, supra note 2, at 842; Linda S. Mullenix, Discovery in Disarray: The Pervasive Myth of Discovery Abuse and the Consequences for Unfounded Rulemaking, 46 Stan. L. Rev 1393 (1994) [hereinafter Mullenix, Discovery in Disarray]; Mullenix, supra note 3, at 854; Jeffrey W. Stempel, Politics and Sociology in Federal Civil Rulemaking: Errors of Scope, 52 Ala. L. Rev. 529 (2001); Marc Galanter, An Oil Strike in Hell: Contemporary Legends About the Civil Justice System, 40 Ariz. L. Rev. 717, 744–47 (1998). ↑
7. Stephen Burbank was one of the earliest to voice this concern forcefully in evaluations of the Advisory Committee’s 1983 Amendments to Rule 11. As he put it: “The Advisory Committee knew little about experience under the original Rule, knew little about the perceived problems that stimulated the efforts leading to the two packages of Rules amendments in 1980 and 1983, knew little about the jurisprudence of sanctions, and knew little about the benefits and costs of sanctions as a case management device.” Burbank, Transformation of American Civil Procedure, supra note 3, at 1927. ↑
8. Id. at 1936 (describing Carrington’s view of rulemaking neutrality as “‘reduc[ing] the level of political interest in procedural rules.’ And according to this view, both substance-specific procedures and empirical investigation of supposedly neutral rules are anathema: the former because they will be likely to attract rather than ‘deflect political attention’ and the latter because data on experience under the Rules may cause organized groups to realize that they have a stake and hence to regard the ‘neutral’ rule as a legitimate object of political interest.”). ↑
9. “[The Advisory Committee] demonstrated a radically different attitude towards the role of social science research in law reform; the committee’s 1993 amendments to Rule 11 were informed by serious, methodologically sound, empirical work.” Stephen B. Burbank, Procedure and Power, 46 J. Legal Educ. 513, 516 (1996). ↑
10. Bell Atl. Corp. v. Twombly, 550 U.S. 544, 559 (2007); Ashcroft v. Iqbal, 556 U.S. 662, 665 (2009). ↑
11. December 9, 2009 Letter from Hon. Lee Rosenthal and Hon. Mark Kravitz to the Hon. Patrick Leahy, Agenda Book at 147–49 (March 2010) [hereinafter Rosenthal Letter]. ↑
12. Garth, supra note 2, at 105; Michael Heise, The Importance of Being Empirical, 26 Pepp. L. Rev. 807, 810–12 (1997). ↑
13. David Engstrom, The Twiqbal Puzzle and Empirical Study of Civil Procedure, 65 Stan. L. Rev. 1203, 1213–19 (2013); Noah Gelbach, Material Facts in the Debate over Twombly and Iqbal, 68 Stan. L. Rev. 369, 370 (2016). Robert Bone has argued persuasively that empirics are not enough. Robert G. Bone, Making Effective Rules: The Need for Procedure Theory, 61 Okla. L. Rev. 319, 319 (2008) (“Many proceduralists today are quick to blame rulemaking failures on a poor empirical understanding of how rules actually operate in practice.”). His is a normative caution: there are non-empirical questions that must be answered and that the empirics cannot answer for us. Gelbach approaches the question from the other direction: given the question is an empirical one, we would anticipate that it can be answered using the econometric techniques designed to answer such questions. Yet in Material Facts in the Debate over Twombly and Iqbal, Gelbach succeeds, frustratingly, in demonstrating that the answer remains indeterminate. Gelbach, supra note 13, at 370. ↑
14. “It is at least hoped that empirical scholarship can more easily separate the normative from the descriptive and better maintain neutrality . . . . [T]he . . . best [empirical legal scholars] endeavor to approach their research questions objectively and their methodology of empirical choice facilitates as much objectivity as is humanly possible.” Heise, supra note 12, at 814. ↑
15. A. Benjamin Spencer, The Restrictive Ethos in Civil Procedure, 78 Geo. Wash. L. Rev. 353, 358–66 (2010); Stephen N. Subrin & Thomas O. Main, The Fourth Era of American Civil Procedure, 162 U. Pa. L. Rev. 1839, 1841–42 (2014). ↑
16. See Subrin & Main, supra note 15, at 1848–49. ↑
17. See Patricia Hatmyar Moore, The Anti-Plaintiff Pending Amendments to the Federal Rules of Civil Procedure and the Pro-Defendant Composition of the Federal Rulemaking Committees, 83 U. Cin. L. Rev. 1083, 1087 (2015); Stempel, supra note 6, at 554–55. ↑
18. Paul D. Carrington, Politics and Civil Procedure Rulemaking: Reflections on Experience, 60 Duke L. J. 597, 618–19 (2010); Galanter, supra note 3, at 747–48; Mullenix, Discovery in Disarray, supra note 6; Stempel, supra note 6. ↑
19. Sean Farhang & Stephen Burbank, Rights and Retrenchment: The Counterrevolution Against Federal Litigation 1–25 (2017). ↑
20. Moore, supra note 17, at 1087. ↑
21. Danya Shocair Reda, The Cost-and-Delay Narrative in Civil Justice Reform: Its Fallacies and Functions, 90 Or. L. Rev. 1085, 1097–98 (2012) [hereinafter Reda, Cost-and-Delay Narrative]; Stempel, supra note 6; Linda S. Mullenix, The Counter- Reformation in Procedural Justice, 77 Minn. L. Rev. 375 (1992) [hereinafter Mullenix, The Counter-Reformation in Procedural Justice]. ↑
22. See Mullenix, Discovery in Disarray, supra note 6, at 1393; Stempel, supra note 6, at 605; Moore, supra note 17, at 1087. ↑
23. Brooke D. Coleman, #Sowhitemale: Federal Procedural Rulemaking Committees, 68 UCLA L. Rev. Discourse 370, 388, 397–400 (2020). ↑
24. For an in-depth consideration of this as it relates to private rights enforcement and the “rights counterrevolution,” see generally Farhang & Burbank, supra note 19. ↑
25. It is possible that a radically altered committee composition, designed to reflect an entirely different class of legal interests, for example a committee stacked with direct legal services attorneys, might produce a different approach to identifying areas for reform and to assessment of any proposals. ↑
26. Two prominent examples from the last few years are the United Kingdom’s referendum to withdraw from the European Union (affectionately termed “Brexit”) and the 2016 election of Donald Trump as President of the United States. One of the top stories in the New York Times on the eve of the U.S. presidential election read: “Clinton has an 85% chance to win.” Josh Katz, 2016 Election Forecast: Who Will Be President, N.Y. Times (Nov. 8, 10:20 PM), https://www.nytimes.com/interactive/2016/upshot/presidential-polls-forecast.html [https://perma.cc/6523-VTJP]. Some of the same dynamics prominent in the context of civil justice reform are also present in current debates about COVID policy, including heightened polarization and the expectation that data can resolve political questions and end debate. ↑
27. This argument is related to Robert Bone’s work arguing for the importance of normative procedural theory in developing and amending procedural rules. Bone, supra note 13, at 319. Professor Bone challenges the procedural rules committees to embrace their role in the evolution of procedural theory and posits that problems with procedural rulemaking are not only reflections of incomplete empirical data or poor comprehension of the empirical picture but also of poor theory. Id. ↑
28. Much of the inequality refracts through state courts, in areas such as consumer debt collection. See Chris Albin-Lackey, Human Rights Watch, Rubber Stamp Justice: U.S. Courts, Debt Paying Corporations, and the Poor 16 (2016); Pew Charitable Trusts, How Debt Collectors Are Transforming the Business of State Courts 13 (2020) (reporting that cases in which both parties are represented have dropped to less than 50 percent of state court dockets); Paula Hannaford-Agor et al., Nat’l Ctr. for State Cts., The Landscape of Civil Litigation in State Courts 31 (2015) (noting that in 1992, all parties were represented in over 95 percent of cases, but reporting a decrease by more than half of representation for defendants by 2015). See generally Jessica K. Steinberg, Adversary Breakdown and Judicial Role Confusion in “Small Case” Civil Justice, 2016 BYU L. Rev. 899, 899 (arguing that the ideal of adversary procedure poses unrealistic demands on today’s civil justice system). ↑
29. This connection is vital because many scholars concerned about unequal outcomes of the civil justice system nonetheless continue to support neutrality and even to demand it. ↑
30. See James S. Kakalik et al., Discovery Management: Further Analysis of the Civil Justice Reform Act Evaluation Data xv–xxviii (1998); Paul R. Connolly et al., Fed. Jud. Ctr., Judicial Controls and the Civil Litigative Process: Discovery 1–3 (1978); William A. Glaser, Pretrial Discovery and the Adversary System 41–43 (1968); Galanter, supra note 6, at 744–47; Deborah Hensler, Taking Aim at the American Legal System: The Council on Competitiveness’s Agenda for Legal Reform, 75 Judicature 244, 248–50 (1992); Mullenix, Discovery in Disarray, supra note 6, at 1443–45; Stempel, supra note 6, at 636–37; Arthur R. Miller, From Conley to Twombly to Iqbal: A Double Play on the Federal Rules of Civil Procedure, 60 Duke L.J. 1, 9 (2010). ↑
31. Reda, Cost-and-Delay Narrative, supra note 21, at 1088. ↑
32. Ashcroft v. Iqbal, 556 U.S. 662 (2009); Bell Atl. Corp. v. Twombly, 550 U.S. 544 (2007). For a discussion of the Advisory Committee’s deliberations leading up to proposing the Discovery Amendments, see Reda, Cost-and-Delay Narrative, supra note 21, at 1087–1102. ↑
33. Emery G. Lee III & Thomas E. Willging, Fed. Jud. Ctr., Preliminary Report to the Judicial Conference Advisory Committee on Civil Rules 2 (2009). For a detailed discussion of these studies and their findings, see Reda, Cost-and-Delay Narrative, supra note 21, at 1088–89. ↑
34. “Perhaps most surprising was the finding that, at the median, the reported costs of discovery, including attorneys’ fees, constituted 1.6% of the reported stakes for plaintiffs and 3.3% of the reported stakes for defendants.” Reda, Cost-and-Delay Narrative, supra note 21, at 1089. ↑
35. Id. ↑
36. Id. ↑
37. Id. ↑
38. See Lee III & Willging, supra note 33; Thomas E. Willging et al., Fed. Jud. Ctr., Discovery and Disclosure Practice, Problems, and Proposals for Change: A Case-Based National Survey of Counsel in Closed Federal Civil Cases 52 (1997). ↑
39. Overwhelming and abusive discovery costs were on center stage in both the Twombly and Iqbal decisions. “[I]t is one thing to be cautious before dismissing an antitrust complaint in advance of discovery, but quite another to forget that proceeding to antitrust discovery can be expensive . . . . [A] district court must retain the power to insist upon some specificity in pleading before allowing a potentially massive factual controversy to proceed . . . . [T]he costs . . . and the increasing caseload of the federal courts counsel against sending the parties into discovery when there is no reasonable likelihood that the plaintiffs can construct a claim from the events related in the complaint . . . . [T]he threat of discovery expense will push cost-conscious defendants to settle even anemic cases before reaching those proceedings. Probably, then, it is only by taking care to require allegations that reach the level suggesting conspiracy that we can hope to avoid the potentially enormous expense of discovery in cases with no reasonably founded hope that the discovery process will reveal relevant evidence.”Bell Atl. Corp. v. Twombly, 550 U.S. 544, 549–50 (2007) (internal quotation marks and citations omitted). “It is no answer to say that a claim just shy of a plausible entitlement to relief can, if groundless, be weeded out early in the discovery process through careful case management given the common lament that the success of judicial supervision in checking discovery abuse has been on the modest side . . . . The basic thrust of the qualified-immunity doctrine is to free officials from the concerns of litigation, including avoidance of disruptive discovery.”Ashcroft v. Iqbal, 556 U.S. 662, 685 (2009) (internal quotation marks and citations omitted). ↑
40. Even with the extent and richness of data on civil justice available to us today, one might still persuasively argue that we do not actually have the data we need to understand the challenges facing civil litigants or to understand the plethora of civil wrongs that are never developed into a civil case. See, e.g., Rebecca A. Johnson & Tanina Rostain, Tool for Surveillance or Spotlight on Inequality? Big Data and the Law, 16 Ann. Rev. L. & Soc. Sci. 453, 454 (2020). ↑
41. “Given the narrative’s continued persuasiveness over a span of decades, in the face of empirical work that appears to contradict its premises, we can safely say that resilience of the cost-and-delay narrative does not depend on its accuracy in reflecting the state of civil litigation.” Reda, Cost-and-Delay Narrative, supra note 21, at 1090. ↑
42. For an early articulation of this argument, see Burbank, Transformation of American Civil Procedure, supra note 3, at 1927–28, 1934–41, 1957–59; Stempel, supra note 6; Galanter, supra note 6, at 744–47. ↑
43. Reda, Cost-and-Delay Narrative, supra note 21, at 1117–18. ↑
44. Id. at 1119–20. Cognitive biases, such as confirmation bias, lead the audience to be more receptive to evidence confirming what it believes it already knows. This helps account for the high confidence that lawyers and businessmen may have in their own estimations of the costs and delays in the litigation system overall. Error results when individuals generalize from the pool of visible cases, available through published opinions, to the much larger pool of unobserved cases. Id. at 1120 n.163. ↑
45. Id. at 1125. In rulemaking, concerns over ideological bias of federal rulemakers stretches back at least two decades. See Elizabeth G. Thornburg, Giving the Haves a Little More: Considering the 1998 Discovery Proposals, 52 SMU L. Rev. 229, 243–44 (suggesting that the Advisory Committee is susceptible to pleas of defense interests because of the loyalties of committee members themselves); Stempel, supra note 6, at 530 (raising concerns about the defense-oriented backgrounds of Advisory Committee members). Burbank and Farhang explore the influence of political ideology on procedural decision-making outside of the rulemaking process, demonstrating that political ideology does map onto procedural policy in federal circuit courts. Stephen B. Burbank & Sean Farhang, Politics, Identity, and Class Certification on the U.S. Courts of Appeals, 119 Mich. L. Rev. 231, 237 (2020). ↑
46. See Arthur R. Miller, What Are Courts for? Have We Forsaken the Procedural Gold Standard?, 78 La. L. Rev. 739, 798 (2018) (describing “pro-business political and defense interests” working to limit litigation and access to courts); J. Maria Glover, “Encroachments and Oppressions”: The Corporatization of Procedure and the Decline of Rule of Law, 86 Fordham L. Rev. 2113, 2113 (2018) (describing the ways that corporate advocacy for procedural reform has led to substantial reorganization of the civil justice system); Marc Galanter, Real World Torts: An Antidote to Anecdote, 55 Md. L. Rev. 1093, 1098 (1996) (an early articulator of this connection). ↑
47. Some of these concerns center on the ways in which federal rulemakers will be open to persuasion in line with their priors. See, e.g., Elizabeth Thornburg, Cognitive Bias, the “Band of Experts,” and the Anti-Litigation Narrative, 65 DePaul L. Rev. 755, 756 (2016) (arguing that political biases make rulemakers more susceptible to particular erroneous conclusions, relying on predictable cognitive short cuts). Others suggest a distortion of the procedural policymaking process, in which rather than the ordinary circumstances of inherently substantive impact of procedural rules, rulemakers instead engage in bad faith decision-making to support prior political commitments. See, e.g., Burbank, Ignorance, supra note 2, at 846 (noting there is a “difference between the inevitable non-neutrality of procedure and the notion that the rulemakers are or might as well be animated by an overtly political agenda”). ↑
48. See Coleman, supra note 23, at 397–400 (2020) (discussing the lack of diversity on the federal rulemaking committees and benefits of diversity); Stephen B. Burbank & Sean Farhang, Federal Court Rulemaking and Litigation Reform: An Institutional Approach, 15 Nev. L.J. 1559, 1567–76 (2015) (analyzing data as to professional composition of rulemaking committees and their legal experience over decades extending back to 1960); Stempel, supra note 6; Thornburg, supra note 47; Zachary D. Clopton, Making State Civil Procedure, 104 Cornell L. Rev. 1, 40–43 (2018) (on the higher levels of diversity in state courts and the greater competence such diversity enables). ↑
49. Reda, Cost-and-Delay Narrative, supra note 21, at 1126–27. ↑
50. This is not to suggest that there are not many fruitful areas for new and additional data collection. Certainly, we can always make use of more information about our civil justice system, and we might even discover that we are collecting the wrong data in order to produce the knowledge that we wish. ↑
51. See supra notes 8–10 and accompanying text. ↑
52. See Moore, supra note 17, at 1154; Carrington, supra note 18, at 606–09. ↑
53. Burbank, Ignorance, supra note 2; Moore, supra note 17. ↑
54. Procedural commentary on the federal rules system and civil justice reform has come to widely recognize the role of the cost-and-delay narrative as rationale for the new era of civil procedure. See Burbank & Farhang, supra note 48, at 1571–76; Arthur R. Miller, Widening the Lens: Refocusing the Litigation Cost-and-Delay Narrative, 40 Cardozo L. Rev. 57, 98 (2018). See generally A. Benjamin Spencer, Pleading and Access to Civil Justice: A Reply to Twiqbal Apologists, 60 UCLA L. Rev. 1710 (2013); Adam Steinman, The End of an Era: Federal Civil Procedure After the 2015 Amendments, 66 Emory L. J. 1 (2016); Subrin & Main, supra note 15; Alexander A. Reinert, The Narrative of Costs, the Cost of Narrative, 40 Cardozo L. Rev. 121 (2018) (analyzing how commentary about disfunction in the courts and attorney misbehavior implicitly relies upon the cost-and-delay narrative); Diego A. Zambrano, Discovery as Regulation, 119 Mich. L. Rev. 71 (2020). ↑
55. Commentators have continued to rely on the cost-and-delay narrative. One prominent jurist has written in detail about what ails current civil practice, beginning yet again from the premise of undue cost and delay, and drawing substantial scholarly engagement. Victor Marrero, Mission to Dismiss: A Dismissal of Rule 12(b)(6) and the Retirement of Twombly/Iqbal, 40 Cardozo L. Rev. 1 (2018); Victor Marrero, The Cost of Rules, the Rule of Costs, 37 Cardozo L. Rev. 1599 (2016). Judge Marrero’s sincere and even-handed account in Mission to Dismiss, relying on his observations from the trenches, focuses on the relationship between lawyer behavior and cost and delay. His work highlights the gap between our expectations for attorney conduct and actual attorney practice, and the role that attorneys’ sharp practices and economic self-interest play in raising costs and exacerbating delays. Judge Marrero articulates commonly shared beliefs about attorney misdeeds. As but one prominent example, Chief Justice Roberts devoted his letter prefacing one year-end report on the federal judiciary to comparing attorney adversarial practice to nineteenth century dueling, in which the nation’s gentlemen needlessly lost their lives, and hoping amendments to the Rules could prevent the nation’s attorneys from a similar fate. John Roberts, 2015 Year-End Report on the Federal Judiciary 11–12 (2015), https://www.supremecourt.gov/publicinfo/year-end/2015year-endreport.pdf [https://perma.cc/LQH5-YZ8E]. Of course, this distinction between expectation of attorney behavior and actual practice has struck me as well; but what seems so peculiar from my vantage point, and as will be discussed further below, is that we would expect anything other than adversarial, interest-based practices in an adversarial system designed to resolve conflicts between parties with conflicting legal interests. The continued commitment to the cost-and-delay narrative resides not only on the bench. Professors Avraham and Hubbard’s recent, creative effort to synthesize a new theory of civil litigation as regulation leans into the cost-and-delay narrative as a foundational strand of civil justice reform. The Avrahaam and Hubbard model maintains at its core an assumption of scarcity, based in congestion and delay, around which procedural design should factor the trade-offs between positive and negative externalities. Thus, we seem, perhaps, to be back where we started in terms of civil justice discourse—with continued assumptions about and reliance upon the cost-and-delay narrative, without any true effort to grapple with its mythic nature and the implications thereof. Avraham & Hubbard, Civil Procedure as the Regulation of Externalities: Toward a New Theory of Civil Litigation, 89 U. Chi. L. Rev. 1 (2022). ↑
56. See Miller, supra note 46, at 743–57. ↑
57. Id. ↑
58. Danya S. Reda, What Does It Mean to Say That Procedure Is Political?, 85 Fordham L. Rev. 2203, 2224 (2017). ↑
59. In that Article, I examined the reasoning deployed in amending the discovery standard to provide for proportionality and placed it in the context of courts’ reasoning in public utilities regulation cases. Reda, What Does It Mean to Say That Procedure Is Political?, supra note 58, passim. ↑
60. Id. at 2215–16. “The value of Professor Hale’s rate regulation analysis is not due to strong similarities between the rate regulation context and the reform of discovery rules; rather, its utility lies in allowing us to recognize the long pedigree of a form of legal analysis that clings to an appearance of neutrality and objectivity, the functions it tends to serve, and the types of critique to which it may remain susceptible.”Id. at 2216. ↑
61. David Marcus explores how this plays out historically for procedural policymaking and the ways that the breakdown of consensus impacts the capabilities of today’s rulemaking committee. David Marcus, The Collapse of the Federal Rule System, 169 U. Pa. L. Rev 2485 (2021). ↑
62. Id. at 2489–92; Miller, supra note 46, at 773–86. ↑
63. Of course, there are good reasons for this reliance, including the tricky question of legitimate authority of a judicial body to promulgate substantive, nay, political, rules. See generally Stephen B. Burbank, Procedure and Power, 46 J. Legal Educ. 513 (1996). ↑
64. “Since nearly all incomes are generated through (private) coercion, there is no posture the law can inhabit that does not involve coercion. The law allows, amplifies, counterbalances, or flips existing coercion so that in every instance the law is choosing among forms of coercion.” Reda, What Does It Mean to Say That Procedure Is Political?, supra note 58, at 2217. ↑
65. Robert L. Hale, Coercion and Distribution in a Supposedly Non-Coercive State, 38 Pol. Sci. Q. 470 (1923); Robert L. Hale, Freedom Through Law: Public Control of Private Governing Power 3–11, 35–37, 189–96, 366–82 (1952). ↑
66. The classic article that proceeds on the premise that legal controversies are bargains structured by law, both by substantive law and the probabilities of success shaped through procedure, is Robert Cooter et al., Bargaining in the Shadow of the Law: A Testable Model of Strategic Behavior, 11 J. Legal Stud. 225 (1982). ↑
67. Danya Shocair Reda, How the Anchoring Effect Might Have Saved the Civil Rule-Makers Time, Money, and Face, 34 Rev. Litig. 751 (2015) [hereinafter Anchoring Effect] (arguing that rulemakers have not done so as much as they should, in part because of a failure to recognize the importance of party interests in understanding how rules will be operationalized and suggesting the utility of the literature on cognitive biases for an examination of how rules will work in action). ↑
68. Andrew Hammond, Pleading Poverty in Federal Court, 128 Yale L.J. 1478, 1484 (2019). ↑
69. See generally id. ↑
70. Brooke Coleman, Endangered Claims, 63 Wm. & Mary L. Rev. 345, 361–88 (2021). ↑
71. Id. at 351–53. ↑
72. Id. at 393–401. ↑
73. “The divide between substance and procedure was the landmark jurisprudential achievement that made the Federal Rules System possible.” Marcus, supra note 61, at 2510. Along similar lines to my argument in What Does It Mean? and this Article, Marcus explains that the substance/procedure divide is no longer tenable in a world that lacks the political cohesion that elites maintained in the U.S. post-war consensus. Id. Faith in the cardinal principle of procedural neutrality has crumbled and cannot be effectively reconstituted. Id. ↑
74. Id. at 2513–17 (arguing that contemporary critiques of procedural reform “have merit, but they remain committed to principles that have eroded, ones that do not engage frontally with the principle of neoliberalism”). Marcus suggests that there is no opportunity for reversal today—with the core principles of procedure, new principles must be developed. According to Marcus, the conservative legal movement has already developed a principle of neoliberalism that it would have guide procedural development in the future. Marcus proposes that progressives develop a principle of equality to guide further procedural policymaking. ↑
75. Id. at 2510–13. ↑
76. Marcus instead presents a preliminary attempt to sketch out a countervailing principle around which to organize procedural policy—a principle of equality. Id. at 2513–17. ↑
77. By examining the specific question of how procedural rulemakers have approached discovery reform, I aim to clarify not only what may now be obvious, that the procedural rulemaking task is an inherently political one, but also to suggest that our fear of this fact and efforts to ignore or suppress it have heightened the bitterness of the struggle and made it difficult to focus on the substance of the dispute—the inequality that procedure either reinforces or alleviates. ↑
78. The analysis also recognizes that procedure is, once again, slow to the controversy over empiricism and so draws on the insights of other legal fields that have grappled with the aspirations to empirical objectivity and the limitations of such aspirations. ↑
79. This type of analysis can also assist in addressing the slow-boiling pro se crisis that the civil courts face nationwide. What type of guardrails are needed when cases will not be adversarially contested? What types of infrastructure, whether procedural (e.g., aggregation mechanisms) or legislative (e.g., publicly funded advocates), should be built to ensure that cases can receive the party-controlled adversary process upon which U.S. civil justice relies? Some scholars have embarked on endeavors that are suggestive of how we might move forward. See generally Steinberg, supra note 28 (demonstrating the limitations of our adversary assumptions when examining cases that individuals face in our state court systems); Hammond, supra note 69. ↑
80. Professor Bone has considered these questions throughout his body of work. See generally Bone, supra note 27; Robert G. Bone, Procedure, Participation, Rights, 90 B.U. L. Rev. 1011 (2010). ↑
81. Bone, supra note 27, at 320. ↑
82. Id. ↑
83. Id. ↑
84. See Garth, supra note 2, at 114. ↑
85. See generally Garth, supra note 2. As an example, Garth explains how, in the aftermath of the RAND study of the Civil Justice Reform Act, one could no longer argue that litigants and lawyers hate alternative dispute resolution nor that it saves time and money. Id. at 121. The existing data does not support such arguments, so proponents or critics must shape their arguments differently to be credible. Cf. Galanter, supra note 6 (lamenting that the accumulation of reliable empirical data has had little effect on the arguments that advocates make and appear to make with some force/credibility/effect). ↑
86. See generally Thomas Kuhn, The Structure of Scientific Revolutions (1962); Richard J. Bernstein, The Restructuring of Social and Political Theory (1976); Theoretical Perspectives in Sociology (Scott G. McNall ed., 1979). ↑
87. Fortunately, although this discussion has not been had explicitly in the context of civil procedure scholarship, legal scholars have considered the question extensively. ↑
88. Heise is an exception to this, defining empirical legal scholarship straightforwardly as “the subset of empirical legal scholarship that uses statistical techniques and analyses. By statistical techniques and analyses I mean studies that employ data (including systematically coded judicial opinions) that facilitate descriptions of or inferences to a larger sample or population as well as replication by other scholars.” Heise, supra note 12, at 810. His choice of definition, however, furthers the point that the concept of empirical is simplified rather than complicated in theoretically useful and perhaps necessary ways. Unlike Heise, Engstrom does not address the concept directly, though we may assume that there is enough of a record of discussion among scholars engaging in or writing about empirical legal studies that it may not seem necessary. Engstrom’s account does implicitly recognize the porous boundaries of empirical studies, where he suggests that one of the costs of the increasing attention to large data sets and sophisticated technical methods is a potential inattention to qualitative analysis and a risk of “naïve empiricism.” Engstrom, supra note 13, at 1238. ↑
89. While the scholarship discussed supra notes 6–14 and infra notes 97–120 recognizes there are important questions to be raised concerning the intersection of empirical study and procedural policy or reform, none of this work addresses a foundational question as to what is meant by “empirical.” This elision is not surprising, given what a thorny question it is, how long it has plagued legal and non-legal studies of society, and how fundamentally it implicates epistemological questions of what constitutes knowledge. ↑
90. See David Trubek, Where the Action Is: Critical Legal Studies and Empiricism, 36 Stan. L. Rev. 575 (1984); Joan C. Williams, Critical Legal Studies: The Death of Transcendence and the Rise of the New Langdells, 62 N.Y.U. L. Rev. 429, 439–40 (1987); Martha L. Fineman & Anne Opie, The Uses of Social Science Data in Legal Policymaking: Custody Determinations at Divorce, 1987 Wis. L. Rev. 107. ↑
91. Trubek, supra note 90, at 602. Trubek contrasts this with scholars who, on the other hand, see systems of ideas, including law, as “presenting as essential, necessary, and objective what is contingent, arbitrary, and subjective.” Id. at 606. ↑
92. Id. at 600. ↑
93. Id. at 603. ↑
94. Id. ↑
95. See James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (1998); Lorraine Daston & Peter Galison, Objectivity (2007); Thurka Sangaramoorthy & Adia Benton, Enumeration, Identity, and Health, 31 Med. Anthro. 287 (2012). ↑
96. This is a claim open to contestation since the introduction of increasingly sophisticated statistical analysis into civil claims has been both a challenge to traditional judicial functions and a fact of life over the last several decades. ↑
97. The turn to quantitative methods is often explained as a straightforward result of the unparalleled advances made in quantitative analysis and technological capacity for data gathering and manipulation. However, the existence of such tools does not explain their rising dominance nor the changing relationship to how we understand knowledge that their use engenders. ↑
98. Citing Posner, Michael Heise suggests that one of the reasons for empirical research is its ability to better separate descriptive from normative claims. “It is at least hoped that empirical scholarship can more easily separate the normative from the descriptive and better maintain [the best empirical legal scholars’] endeavor to approach their research questions objectively and their methodology of empirical choice facilitates as much objectivity as is humanly possible.” Heise, supra note 12, at 814 (citing Richard A. Posner, Overcoming Law 210 (1995)). Heise makes sure to acknowledge that “[i]t is perhaps unavoidable that research questions are posed for a reason and that ‘all measurement is lightly or heavily scented with the values of those whose hands are on the switch.’” Id. (quoting Lawrence Friedman, The Law and Society Movement, 38 Stan. L. Rev. 763 (1986)). In addition to its neutrality, Heise commends the design of research questions that can be tested and are falsifiable. Heise posits that doctrinal and theoretical scholarship often rests, “some precariously so,” on empirical assumptions. Michael Heise, The Past, Present, and Future of Empirical Legal Scholarship: Judicial Decisionmaking and the New Empiricism, 2002 Ill. L. Rev. 819, 827 (2002). “Once these key assumptions are identified and transformed into hypotheses, however, they then become amenable to the rigors of empirical testing.” Id. ↑
99. Garth, supra note 2, at 103. ↑
100. Garth also advocates in this piece for a third approach to social science research on the civil justice system. He argues that scholars and reformers should encourage empirical research that is not driven by the needs and concerns of law, but rather by “the theories and concerns of social scientific disciplines.” Id. at 105. Garth’s advocacy for social scientific research into legal institutions is, he suggests, perhaps a better way to understand relevant questions to the law. Id. Just as a physician understands that medicine should be accessible and effective, yet medicine studies behavior outside of hospitals and doctors’ offices to learn about the incidence of heart disease, so too should legal practitioners and policymakers recognize that studying the social world in which courts exist will provide valuable insights for the court system. This argument has resonance with the argument laid out in Part VI below on incorporating an analysis of strategic decision-making and the balance of power outside of courts into rulemaking efforts. ↑
101. Id. at 113. ↑
102. Id. at 105. ↑
103. Id. ↑
104. Id. at 111. ↑
105. Id. at 105. ↑
106. Many of these challenges, described by Bryant Garth in 1997, remain today, although increasingly sophisticated methods as well as a growing set of skilled scholars utilizing social science techniques has meant great advances in what empirical research can achieve. ↑
107. Garth, supra note 2, at 105. ↑
108. Id. at 113. ↑
109. Gelbach, supra note 13, at 373. ↑
110. Id. at 377. ↑
111. Id. ↑
112. Id. at 377 n.18 (“The direction of Twombly and Iqbal’s quality-filtering effects is hardly alone on the list of important empirical questions that feasible data and methods are unlikely to resolve. See, for example, the debate over the existence of a deterrent effect of the death penalty, as investigated in John J. Donohue & Justin Wolfers, Uses and Abuses of Empirical Evidence in the Death Penalty Debate, 58 Stan. L. Rev. 791, 794 (2005): Our estimates suggest not just ‘reasonable doubt’ about whether there is any deterrent effect of the death penalty, but profound uncertainty. We are confident that the effects are not large, but we remain unsure even of whether they are positive or negative. The difficulty is not just one of statistical significance: whether one measures positive or negative effects of the death penalty is extremely sensitive to very small changes in econometric specifications. Moreover, we are pessimistic that existing data can resolve this uncertainty.”). ↑
113. Engstrom, supra note 13, at 1206. ↑
114. Id. ↑
115. Id. at 1236. ↑
116. Id. ↑
117. Id. at 1238. ↑
118. Id. at 1220. ↑
119. Id. at 1229–30 n.88 (“[B]ottom-line social-welfare judgments are notoriously difficult to make with any precision.”). ↑
120. Brooke Coleman provides a valuable investigation of this tendency underlying procedural policymaking today. Brooke Coleman, The Efficiency Norm, 56 B.C. L. Rev. 1777, passim (2015). ↑
121. Id. ↑
122. This second critique benefits from work already well-developed outside of law that recognizes the ways that particular metrics or methodological approaches constrict our field of vision—that they narrow our ways of seeing. E.g., Scott, supra note 95. ↑
123. Engstrom’s essay implicitly recognizes this when it calls attention to the important role empirical study may play in shaping the very subjects of its research, particularly where those subjects are the perceptions of the bench and bar. Engstrom, supra note 13, at 1214–29. ↑
124. “Many proceduralists today are quick to blame rulemaking failures on a poor empirical understanding of how rules actually operate in practice.” Bone, supra note 13, at 319. The contemporary era begins in the 1970s with a sense of crisis and demands for reform. Two earlier periods are well-known to those who follow procedural developments. The first is the era of the code reform movement, which was followed by the era of the 1938 Rules. ↑
125. Id. at 319–20. ↑
126. Id. at 320–21. ↑
127. Id. at 321. ↑
128. Id. ↑
129. This is the phenomenon of which Resnick’s aptly named and brilliantly described Managerial Judges is a part. Judith Resnik, Managerial Judges, 96 Harv. L. Rev. 374 (1982). There is little rumination, certainly among judges and rulemakers, of other social functions of courts and cases, aside from some lyrical voices in the academy. ↑
130. Indeed, in the midst of much of this criticism, the civil rulemaking committees and judicial control over procedural reform itself faced stiff opposition, and thus the Advisory Committee had every reason to incorporate these suggestions. For an account of the controversy surrounding Supreme Court control of the rulemaking process in the 1980s, see generally Burbank & Farhang, supra note 19. More recent recognition of this has been expressed by rulemakers: “Judge Rosenthal pointed out that there has been increased Congressional scrutiny of the rule making process. The rules committees, she said, have taken pains to make sure that Congress knows what actions the committees are contemplating early in the rules process, especially on proposals that may have political overtones or affect special interest groups.”Standing Comm. Rules of Practice & Procedure, Minutes, January 7–8, 2010, at 4–5 (2010). In discussions about pleading, a lawyer member of the Committee articulated praise for an empirical approach: “[O]ne of the great strengths of the rules process is that the advisory committees rely strongly on empirical evidence . . . before proceeding with potential rule adjustments, the committee should obtain sound empirical data to ascertain whether any real problems have in fact been created by Twombly and Iqbal.”Id. at 20. Another committee member justified the rules committees’ existence in part due to their use and reliance on empirical data: “Procedural rules, she said, are sometimes made by Congress or the Supreme Court. But the rules committees are the appropriate forum to draft rules because the committees demand a solid empirical basis for amendments, seek public comments from all sides, and give all proposals careful and objective deliberation. Therefore, the Advisory Committee on Civil Rules should proceed to gather the empirical information necessary to support any change in the pleading rules.”Id. at 32. ↑
131. Advisory Comm. on Civ. Rules, Minutes, February 2–3, 2009, at 10 (2009) (discussing preparation for the Duke Conference and gathering of empirical data). ↑
132. Ultimately, the Duke Conference’s aim was “to discover where we are, how to make the system better. There will be many points of view. Consensus will be welcome where it emerges and will help to guide future projects. Work will continue on areas of disagreement.” Advisory Comm. on Civ. Rules, Minutes, October 8–9, 2010, at 3 (2010). ↑
133. Advisory Comm. on Civ. Rules, Minutes, October 8–9, 2009, at 3 (2009). ↑
134. Some of this was inevitable, as some of the surveys were conducted before the idea for the Duke Conference had been developed. Indeed, to some extent the American College of Trial Lawyers (ACTL) and Institute for Advancement of the American Legal System (IAALS) survey results served as an impetus for a new investigation into discovery cost and litigation delay. See Advisory Comm. on Civ. Rules, Minutes, November 17–18, 2008, at 18 (2008); Standing Comm. on Rules of Practice & Procedure, Minutes, January 2009, at 32–41. This was a “new” investigation into discovery cost and delay because such inquiries have arisen repeatedly over the last forty years. See Reda, Cost-and-Delay Narrative, supra note 21; Mullenix, supra note 6. Given the Advisory Committee’s interest in the results of the ACTL/IAALS survey, it would have been valuable to arrange some analysis of its results—that is, to use the conference as a venue for evaluating the survey’s findings and assessing what conclusions might be drawn from it. One might have expected the Advisory Committee to engage in precisely the activities it advertised to Congress: “obtaining . . . reliable empirical information” and “gathering and studying the information.” Rosenthal Letter, supra, note 11. As will be discussed below, this strong stewardship of information collection and interpretation was absent from the rulemaking process. The (mostly) judges of the civil rules committees abdicated their responsibility to exercise this control. ↑
135. The “other” category is one that is entirely un-theorized—any type of information that rulemakers do not understand to be “empirical” or obtainable through empirical methods, are thrown into this catchall category. We can describe this grab bag of information on the procedural system as impressionistic observation, and it will be treated in detail in this Part IV. ↑
136. That the Advisory Committee has a hazy conception of the empirical is perhaps to be expected, given our own scholarly imprecision. See supra notes 80–90 and accompanying text. ↑
137. Rosenthal Letter, supra, note 11. ↑
138. Id. ↑
139. The real starting point, of course, is the questions the Advisory Committee sought to answer when it considered further discovery reform. The Advisory Committee’s starting premises have already been subjected to analysis. I have previously criticized the Advisory Committee for its susceptibility to resilient myths about litigation cost and delay, the assumptions of which launched the entire Duke Conference process culminating in the 2015 amendments. See generally Reda, Cost-and-Delay Narrative, supra note 21, at 1085. ↑
140. Advisory Comm. on Civ. Rules, Minutes, November 17–18, 2009, at 71 (2009); Advisory Comm. on Civ. Rules, Minutes, April 7–8, 2008, at 35 (2008). ↑
141. Advisory Comm. on Civ. Rules, Minutes, November 17–18, 2009, at 71 (2009). ↑
142. Professor Marcus had been charged with conducting an exploration of how the 2006 e-discovery amendments were working, “with an eye to determining whether there are problems that need to be fixed.” Advisory Comm. on Civ. Rules, Minutes, April 7–8, 2008, at 35 (2008). In that context, a member noted that the ACTL was funding research on “the cost of discovery” that might provide the Committee with information. Id. ↑
143. Advisory Comm. on Civ. Rules, Minutes, November 17–18, 2008, at 17 (2008). Judge Kravitz described the project thus: “A major focus would be to find out whether discovery really is out of control. Is there anything that can be done to reduce the costs, whether or not the problems might be characterized so dramatically? Do pleading reforms offer a meaningful alternative by limiting access to discovery? Is it possible to develop a simplified procedure for cases that are harmed, not helped, by full-blown discovery? We are told there is a flight from federal courts to state courts—is that true? Why might it be true?”Id. at 17–18. ↑
144. Id. at 18. Thomas Willging appears to have concurred that FJC input would be beneficial, explaining that “the [ACTL] survey tends to draw from elite lawyers. Empirical inquiry by the [FJC] would give quite a different picture of what goes on day by day by covering the full variety of cases and practice.” Id. ↑
145. Id. at 104; Advisory Comm. on Civ. Rules, Minutes, February 2–3, 2009, at 10 (2009) (“The Federal Judicial Center is moving forward on pulling together empirical data. Tom Willging and Emory Lee are designing a new discovery survey. RAND is working on e-discovery. Other researchers also are gathering empirical information.”). ↑
146. Cases were drawn from a broad cross section of federal litigation, but the sample was selected to ensure data on discovery—therefore, cases unlikely to involve discovery were excluded from the study sample. ↑
147. At the time, legislation had been introduced in both houses of Congress to restore pleading standards to their pre-Twombly state. The Standing Committee found both bills to “have serious flaws” and aimed to identify a “proper approach for the rules committees” given that the public discussion in Congress was “intensely political.” Standing Comm. on Rules of Practice & Procedure, Minutes, January 7–8, 2010, at 7 (2010). It would be necessary to avoid “dragging the committees into the political fray,” without abdicating responsibility for considering what changes might be necessary to the rules. Advisory Comm. on Civil Rules, Legislative Report: Civil Pleading Standards 7 (2010). ↑
148. Standing Comm. on Rules of Practice & Procedure, Minutes, January 7–8, 2010, at 7 (2010). ↑
149. Id. at 13. ↑
150. Comm. on Civ. Rules, Civil Rules: May 2010 Civil Litigation Review Conference 13 (2010). Query whether a “major conference” was the most productive way to communicate with users of the federal courts about the state of litigation, or whether, if a large conference was to be had, how the committee determined whom to invite or what a broad and representative range of lawyers and litigants (to say nothing of law professors and judges) was. This selection process is discussed in greater detail in Part V concerning qualitative data. ↑
151. For example, the Advisory Committee’s report to the Standing Committee on preparations for the Duke Conference stated: “[T]he conference will rely heavily on empirical data to provide an accurate picture of what is happening in the federal litigation system.” Id. at 14. ↑
152. For an extensive analysis of the difference between the FJC studies and the remainder of the work presented as “empirical research” by the Advisory Committee, see Reda, Cost-and-Delay Narrative, supra note 21, at 1107–09. ↑
153. Lee III & Willging, supra note 33; Emery G. Lee III & Thomas E. Willging, Fed. Judicial Ctr., Preliminary Report to the Judicial Conference Advisory Comm. on Civil Rules. ↑
154. Advisory Comm. on Civ. Rules, Minutes, March 2, 2010, at 7 (2010). ↑
155. Id. ↑
156. Id. at 2. ↑
157. Id. at 3. ↑
158. Id. ↑
159. Id. ↑
160. Standing Comm. on Rules of Practice & Procedure, Minutes, January 7–8, 2010, at 14 (2010). ↑
161. Id. ↑
162. Id. ↑
163. Id. ↑
164. Advisory Comm. on Civ. Rules, Minutes, October 8–9, 2009, at 3 (2009), at 43. ↑
165. Standing Comm. on Rules of Practice and Procedure, Minutes, June 14–15, 2010, at 26 (2010). ↑
166. Id. ↑
167. Id. ↑
168. Id. Judge Kravitz concludes this discussion by identifying an aim: “The goal in each federal civil case . . . should be to agree on a sensible and proportionate discovery plan that relates to the stakes of the litigation.” Id. ↑
169. Id. ↑
170. Id. Lee provided the following example: “[R]esponses from lawyers in a perception study showed that they believe that about 70% of litigation costs are associated with discovery. The empirical studies, on the other hand, demonstrate that discovery costs were actually much lower, ranging between 20% and 40%.” Id. “By way of further example, a recent perception-study showed that 80% or 90% of lawyers agree that litigation is too expensive. Yet the Federal Judicial Center studies demonstrate empirically that costs in the average federal case were only about $15,000 to $20,000.” Id. ↑
171. Id. ↑
172. Id. at 27. ↑
173. Id. ↑
174. Panelists and conference participants seemed unable to make sense of the FJC findings. Justice Rebecca Kourlis, introducing another study that surveyed in-house counsel at Fortune 200 companies, described the FJC data as a subset of the federal docket different from the Fortune 200’s cases. Similarly, Lorna Schofield, presenting on behalf of the ABA’s Section on Litigation, described the discrepancies between the ABA survey results and the FJC survey results as “two different populations with two different kinds of cases.” Yet the 2009 FJC study was not a “subset” or a “different population.” On the contrary, it reflected a broad cross section of federal cases, and the sample was carefully groomed to ensure that the largest cases would be represented. Author’s Notes from the 2010 Civil Litigation Conference at the Duke University School of Law (May 10–11, 2010) (on file with author). ↑
175. For example, presenting findings of the FJC’s multivariate analysis of the Closed-Case Study data, Lee and Willging led with the finding that cost increases track increases in the dollar stakes of a litigation. They concluded: “It does not seem likely that revisions in the Civil Rules can do anything to affect the stakes involved in litigation.” Advisory Comm. on Civ. Rules, Minutes, March 18–19, 2010, at 4 (2010). Presenting the results of the Attorney Follow-Up Survey, Lee and Willging said attorneys respond that “[t]he quest is not for perfect information, but for enough information in relation to the stakes. This is self-monitoring, not a result of enforcing the discovery rules.” Id. at 6. ↑
176. Standing Comm. on Rules and Practice Procedures, June 14–15, 2010, at 25 (2010). ↑
177. Id. ↑
178. Emery G. Lee III, Motions for Sanctions Based upon Spoliation of Evidence in Civil Cases: Report to the Judicial Conference Advisory Committee on Civil Rules 8–9 (2011). Motions for spoliation sanctions were found in .15 percent of the cases in the sample. Advisory Comm. on Civ. Rules, Minutes, November 15–16, 2010, at 13 (2010). Those in which motions for sanctions were made were described as “very odd cases.” Id. ↑
179. The decision to move forward with a rule amendment addressing this highly unusual problem is also better understood by considering the role that rulemakers’ discomfort with and inconsistent attention to the strategic behavior inherent in adversarial legal process. See infra Part V. ↑
180. Standing Comm. on Rules of Practice and Procedure, Conference Agenda, May 10–11, 2010, at 2 (2010). ↑
181. Id. ↑
182. For a discussion of this literature and its longevity, see Reda, Cost-and-Delay Narrative, supra note 21. ↑
183. Standing Comm. on Rules of Practice and Procedure, Conference Agenda, 2010 Civil Litigation Conference, at 2 (2010). ↑
184. Id. at 3. The five bar associations I characterize as reflecting a defense bar membership are the ACTL, the ABA Litigation Section, the Association of the Bar of the City of New York, the Lawyers for Civil Justice, and the Defense Research Institute. The organization representing the plaintiffs bar is the American Association for Justice. ↑
185. Advisory Comm. on Civ. Rules, Minutes, March 18–19, 2010, at 14 (2010). ↑
186. I wouldn’t argue with it and in fact wish to suggest that it is far more important than currently envisioned. ↑
187. For a discussion of cognitive biases, see supra notes 45–47; Reda, Cost-and-Delay Narrative, supra note 21, at 1119–20; see generally Reda, Anchoring Effect, supra note 67. ↑
188. The reasoning from the Discovery Subcommittee (i.e., a subcommittee of the Advisory Committee predating the Duke Conference and involved in developing the 2015 amendments) discussions of preservation and sanctions issues may be worth considering. There, the Discovery Subcommittee explained the value of a “mini-conference” thus: “The purpose of the meeting was to discuss how to address preservation/sanctions issues during the full Committee’s April meeting and beyond. Judge Campbell introduced the issues by suggesting that it may be that the Subcommittee needs input from outside the Committee. Compare, for example, the Mini-Conference with lawyers familiar with New Jersey practice in relation to the discoverability of lawyer/expert interactions. That event provided not only very valuable insights, but also the reassurance that ‘real world’ considerations underlie rule-change proposals . . . becoming fully informed makes sense. For that purpose, holding a conference would probably be essential.”Advisory Comm. on Civ. Rules, Notes of Discovery Subcommittee Meeting, February 20, 2011, at 220 (2011). ↑
189. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 309 (2012). ↑
190. Id. ↑
191. Id. ↑
192. Id. ↑
193. Id. ↑
194. Fed. R. Civ. P. 1. ↑
195. The Rules Enabling Act prohibits any expansion or contraction of a substantive right. 28 U.S.C. § 2072(b) (“Such rules shall not abridge, expand, or contract any substantive right.”) Maybe the difficulty suffered is one of original sin. How does the Committee know what is “just” disposition? Speedy and inexpensive seem easier to understand. ↑
196. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 309 (2012). ↑
197. It is possible here that the Advisory Committee is sensitive to complaints that it “tinkers.” See, e.g., Marcus, supra note 61, at 2497–508. ↑
198. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 309 (2012). ↑
199. Id. ↑
200. This Article suggests that such information is not simply one useful type of information to be gained from practitioners but likely the most significant. ↑
201. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 309 (2012). ↑
202. Id. at 317. ↑
203. For example, Jocelyn Larkin, a civil rights litigator and provider of funding for impact litigation, drew on experience of “underfunded litigation” to provide her impressions of changes suggested through the Rules sketches. However, comments never moved past a general gloss on what types of changes she was for or against. Id. at 315. Similarly, Michele Larkin, whose practice involves mainly large commercial litigation, stated that based on that experience, lawyers will ask for “everything and anything,” and made general references to “abusive discovery” and the process being “out of control.” This context was referenced to support her suggestion that requiring proportionality and presumptive limits on discovery devices would be “a start.” Id. at 311. ↑
204. Id. at 315. ↑
205. Id. ↑
206. Id. The missing details of this anecdote are clear. We are told the cost of e-discovery, but neither the amount at stake in the controversy nor the size of the client, nor the length of the litigation, nor how these storage costs compare with, or possibly even overlap with, other information access systems the company already employs. ↑
207. Id. at 327. ↑
208. Id. ↑
209. This framing, requesting that participants recall the last time they, or their opposing counsel, made reference to proportionality, is important because it ties analysis to a particular case, helping to minimize cognitive biases such as availability or confirmation bias. Rulemakers might also use additional questioning about the sociology of the case, for example asymmetry in party resources or access to relevant discovery, to determine how litigation factors affect whether or not parties invoke proportionality. ↑
210. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 327 (2012). ↑
211. This is consistent with longstanding characteristics of discovery reform and discussions of procedural reform more broadly. For a discussion of the repeated reliance on the “fact” of excessive cost and delay in U.S. civil discovery, despite consistent evidence to the contrary is, see generally Reda, Cost-and-Delay Narrative, supra note 21. ↑
212. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 327 (2012). ↑
213. Id. ↑
214. Id. ↑
215. Id. at 328–29, 331. ↑
216. It is possible invitees reflect instead those that the Advisory Committee understands to be significant enough to the rulemaking question as to require consultation as a matter of conducting a legitimate process. ↑
217. Notice how this question is presented. The Mini-Conference participant is asked whether “proportionality” is understood by others, that is, litigants generally, or judges. What is a participant to draw on in offering their conclusion? Might it have made sense to begin by asking the participant whether they think there is such a thing as proportionality in civil discovery, and if so, what they understands it to mean. Do you have reason to think others share your understanding? To the extent you do, please think about why. To the extent you believe otherwise, please think about why. This alternative framing also resolves the ambiguity of the words “have meaning,” because it begins by soliciting definitions, which might provide some feedback as to whether participants share the Committee’s view of this malleable concept. There is ambiguity in the question, “does the concept . . . have meaning to civil litigants,” because it is not clear whether what is being asked is: Is there any shared meaning to this concept among actors in the civil system, or rather does it have utility (in the sense that something “has meaning” when it is impactful), or is it a concept that litigants care about? ↑
218. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 400 (2012). ↑
219. Id. ↑
220. Availability bias describes the cognitive shortcut that leads to a demonstrated human tendency to recall unusual or unrepresentative incidents or events more than representative ones. Confirmation bias refers to the mental shortcut that results in a tendency to recall and believe that which confirms an individual’s preexisting understandings. ↑
221. Standing Comm. Rules of Practice & Procedure, Minutes, June 14–15, 2010, at 27 (2010). The minutes describe how FJC study author Emery Lee explains the FJC study design and why its cost and discovery estimates are significantly lower than the general opinion surveys collected for the Duke Conference. Id. ↑
222. Advisory Comm. on Civ. Rules, Duke Subcommittee Rules Sketches, November 1–2, 2012, at 400 (2012). ↑
223. For a discussion of the limitations on the substantial differences between discovery availability in civil and criminal contexts, and the problems they engender for the quality of criminal justice, see, for example, Russell M. Gold et al., Civilizing Criminal Settlements, 97 B.U. L. Rev. 1607 (2017); Issachar Rosen-Zvi & Talia Fisher, Overcoming Procedural Boundaries, 94 Va. L. Rev. 79 (2008); Jenny Roberts, Too Little, Too Late: Ineffective Assistance of Counsel, the Duty to Investigate, and Pretrial Discovery in Criminal Cases, 31 Fordham Urb. L.J. 1097 (2004). Attention to the divide, and to the disadvantages it places upon criminal defendants, has a long pedigree. See Robert L. Fletcher, Pretrial Discovery in State Criminal Cases, 12 Stan. L. Rev. 293, 316 (1960); Abraham S. Goldstein, The State and the Accused: Balance of Advantage in Criminal Procedure, 69 Yale L.J. 1149, 1192 (1960). ↑
224. Though discovery constraints have been a longstanding target for reform, the last decade alone, spurred on by advances in DNA technology and the extensive exoneration work it has fostered, has seen a wealth of reform efforts designed to expand discovery and reduce prosecutorial information advantages. For a recent articulation of expanded discovery demands, see Rachel Barkow & Mark Oster, 14 Steps Biden’s DOJ Can Take Now to Reform America’s Criminal Legal System, The Appeal (Mar. 15, 2021), https://theappeal.org/the-lab/white-paper/14-steps-bidens-doj-can-take-now-to-reform-americas-criminal-legal-system [https://perma.cc/N6CV-F9SW] (calling for, among other things, the Justice Department to follow the lead of states like North Carolina and Texas by adopting open file discovery to create more accurate prosecutions that are less likely to run afoul of prosecutors’ constitutional obligations under Brady v. Maryland, 373 U.S. 83 (1963)). ↑
225. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 400 (2012). ↑
226. Id. Note that we are now using “some’s” characterization of discovery as “free,” unattributed, and descriptive rather than normative. The questions that follow question the proposal (not the original belief of “some,” but the proposal that the Mini-Conference appears to be putting forward), suggesting that courts already have the power to do so and asking whether or not courts already use this power: “Don’t courts currently have the power to allocate costs if the production sought is unwarranted? Do they not use whatever power they have?” Id. But even these are bracketed by additional requests for suggestions of how to address the problem that “some” have proposed exists. “Are there any additional ways that the Rules should address the issue of cost-sharing or the reduction of costs?” Id. The need for and utility of cost sharing is presumed; the prompt does not seek to examine or question that presumption, yet the rulemakers never own this presumption or even explicitly acknowledge it. ↑
227. In fact, the sentence construction is identical. Participants are asked: “Does the concept of ‘cooperation’ in civil discovery have meaning to civil litigants and judges? Do you find it is regularly considered in the cases that you handle? Would adding the concept to Rule 1 increase the efficiency and effectiveness of civil litigation?” Id. at 401. ↑
228. For a detailed explanation as to why this is the case, see Reda, Cost-and-Delay Narrative, supra note 21, at 1117–23. ↑
229. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 401 (2012). ↑
230. Id. ↑
231. The Advisory Committee uses the term sketches to indicate draft concepts for amending the Rules. These sketches provide an early overview of the form proposed amendments may take, in order to solicit feedback prior to developing formal proposed amendments. ↑
232. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 353 (2012). ↑
233. Id. ↑
234. Notice that the proposals (which were ultimately withdrawn) to alter presumptive limits on discovery were understood to provide “more specific means of encouraging proportionality.” Id. at 354. Thus, proportionality was clearly identified with a reduction in discovery rather than its opposite and the problem was never articulated as such but was clearly understood to be over-discovery rather than under-discovery or discovery obstruction. The following sentence refers to proposed amendments that appear aimed at the problem of obstruction, but this is described as a different way to achieve the same thing, it seems: “Another approach is a set of provisions to improve the quality of discovery objections and the clarity of responses.” Id. ↑
235. Id. ↑
236. Id. ↑
237. Public Hearing on Proposed Amendments to the Federal Rules of Civil Procedure Before the Judicial Conf. Advisory Comm. on Civ. Rules, 113th Cong. 56–57 (2013); Public Hearing on Proposed Amendments to the Federal Rules of Civil Procedure Before the Judicial Conf. Advisory Comm. on Civ. Rules, 113th Cong. 143 (2014). ↑
238. Fed. R. Civ. P. 26 advisory committee’s note to 2015 amendment. ↑
239. Id. Note also that this characterization is peculiar, given the language from the 1983 note quoted just a few lines later: “The objective is to guard against redundant or disproportionate discovery by giving the court authority to reduce the amount of discovery that may be directed to matters that are otherwise proper subjects of inquiry.” (emphasis added). The Committee note then suggests that the current understanding, that proportionality was simply a back-end safety valve, a limitation on otherwise-allowable discovery, is the inadvertent result of the 1993 amendments. “Subdividing the paragraphs, however, was done in a way that could be read to separate the proportionality provisions as “limitations,” no longer an integral part of the (b)(1) scope provisions.” Id. ↑
240. Id. ↑
241. Id. The extent to which these assertions were unfounded in 1993 has been well-documented in Reda, Cost-and-Delay Narrative, supra note 21, at 1111–16. See also Mullenix, Discovery in Disarray, supra note 6, at 393–405. The Committee in 2015 should have been well aware of the literature documenting a lack of empirical support for claims of excessive cost and increasing delay. ↑
242. Fed. R. Civ. P. 26 advisory committee’s note to the 2015 amendment. ↑
243. Id. ↑
244. Id. (“It also is important to repeat the caution that the monetary stakes are only one factor, to be balanced against other factors.”). ↑
245. Notice the formal equivalency here. There is a danger that an impecunious party might be shielded from discovery altogether, rendering the system impotent to hold them accountable. On the other end of the spectrum, there is a danger that a wealthy party may be harassed with unending requests. The lurking resource battle at the heart of discovery controversy bubbles to the surface here. The Note again leans on the 1983 Advisory Committee for support and legitimacy on this question: “The 1983 Committee Note cautioned that ‘[t]he court must apply the standards in an even-handed manner that will prevent use of discovery to wage a war of attrition or as a device to coerce a party, whether financially weak or affluent.’” Id. ↑
246. The portion of strategic behavior problematized in Committee analysis follows predictable patterns. ↑
247. E.g., Bone, supra note 13, at 334–37. ↑
248. Even before the shift to a plausibility pleading standard, or the past decade’s Supreme Court jurisprudence vastly reducing the possibility of a class action forum, the FJC’s own studies indicated that over 46 percent of plaintiffs’ attorneys did not agree that the outcomes of cases in the federal system are generally fair. See Lee III & Willging, supra note 33, at 68–69. Moreover, a 2005 study of attorneys’ choice of forum noted attorneys’ beliefs that “state courts favor plaintiffs and federal courts favor defendants.” See Thomas Willging & Shannon R. Wheatman, An Empirical Examination of an Attorney’s Choice of Forum in Class Action Litigation 3 (2005), https://www.uscourts.gov/sites/default/files/clact05.pdf [https://perma.cc/F2VG-RBLF]. See generally Patricia W. Moore, Comment on Proposed Amendments to the Federal Rules of Civil Procedure (Jan. 31, 2014), https://dx.doi.org/10.2139/ssrn.2388413 [https://perma.cc/34KR-6BYM]. ↑
249. See Burbank & Farhang, supra note 48; Deborah R. Hensler & Jonah B. Gelbach, What We Don’t Know About Class Actions but Hope to Know Soon, 87 Fordham L. Rev. 65 (2018); Brooke D. Coleman, Summary Judgment: What We Think We Know Versus What We Ought to Know, 43 Loy. U. Chi. L.J. 705 (2012). ↑
250. It is not just a recognition that there are limits to the data we can obtain. For example, it is impossible to measure which cases aren’t being filed or how much more such filings are costing. It is also the fact that many of the questions the rulemakers are tasked with evaluating may be usefully informed by empirical data, but they are not susceptible to empirically garnered answers. For instance, is there too much discovery? Or better still, what goals does discovery serve and which of those do we wish to prioritize? ↑
251. Advisory Comm. on Civ. Rules, Minutes, November 15–16, 2010, at 25 (2010). ↑
252. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 309 (2012). ↑
253. Advisory Comm. on Civ. Rules, Minutes, November 15–16, 2010, at 26 (2010). ↑
254. Advisory Comm. on Civ. Rules, Duke Conference Subcommittee Miniconference Notes, October 8, 2012, at 368 (2012). ↑
255. Advisory Comm. on Civ. Rules, Minutes, November 15–16, 2010, at 28 (2010). ↑
256. Id. ↑
257. Committee members pointed to differences in results between surveys conducted by the ACTL, IAALS, and the ABA, which reported broad agreement among attorneys that costs were “out of control” and “not proportional,” and the results of the FJC Closed-Case Study, which found attorneys reporting discovery costs as “proportional” and “appropriate” in the majority of cases. See, e.g., id. at 27. ↑
258. Id. at 26. ↑
259. Id. at 27. Although the FJC Study indicates that a large number of attorneys say cost is actually appropriate, “[a]t the same time, [the attorneys] suggested that overall the system is too expensive, that litigants are being priced out of federal court.” Id. at 28. ↑
260. Id. ↑
261. Id. ↑
262. Id. at 29. ↑
263. Id. at 27. ↑
264. Id. at 29. ↑
265. “[I]t was suggested that encouraging more basic research on what is really happening may be an important response to the Conference materials.” Id. at 28. ↑
266. Id. at 29. ↑
267. Id. ↑
268. Advisory Comm. on Civ. Rules, Minutes, November 7–8, 2011, at 8 (2011). ↑
269. Elsewhere I explain how the multi-factor proportionality standard emerges as a product of rulemakers’ need for and commitment to neutral intervention in the procedural rules. Reda, What Does It Mean to Say That Procedure Is Political?, supra note 58, at 2223. ↑
270. Advisory Comm. on Civ. Rules, Duke Subcommittee Conference Call, September 10, 2010, at 3 (2010). ↑
271. Advisory Comm. on Civ. Rules, Conference Subcommittee Meeting, March 4, 2011, at 3 (2011). ↑
272. Id. at 3–4. ↑
273. This proposal to incorporate findings from cognitive psychology is laid out in the context of the proposed amendments to presumptive limits that were originally part of the 2015 amendments package but were subsequently withdrawn. Reda, Anchoring Effect, supra note 67. Potential applications are, of course, not limited to this context and are potentially quite broad. Cognitive bias is also a helpful evaluative technique when considering empirical data. For a discussion of misinterpretation of empirical research, see supra text accompanying notes 41–45. Here, experts at the FJC can be helpful in facilitating members’ assessment of the information but being attuned to the state of knowledge about cognitive heuristics will allow members to take that on board more effectively. ↑
274. These may differ depending on the procedural issues under consideration. ↑
275. Many of these are supported by the data that the FJC has collected, including that the size of a firm raises cost, nonmonetary issues raise costs, and fee structure affects cost. ↑
276. “Issues of cost and delay will be addressed with the purpose of seeing how we can do better. The panels are well balanced, with lawyers who regularly represent plaintiffs, those who regularly represent defendants, and those who dwell in the academy.” Advisory Comm. on Civ. Rules, Minutes, March 18–19, 2010, at 2 (2010). ↑
277. For a discussion of these cognitive heuristics in the context of the Rules and civil rulemaking, see Reda, Cost-and-Delay Narrative, supra note 21. ↑