WHY FDA WITHHOLDS DRUG APPROVAL
Evidence from 443 Publicly Released Complete Response Letters
Alex Tabarrok

COMPLETE DATA AND CODE -- v20260807_r20261103
Local refresh: November 18, 2026 (stable archive filename).
Letter-date vintage: August 7, 2026. Source acquisition: August 28, 2026.

This self-contained archive supplies all study source data, code, frozen model
outputs, existing validation materials, and final analysis datasets. It includes
the revised paper and appendix. No API key, credentials, correspondence, referee
response, editing history, or working notes are included.

START HERE
1. Extract into a new folder, preserving paths.
2. Use Python 3.12 and install requirements-replication.txt.
3. Verify all supplied files:
     python code/reproduce_revision.py --verify-only
4. Reconstruct the data and revised analyses from frozen inputs:
     python code/reproduce_revision.py --skip-pdf
   Outputs go into a new build/revision folder. The program verifies that all 10
   original final-data CSVs and all revised tables, application summaries, and generated TeX tables match the supplied files.
   Rebuilt text is compared after normalizing CRLF/LF line endings; no other
   content differences are ignored. Archive and frozen-input hashes remain byte-exact.
   It never overwrites supplied data. For another run, choose a new folder with
   --build-dir build/second-run.
5. To compile the paper too, install a TeX distribution containing latexmk,
   pdfLaTeX, and BibTeX, and omit --skip-pdf. The paper with the author's name is
   Paper/revision/latex/main-with-author.pdf. Compilation also creates main.pdf,
   an anonymous version of the same paper.

After software installation, reproduction makes no network or paid model calls.
Original model responses are replayed as immutable inputs. Scripts retain their
original numbered stage names so dependencies remain explicit.

FAST REANALYSIS OF THE SUPPLIED PANEL
  python code/15_revision_analysis.py
  python code/19_application_summary.py
  python code/16_revision_validation.py
  python code/22_review_facility_recurrence.py
  python code/check_corrected_sequences.py
These regenerate the revised analyses and verify the existing human workbook.
Full reconstruction also rebuilds the original panel, linkages, segmentation
sensitivity, and classification-derived variables.

INDEPENDENT RECODING
data/raw/crl_20260828.json contains all 458 source records and released letter text.
data/interim/crl_letters_parsed_v20260807.jsonl preserves parsed text and identifiers.
deficiency_segments_v20260807.jsonl contains the original classifier inputs;
deficiency_segments_patched_v20260807.jsonl contains corrected extraction.
notes/phase3_codebook.md and code/04_classify_deficiencies.py define the codebook,
prompt assembly, and output schema. Facility-subcategory instructions and code
are included. Stored responses retain dates, model aliases, reasoning effort,
prompt/segment hashes, rationales, and response IDs.

Primary classifications are unchanged. The corrected analysis panel includes three secondary facility codes recovered by
the segmentation correction, excludes one duplicate and one incomplete-response
acknowledgement, and separates the two BLA 761303 original-submission sequences.
The frozen original panels retain their historical counts; use output/revision/
for the current manuscript results.
Researchers can apply alternative classifiers to the complete text and compare
results by letter_id. To rerun the original API classifier, work in a separate
copy with a new output version: the script resumes stored results rather than
overwriting them. Fresh calls require provider access and may yield different
outputs; they are not an exact historical replay.

DATA AND VALIDATION GUIDE
- data/raw/: frozen FDA downloads and Drugs@FDA linkage sources.
- data/final/: original analysis panels and panel_codebook.csv.
- data/interim/: model handoffs, parsed and corrected text, and application links.
- data/manual/: existing review inputs and case-level adjudications, with provenance.
- output/human-validation/: completed human primary-coding workbook.
- output/pdf/: the blinded packet supplied to that reviewer.
- output/revision/tables/: final analyses and human-label mapping.
- output/revision/application-analysis/: first observed letters, category shares,
  and classification-status sensitivities for 333 applications (128 recent).
- output/revision/figures/: final figures.
- replication/REVISION_DATA_DICTIONARY.txt: revised-table fields and denominators.
- replication/frozen_inputs.json: source and handoff integrity hashes.
- RELEASE_MANIFEST.json and SHA256SUMS.txt: all release-file integrity hashes.

The 50-letter classification-validation sample contains only strict CRLs.
The separate 30-letter extraction review includes four predecessor actions
excluded from the 443-letter study sample. Workbook letter numbers map to the
original validation-sample row order. Reviewer identity and dates are not recorded
in the workbook and are not inferred. Case files retain not_recorded values where
historical review provenance was unavailable.

Letters are reproduced as released by FDA, including redactions and OCR defects.
No confidential unredacted originals are included. Scanned PDF archives were not
study inputs; all released text actually analyzed is included. The complete study
corpus is not a census of all CRLs issued by FDA.

DOWNLOAD
https://alextabarrok.com/research/fda-crl/complete_replication_v20260807_r20261103.zip

COMPARISON REGRESSION TESTS
  python code/test_reproduction.py
Tests accept CRLF/LF differences but reject changed values, other whitespace
changes, missing outputs, and any tampering with supplied release files.

NOVEMBER 18 UPDATE
443 letters; 333 applications; 109 linked pairs across 81 applications.
Two-year facility versus other: 24/84 versus 24/166. NDA/BLA stratification
and all-category first-letter counts are included. Linkage evidence is in
output/revision/sequence-check/; the LLM-assisted review of 28 facility
recurrences is in output/revision/facility-recurrence-check/. Neither is an
additional independent human-validation exercise. See RELEASE_NOTES.txt.
