The Engineering behind Biological Alpha
At the current pace of drug discovery, it will take more than 500 years to develop treatments for every known disease. We have written previously about why we believe this trajectory is not inevitable and why improving the way we discover drugs is one of the highest-leverage problems in medicine.1
Pheiron is hiring on the technical team. If this problem space resonates and you’re eager to push the frontier, send us a note.
Roughly nine out of ten drug programs entering clinical development fail. For programs pursuing novel target biology, the probability of eventually becoming an approved medicine can be as low as two percent.
Drug discovery is fundamentally a problem of decision-making under uncertainty. Every experiment, method, and source of evidence exists for the same reason: to reduce uncertainty about whether a biological hypothesis will translate to patients.
Meanwhile, biology has become extraordinarily data-rich. Genetic sequencing, transcriptomics, multi-omic profiling, imaging methods, single-cell perturbation screens, and foundation models generate biological data at an unprecedented scale. Estimates place genomics data alone in the range of several to tens of exabytes.2
But data does not equal information. We discussed this distinction in our benchmark on engineering biological alpha.3
Transitioning from data to insights remains one of the central bottlenecks in modern drug discovery: Scientists must identify relevant data across fragmented resources, harmonize incompatible data structures, integrate fundamentally different experimental modalities, execute analyses across disparate compute environments, and interpret results that are often incomplete or conflicting. Often, this work happens before a scientific question can even be formalized. These obstacles are difficult enough for scientists. For agents, they become a fundamental limitation. Today’s scientific agents operate reliably only in constrained environments with access to a small number of local files and tools. Scaling them to reason across millions of files and petabytes of biological data requires infrastructure built for that purpose.
Engineering biological alpha therefore begins with engineering how information is constructed. The quality of that engineering determines the quality of every downstream decision, from target selection to capital allocation, and ultimately which biology reaches patients.
We formalize the problem, by extending on the concept of data friction4 and define Information Impedance.
Information Impedance is a product of
Disjoint Data: Relevant information is fragmented across hundreds of data assets with different file types, usually stratified by modality (Nd). Each data asset has specific schemas, nuances, and biases (Hi) that must be accounted for.
Disjoint Software: Each “data system” S∈(Ns) requires a bespoke data engine (Ej(s)) specifying access and edits and to work with and dedicated analysis methods (Aj(s)); and
Disjoint Compute: Software and data have to work across different compute providers (Nc). It must also work in disparate compute environments and trusted research environments (TREs) with limited tools (Rk).
Every exciting new dataset, method, or compute environment multiplies impedance (Ω). And crucially, information impedance is harder to overcome for Agents.
Reducing information impedance is critical to unlock scalable evidence generation through agents. Engineering at Pheiron, aims to consistently reduce Information Impedance, to allow scientists and agents to be their most effective.
Building Infrastructure to Reduce Information Impedance
Reducing Information Impedance is fundamentally an infrastructure problem. Pheiron’s infrastructure systematically reduces information impedance by solving for each of its three factors, Disjoint Data, Disjoint Software, Disjoint Compute.
We do this by executing on three core bets:
Bet 1 Data Objects: Phi-Table, Phi-Arr and Phi-Group
Standardized and modular data objects to homogenize the way data is accessed, stored, and maintained.Bet 2 Data Transforms: The Phipe Workflow Engine
Versatile data transform with a standardized interface to simplify analysis under provenance.
Bet 3 Data Identification: Phi-Knowledgebase
An infrastructure and MCP built for agentic knowledge management at exabyte scale.
We implement these bets through sets of tools, which we refer to as primitives. Together, our infrastructure primitives form the bedrock for AI-native science.
Through systematically reducing the burden of having to use hundreds of different access patterns and tools, data engineers can rapidly and systematically onboard new datasets at scale and scientists immediately have the schema, read tools, and context to engineer information from novel and diverse data.
Most importantly though, Pheiron’s infrastructure unlocks Agentic Science at scale, by equipping agents with tools to identify, access and retrieve the right data under provenance at exabyte scale.
Bet 1: Data Objects to homogenize access, storage and maintenance
In modern computational biology, data is deposited across hundreds of unique file formats. Accessing data across each file format under the format-specific access patterns, resembles reading a book, with every chapter written in a new language. This creates an exceptional burden for cross sectional analysis.
Positive examples exist: an HDF file storing expression data allows simple data access and with clear semantics. AnnData5 demonstrates this by granting straightforward numpy-like access to data with explicitly labelled components like cell or gene annotations. XArray6 takes clear semantics a step further by allowing the scientists to author and label components and dimensions the way they like to.
Drawing from these positive examples and honoring the complexity of the modern data landscape and aiming for flexibility, we built a uniquely universal approach to structuring data, following 4 key principles:
All Data is an Array (or Table)
Arrays are a universal data model7. Though, in practice, a lot of data is better modelled as a table8. The use of these abstractions reduces the ‘heterogeneity’ of data (Hi) into two ‘types’.Schemas in Code
Writing schemas in code makes data transparent (inspired by ORMs9), reducing the effect of Nd.Universal Data System
Decoupling schema, in-memory data, the data engine, and storage format provides a paved path to defining anything through PhiTable, PhiArr, and PhiGroup (minimizing Hi).Groups
Hierarchy is key to structured information10. Groups provide structure by composing data into datasets (see AnnData11 semantics and SQL Schemas12).
With these four principles guiding the design of Pheiron’s data primitives, the only infrastructure needed is a filesystem, reducing the impact of Rk (see Figure 3).
This makes integrating then interrogating complex, large-scale datasets trivial. A scientist has a paved path (and an onboard-asset skill) to:
Define the schema for the data.
Select a data engine (specific backend) for the given format.
Multi-modal datasets can immediately be shared, integrated, and analyzed at scale through to_polars_lf or read. This framework is intuitive to scientists and friendly to agents while remaining heavily optimizable by engineers.
Bioinformatics is full of one-off engineering. Every new dataset comes with new code, new assumptions, and new ways of doing the same thing. The result is duplicated work for scientists and an impossible environment for agents.
Our primitives standardize that engineering. Scientists spend less time rebuilding infrastructure. Agents inherit reusable workflows instead of generating bespoke ones.
As an example, Pheiron’s Chief Scientific Officer (CSO) was able to onboard 80 complex datasets with only a few hours of effort and a fleet of agents.
Bet 2: Data Transforms to simplify analysis
In-silico science requires transforming data15. If transform data presents friction, analysis can’t be scaled. In modern bioinformatics workflow engines operationalize analysis pipelines by handling data transforms in a scalable and centralized way.
While engines like nextflow16, snakemake17, dagster18, viash19, and redun20 present great choices for the naive human user, they do not natively interface with Pheiron’s core bets and primitives, and more critically, are not designed for agents.
We built on the most valuable features from these projects to build Phipe: Pheiron’s proprietary workflow engine, designed for both agents and scientists.
Phipe makes executing transforms simple and scalable across compute providers (solving for Nc), while modularizing workflows into containerized21 steps (Aj), and recording executed workflows for data provenance22. Steps that have already been run on a given set of inputs skip (memorization saves compute) and each step of the pipeline can be audited (data provenance).
Importantly, Phipe adds two novel key features beyond standard workflow engines:
Phipe is fully integrated with Pheiron’s data primitives. Phi-Table, Phi-Arr and Phi-Group, are explicitly wired through task inputs and outputs.
Phipe is built on a filesystem-backed database, minimizing infrastructure overhead and allowing Phipe to be used in controlled compute environments (reducing Rk).
Given the data primitives Phi-Table, Phi-Arr and Phi-Group and the workflow engine Phipe, writing data pipelines becomes a standardized formula, agents can follow reliably:
Using this framework, agents are wrangled into building reliable and scalable pipelines.
One example of this can be seen in our recent blog23, where we quantified the predictive validity of data assets, evidence sources and methods. This involved building scalable and reliable data pipelines to sample across over 2 petabytes of data and calculate the eval per evidence type.
Bet 3: Data Identification for agents
Identifying and retrieving specific data points across 4+ million files and 2 petabytes of data is akin to trying to find a needle in a haystack. The problem is exacerbated by the bad tooling typically encountered in computational sciences.
Across many organizations, standard data identification workflow may look like this:
Query SQL/NoSQL databases, list S3 buckets, scan a random Google Drive, and scour the internet (Nd) to find a correct-looking dataset. Message several coworkers and wait for confirmation.
Inspect data schemas and read to sanity-check dataset contents (Hi).
Try (and often fail) to contextualize using similar datasets for study power or orthogonal datasets for additional dimensions to the data.
Try (and often fail) to find provenance to evaluate whether it impacts downstream biological alpha.
While human scientists may manage to work through these obstacles, agents will almost certainly fail24. At least, tokenspend will be high and context will be effectively used.
Agentic science requires infrastructure optimized for effortless data identification and retrieval.
Pheiron reduces the data identification challenge to one step: search the knowledgebase (with search, neighbours, inspect...). Building on the data primitives and the workflow engine, Pheiron models data as a graph: ‘Nodes’ are a given dataset (table, array, or group), ‘edges’ are relationships (group membership, transforms, shared index...). Where a ‘normal’ data catalog25 dryly enumerates datasets, Pheiron’s knowledge graph organizes data semantically: explaining what things mean, how they relate, and what you can ask26.
We add a model context protocol (MCP)27 for searching the knowledgebase and analyzing data through the primitives. Rather than an agent having to reach for 100 MCPs for 100 different data sources, at Pheiron one MCP serves 100 different sources.
This allows agents to effectively identify and retrieve highly specific data points from vast amounts of stored data.
The Information must flow
Information does not emerge from biological data automatically. It has to be engineered.
Throughout this post, we argued that the difficulty of constructing information can be understood as Information Impedance. Information is fragmented across data assets, software systems, and compute environments. Every new dataset, method, or compute environment increases that fragmentation and makes evidence harder to construct.
At Pheiron, we reduce Information Impedance through three engineering bets. Data Objects reduce fragmentation across biological data. Phipe standardizes how analyses are executed across compute environments. The Knowledgebase makes biological information searchable, retrievable, and reusable for scientists and agents. Together, these form the infrastructure required to systematically construct information from biological data.
Every reduction in Information Impedance expands what becomes possible. Scientists spend less time rebuilding infrastructure and more time understanding biology. Agents move beyond isolated workflows and become capable of constructing evidence across millions of files and petabytes of biological data. The ability to systematically construct that evidence is what ultimately allows us to engineer biological alpha and underwrite biology risk.
Acknowledgements
The design of our infrastructure leveraged the collective experience of Pheiron’s staff28. We also drew from an expansive set of prior art (software, data, and publications). We want to acknowledge and thank all of the organisations and individuals referenced for sharing their ideas, work, and experience.
We wrote about our conviction and why we do this before: https://www.pheiron.com/500-years
Stephens, Z. D. et al. Big Data: Astronomical or Genomical? PLoS Biol. 13, e1002195 (2015).
We discussed this distinction in our benchmark on Engineering Biological Alpha:
Warfield, A. S3 Files and the changing face of S3. All Things Distributed (2026). https://www.allthingsdistributed.com/2026/04/s3-files-and-the-changing-face-of-s3.html
AnnData documentation. https://anndata.readthedocs.io/en/latest/index.html
Xarray documentation. https://docs.xarray.dev/en/stable/
Papadopoulos, S. Why Arrays as a Universal Data Model. TileDB Blog (2022). https://www.tiledb.com/blog/why-arrays-as-a-universal-data-model
Wickham, H. Tidy Data. J. Stat. Softw. 59, 1–23 (2014).
SQLModel documentation. https://sqlmodel.tiangolo.com/#a-sql-table
Simon, H. A. The Architecture of Complexity. Ch. 8 in The Sciences of the Artificial (MIT Press, 1996). https://public.econ.duke.edu/~kdh9/Source%20Materials/Zurich%20Papers/Simon%20Ch.%208.pdf
AnnData documentation. https://anndata.readthedocs.io/en/latest/tutorials/notebooks/getting-started.html#adding-aligned-metadata
CockroachDB. Database Schema: A Beginner's Guide. https://www.cockroachlabs.com/blog/database-schema-beginners-guide/
MDN Web Docs. URI. https://developer.mozilla.org/en-US/docs/Web/URI
AWS. What is an API? https://aws.amazon.com/what-is/api/
In fact, one could argue that “doing science” is nothing but applying a sequence of data transformations.
Nextflow. https://www.nextflow.io/
Snakemake documentation. https://snakemake-api.readthedocs.io/en/stable/
Dagster. https://dagster.io/
Viash. https://viash.io/
Redun. https://insitro.github.io/redun/
Docker. What is a Container? https://www.docker.com/resources/what-container/
Redun documentation. Design: Provenance and Call Graphs. https://insitro.github.io/redun/design.html#provenance-and-call-graphs
Our benchmark on Engineering Biological Alpha:
Anthropic. Agents in Biology. https://www.anthropic.com/research/agents-in-biology
Datahub. What is a Data Catalog? https://datahub.com/blog/what-is-data-catalog/
Heimsbakk, V. The Semantic Medallion: Building a Knowledge Graph-Powered Data Catalog. Modern Data 101 (2026).
Model Context Protocol documentation. https://modelcontextprotocol.io/docs/getting-started/intro
Pheiron team. https://www.pheiron.com/about
 | A guest post by
|
 |
|
 | A guest post by
|