Common pitfalls in proteomics data pipelines: where months quietly disappear

Proteomics produces enormous amounts of data, and processing it takes several steps, each with their own specialized tools (e.g., Andromeda, MSFragger or Pulsar as search engine?).

Whenever proteomics lies at the heart of a drug discovery engine, automating the data pipeline is non-optional, since the pipeline has to run daily and handing data from tool to tool is not only cumbersome but also error-prone. Building one pays off quickly, but the shortcuts of a rushed design surface later: data quality is passable rather than good, the structures are too rigid, and before long everyone is back to their own Excel sheet.

Here are three pitfalls I keep seeing in my work with proteomics biotechs.

Cheap data checks prevent costly mistakes

You’ve probably heard stories like this before: an analysis on a certain cell line begins. Data is pulled from the database. Conclusions are drawn. Excitement builds. But then it turns out: we only looked at THP1 and not THP-1. The filter is broadened to include THP-1, the effect dissipates. Back to square one.

Preventing this is straightforward: dedicate one layer of the pipeline to checking data integrity, and make it say clearly what failed and why. Sounds like tedious hygiene? It surely is! But it pays heavy dividends in quality, trust and efficiency.

A few data integrity checks to cover:

• Controlled vocabulary (cell lines, digestion methods, TMTs)
• Sequence content (only the 20 AAs + dedicated symbols for PTMs without spaces)
• Peptide–protein sequence matching (each peptide’s sequence should be found in the protein it is matched to)
• Cell line–protein matching (no human proteins in canine cell lines)
• Molecular validity (SMILES must be parseable and chemically valid)
• Cardinality constraints (number of wells per plate or TMT channels per plex)
• Numerical ranges (fold changes > 0)
• etc.

Metadata encodes decisions; don’t let it slip away

“Do these fold changes include the wells from the corners of the plates? Last month, we noticed a spatial pattern that reoccurred across many different samples…”. You’ve probably experienced this situation: someone asks such a question and the scientist responsible grows visibly uneasy. Not because they could not answer it right now, but because they might not be able to answer it at all.

As each stage of a proteomics pipeline typically uses its own specialized tool, data is handed over several times between different pieces of software along the way (e.g., from peptide ID to protein inference to quantification to normalization). This is where metadata gets lost. Against which protein database were your peptides searched? Which threshold was applied for the isolation specificity filter? The list goes on.

The fix is simple: each hand-over must carry the metadata as carefully as the data. This is the only way to know what exactly was done to your data, even after months have passed. The data contains the result, but the metadata anchors the interpretation.

How to do it in practice? Capture all metadata and store it right next to the actual results in the database, for every single run. Even if 90% of the metadata does not change for months, do it anyway, because the remaining 10% matters and storage costs nothing.

“How do we do our FDR correction?” Ask the data pipeline!

“Why did you use Storey–Tibshirani for FDR correction? Didn’t we agree that Benjamini–Yekutieli is better because… ?” As a team learns and grows, myriad agreements on how to best look at the data accumulate. But how to keep track of them? How to enforce them? And how do new joiners learn about them? We all know this: documentation, wikis, etc. are supposed to do that, but only in theory.

Purpose-built data pipelines offer a new solution to this perennial problem: directly operationalize these decisions rather than just document them. This means every outlet of the pipeline, whether it’s a CEO’s dashboard, a data scientist’s working table, or a biologist’s go-to QC analysis, must always build upon the same underlying data.

No decision is set in stone, the data pipeline simply represents the team’s current standard of how to process, analyze, and interpret the data. If that default is no longer the best way to go, it surfaces, a discussion is triggered, the pipeline gets revised, and the new standard is immediately live. This elevates the data pipeline from a piece of infrastructure to a cornerstone of the organization’s collective knowledge.

A few decisions a data pipeline encodes:

• FDR-correction for p-values
• Hit definitions
• Data normalization
• Missing value handling
• Quality filters cutoffs
• etc.

Why this matters

Designing data pipelines for proteomics is not just engineering. Each decision at every layer shapes the analyses that follow. Understanding this is what separates a pipeline that fits the science from one that merely runs. That is the work I have spent the past four years on.