Skip to content
Search

Data Scientist

lastmod 2026-04-02created 2017-02-205 edits

A big data analyst is a big data specialist, also called a ‘digital scientist’ or a ‘data scientist’.

The Revenge of the Data Scientist

  • The Revenge of the Data Scientist | GeekNews
  • With the arrival of LLM APIs, data scientists and MLEs were pushed off the critical path of shipping AI, but the essence of the actual work—experiment design, metric measurement, debugging probabilistic systems—has not gone away
  • Both OpenAI's Codex case and Karpathy's auto-research project show that the core is a harness made up of tests, metrics, and an observability stack, and much of this harness is data science
  • Five evaluation (eval) pitfalls that recur in practice—generic metrics, unverified judge models, bad experimental design, bad data and labels, and excessive automation—are undermining the quality of AI systems
  • The common cause of each pitfall is a lack of data science fundamentals, and existing methodologies such as exploratory data analysis, model evaluation, experimental design, data collection, and production ML are the solutions as they are
  • "Looking at the data directly" is the highest-ROI activity yet the one most often skipped, and the role of the data scientist remains essential in the LLM era

The essence of the harness is data science

  • OpenAI's Harness Engineering concept describes a structure in which agents develop code autonomously in an environment surrounded by tests and specifications
    • The harness includes an observability stack of logs, metrics, traces, etc., so agents can detect abnormal behavior on their own
  • Andrej Karpathy's auto-research project shows the same pattern of iterative optimization against a validation loss metric
  • "Calling an LLM through an API" does not eliminate the work of experiment design, debugging, and metric design, and much of the harness is data science
    • In the past, a lot of time was spent inspecting data, checking label consistency, and designing metrics
    • Now there is a tendency to rely on the model's “vibes” or to use open-source metric libraries as-is
  • Misunderstandings due to a lack of data understanding are especially common in RAG (Retrieval-Augmented Generation) and Evals
  • Claims like "RAG is dead, evals are dead" come up, yet the very teams making those claims are building systems that rely on those concepts
  • The tendency of engineers without a data background to avoid what they don't understand out of fear is pronounced in retrieval and evals
  • "Calling an LLM through an API" does not eliminate the work of experiment design, debugging, and metric design, and much of the harness is data science

5 Eval pitfalls

  • Pitfall 1: Generic Metrics
    • Generic metrics such as a helpfulness score, coherence score, or hallucination score are too broad to diagnose an application's actual failures
    • A data scientist does not adopt metrics right away, but first explores the data and traces directly to figure out "what is actually breaking"
    • "Looking at the data" means reading traces directly, building custom trace viewers, and doing error analysis to categorize failures
    • Generic similarity metrics such as ROUGE and BLEU rarely fit LLM output, and application-specific metrics such as "Calendar Scheduling Failure" or "Failure to Escalate To Human" are needed
    • Looking at the data is the highest-ROI activity and the one most often skipped
  • Pitfall 2: Unverified Judges
    • Most teams using an LLM as a judge have no answer to "how do you trust the judge?"
    • A data scientist treats the judge like a classifier, obtaining human labels and splitting them into train/dev/test to measure its reliability
      • Few-shot examples are taken from the training set, the dev set is used to improve the judge prompt, and the test set is held out to check for overfitting
    • When reporting results, precision and recall should be used instead of plain accuracy — when a failure mode is only 5%, accuracy hides the real performance
    • Classifier validation has become a lost skill in modern AI
  • Pitfall 3: Bad Experimental Design
    • Test set construction: most teams ask an LLM to "generate 50 test queries", which produces unrepresentative, generic data
      • A data scientist first looks at real production data, identifies the important dimensions, and then generates synthetic examples that fit those dimensions
      • Synthetic data should be generated based on real logs or traces, with edge cases injected
    • Metric design: putting an entire rubric into a single LLM call and defaulting to a 1–5 Likert scale hides ambiguity and postpones hard decisions about system performance
      • It should be replaced with binary (pass/fail) judgments on narrowly scoped criteria
  • Pitfall 4: Bad Data and Labels
    • Labeling tends to be delegated to the development team or outsourced because it is unglamorous, but a data scientist insists that domain experts do the labeling themselves
    • There is a deeper reason beyond label quality: the concept of "criteria drift" validated in a paper by Shreya Shankar et al.—users need criteria to evaluate outputs, but they define the criteria in the process of evaluating outputs. In other words, they don't know what they want until they see the LLM output
    • Domain experts and PMs should be put in front of raw data, not summary scores
  • Pitfall 5: Automating Too Much
    • LLMs can help write plumbing code, generate boilerplate, and wire up evaluations
    • But the work of looking at the data directly cannot be automated — because "you don't know what you want until you see the output"
    • Other additional pitfalls mentioned: misusing similarity scores, asking judges vague questions like "Is this helpful?", making annotators read raw JSON, reporting uncalibrated scores without confidence intervals, data drift, overfitting, bad sampling, and hard-to-understand dashboards

Mapping to data science fundamentals

  • All five pitfalls share the same root cause: a lack of data science fundamentals
  • How each pitfall maps to an existing methodology:
    • Reading traces and categorizing failures → exploratory data analysis (EDA)
    • Validating LLM judges with human labels → Model Evaluation
    • Building representative test sets from production data → Experimental Design
    • Labeling by domain experts → Data Collection
    • Monitoring how the product behaves in production → Production ML
  • The names have changed, but the work itself is the same
  • Python is still the best toolset for exploring and working with data
  • Released an open-source plugin (evals-skills) as a tool for analyzing eval pipelines and diagnosing what is wrong

See also

Favorite site

author Zer0 sitemap