← Back
Towards Data Science

Censored vs. Truncated Regression in Python, Explained

•
#llm
Level:Intermediate
For:AI Engineers
✦TL;DR

The article demonstrates that ordinary least squares (OLS) underestimates a simulated $5 k‑per‑year salary effect, shrinking it to $3.2 k, while Tobit and custom truncated‑likelihood models recover the true effect. It walks through building Tobit and truncated regression likelihoods from scratch in Python, highlighting the bias introduced by censoring. Engineers can apply these custom likelihoods to obtain unbiased estimates when dealing with censored or truncated data.

⚡ Key Takeaways

  • OLS estimated the $5 k effect as $3.2 k due to censoring.
  • Tobit regression and a manually coded truncated likelihood correctly recover the $5 k effect.
  • Using standard OLS on censored data trades computational simplicity for biased coefficient estimates.
  • Implement the Tobit or truncated model by defining a custom log‑likelihood function and optimizing it with a Python optimizer (e.g., `scipy.optimize.minimize`).
  • The approach requires writing the likelihood from scratch; no off‑the‑shelf library is used, so users must ensure correct gradient handling and convergence checks.
💡 Why It Matters

Correctly modeling censored or truncated outcomes prevents systematic bias in key business metrics like salary estimates, enabling more reliable decision‑making in production analytics pipelines.

✅ Practical Steps

  1. Define the Tobit or truncated log‑likelihood function in Python as shown in the article.
  2. Use `scipy.optimize.minimize` to fit the model parameters to your censored dataset.
  3. Validate the recovered coefficients against known simulated effects to confirm unbiased recovery.

Want the full story? Read the original article.

Read on Towards Data Science ↗

More like this

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

MIT News AI•#llm

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

Towards a Science of Capable Autonomous Agents & Provable Safety — Part II

CMU ML Blog•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING