After a Year Rebuilding Search, I Had to Rethink Everything

This is not a universal blueprint. It's our story—the lessons from a year spent rebuilding search from the inside out.

The Journey in a Nutshell

The Plan

Start Finish

We started with sound architectural principles.

The Reality

Start Finish

We returned, wiser, to user-centered simplicity.

Lesson #1

Assumption: Search is an Algorithm & Infrastructure Problem

The Truth: Search is a Data & Product Problem first.

Great search is shaped by user behavior, not architecture diagrams. The most powerful architecture isn’t the one that serves the model fastest, it’s the one that learns fastest.

  • Data quality beats model complexity: Simple models on high-quality, near real-time user data often beat complex ones on stale offline sets.
  • Tight feedback loops are key: Minimize the time between user action, logging, retraining, and deployment.
Lesson #2

Assumption: We must engineer for correctness from day one

The Truth: Velocity unlocks correctness.

In complex systems, perfecting every component before shipping kills learning. A rough but working pipeline in production will surface truths you won’t see in a lab.

  • Scrappy end-to-end tests teach fastest: Get a working pipeline live to see real-world issues.
  • Refactor after validation: Once an idea shows impact, then harden it and clean the code.
Lesson #3

Assumption: Offline metrics are our north star

The Truth: Business impact is the north star.

We obsessed over nDCG and recall. Dashboards looked great, but A/B tests told a different story. Offline metrics are useful for debugging, but they are not the goal.

  • Tie performance to economics: Measure tradeoffs in latency and complexity against revenue, clicks, and retention.
  • Ship only what moves the needle: If an experiment doesn’t move a business KPI, it’s not a success.
Lesson #4

Assumption: Functional roles define the work

The Truth: Blurring the lines unlocks synergy.

We began with clean separation: data science, backend, platform. We optimized locally, not globally. Acceleration started when we turned specializations into a shared toolkit.

  • Enable data scientists: Let them run A/B tests and iterate without ticket ping-pong.
  • Upskill backend engineers: Give them data and observability tools so they can see behavior, not just code.
Lesson #5

Assumption: Technical excellence guarantees a great product

The Truth: Product mindset is the true compass.

The market rewards solving the right problem for the right user, at the right cost. A product mindset means you keep asking "who," "what," and "why."

  • Who is the user? What are they trying to do, and what is hard for them today?
  • Which business outcome do we enable? How will we know if we've succeeded?

My Go-Forward Checklist

Focus on signals

Optimize for learning speed

Tie everything to business KPIs

Organize for synergy

Adopt a product mindset