InterviewLegend

Design a Data Enrichment Pipeline

Sometimes asked
Alex Chen24 min readAdded 328 real interview reports

This one was reported in 2021, from a Cloud Engineer interview for an experienced hire, with no band stated. It's a single report, so treat the details as one telling of the prompt. The tagging service was described as already built, with an API the candidate could call but didn't own. The interviewer pushed on three things: how you'd build it and on what stack, which errors could happen, and what happens when one video URL is invalid. Then the same question again, with 100.

Plan to spend most of the round on that last question. Moving files into a queue and calling an API is the easy half. The hard half is what the pipeline does when the data is wrong, when the tagger is slow or down, and when a worker dies halfway through a file. The candidate ran out of time before getting through the errors, and said afterwards that this was an important part: invalid URLs shouldn't be sent to Kafka, because everyone can consume it.

The IBM angle
Every follow-up in the report was about errors, and the tagging service belonged to someone else. So spend your time on how the pipeline shields that service, and the consumers downstream, from bad data. Throughput is the easy part.

IBMPremium lesson · System design round

The IBM design, worked end to end.

  1. Requirements and scopeSystem design
  2. The API and data model
  3. The design, drawn out1 diagram
  4. 5 deep divesWith the tradeoffs
Get the IBM questions

$49 a year · all 107 companies · instant access

328 IBM interview reports · guide updated

IBM interview guide

Know what IBM asks before you walk in.

30 coding · 4 system design · 13 behavioral questions

Get Premium$49 a year · Instant access
Or start freeNo card.

Get their questions first.

Had your IBM interview?How we research

InterviewLegend is independent and not affiliated with or endorsed by any company it covers.