As a data engineer, I’m learning how to use AI better in my daily work. One way I do that is by learning from people who are actively building with AI. I invited Jenny, an AI builder and the writer behind Build to Launch, to share her experience and insights with you.
Jenny spends her days helping people ship real products using AI tools. She sees closely the difference between being “AI-assisted” and becoming “AI-dependent.”
She’s not a data engineer by title, but she has built enough data-heavy systems, pipelines, automations, and database architectures, to understand when AI is helping and when it’s hiding blind spots. That tension is exactly what this post explores.
Someone I work with recently tried to build a data warehouse from scratch using AI assistance. No prior data warehousing experience, just clear requirements and an AI coding tool.
The AI delivered. It generated a schema, wrote the ETL jobs, and set up the loading process. The data flowed. Stakeholders could query it. On paper, it worked.
Then came the performance review.
Queries that should take seconds were taking minutes.
The tables weren’t partitioned in a way that matched how the data was actually accessed.
The AI had built a data warehouse, but not one optimised for this specific workload.
The frustrating part? They couldn’t fix it. Not because the solution was complicated, but because they didn’t understand why the AI had made the choices it made.
What’s a star schema? Why fact tables vs dimension tables? What grain should each table be at? How do slowly changing dimensions work? What partitioning strategy makes sense for his query patterns?
They ended up going back to basics, learning data warehouse fundamentals with AI’s help. Star schemas. Fact and dimension tables. Grain decisions. SCD types. Partitioning strategies. The concepts that separate “it runs” from “it performs.”
By the end, they had a working data warehouse and the knowledge to evolve it. But the detour cost him; he’d built the system twice. Once without understanding, then again after learning what he should have known from the start.
I’ve seen this pattern across every domain where AI coding tools have taken hold. And it raises an uncomfortable question: as AI gets better at writing code, what happens to the engineers who stop understanding what the code actually does?
This isn’t about whether to use AI tools. That ship has sailed. The question is: how do you use them without hollowing out the judgment that makes you hireable, promotable, and indispensable?
In this post, I’ll cover:
The Real Risk Nobody Talks About: skill loss and why fundamentals matter more, not less.
The AI-Enabled Data Engineer (Not AI-Replaced): what to let AI handle vs. what you must own.
Practical Workflows That Preserve Fundamentals: structured approaches for building with AI.
Building the Habits That Grow Over Time: practices that keep your skills sharp.
Let’s get into it.
Pipeline To Insights is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber🙂🙏.
The Real Risk Nobody Talks About
The Losing Skills Trap
When I started building extensively with AI tools, I experienced something I now call “imposter syndrome on steroids”.
I could ship features like crazy. My prototyping speed was 10x what it used to be. But when it came to architectural decisions, the kind that separate junior engineers from senior ones, I felt unqualified.
Early in my vibe coding days, my whole attitude was:
“Just build everything. Claude, make it work.” And it did work. Beautiful, functional apps are showing up on my screen faster than I’d ever managed before.
Until someone asked me about the implementation details. I opened the codebase, stared at the AI-generated logic, and realised: I had absolutely no idea how it worked. I could describe what the app was supposed to do. But the actual implementation? The logic flow? The data structures? Blank.
I felt like a manager who’d been away from the shop floor so long they couldn’t operate their own machine anymore.
This hits data engineers particularly hard. You’re not building toys. You’re building systems that other people depend on for decisions. When your pipeline breaks at 2 AM, AI isn’t getting paged; you are. And if you don’t understand the code you shipped, you’re debugging in the dark.
Why Fundamentals Matter More, Not Less
AI optimises for “works right now,” not “works well over time.” It prioritises finishing the prompt, not fitting into the broader system. It assumes perfect conditions, clean data, fast networks, and rational users. It builds beautiful rooms without checking if they connect to a house.
“AI doesn’t replace engineers, it widens the gap between those who think clearly and those who don’t.”
The engineers who are thriving right now aren’t the ones who can prompt AI the best. They’re the ones whose fundamentals are so strong thatAI becomes a powerful tool instead of something they rely on.
Data modelling decisions. Schema design and evolution. Data quality reasoning. Pipeline architecture. Failure mode thinking. These skills don’t atrophy because AI can write code; they become more valuable because fewer people are developing them.
The AI-Enabled Data Engineer (Not AI-Replaced)
The difference is important. AI-replaced engineers let AI do their thinking. AI-enabled engineers use AI to think better.
What to let AI handle
AI excels at the tedious, the boilerplate, the pattern-matching:
Boilerplate SQL transformations: standard aggregations, window functions, joins.
Initial dbt model scaffolding: getting the structure in place quickly.
Documentation drafts: first pass at explaining what the code does.
Test case generation: covering obvious scenarios.
Debugging assistance: analysing raw logs, spotting syntax issues.
But there’s a line. Cross it, and you’re not using AI; AI is using you.
Data modelling decisions. AI doesn’t understand your business domain. It can generate a star schema, but it can’t tell you whether your fact table should be at the transaction-grain or the daily-aggregate grain. That requires understanding how the data will be consumed.
Schema design and evolution. Migrations are expensive. AI will happily generate a schema that works today and becomes a nightmare when requirements change. You need to anticipate how data will grow and change.
Data quality rules and business logic validation. AI can write validation code. It cannot determine what “valid” means for your specific context. That requires domain knowledge AI doesn’t have.
Pipeline architecture and failure handling. What happens when an upstream source is late? When data volume spikes 10x? When does a transformation produce unexpected nulls? AI thinks about the happy path. You think about everything else.
Performance optimisation reasoning. AI can suggest indexes. It cannot reason about your query patterns, data distribution, or access patterns. Optimisation requires understanding the system as a whole.
Security and access patterns.This is where AI-generated code fails most dangerously. When I built my first production app with AI, I peeked into the database and found passwords stored as literal plaintext, no hashing, no encryption, just sitting there like a “what not to do” security textbook example. In another project, AI dropped my database connection string straight into the script with no .env file. On my personal website, AI-generated code called API keys directly from the client side, and anyone could open dev tools and see my private keys. I only caught these because I already knew to look. Security requires paranoia AI doesn’t possess.
AI-Enabled v.s. AI-Replaced engineer. The difference is what you own.
The Gap Between “Simple Automation” and Real Engineering
I built an automated lead generation system for a client. On the surface, it sounds like a straightforward n8n or Make automation: scrape multiple layers of a website, retrieve data from each layer, identify leads, send personalised DMs for business development.
But “simple automation” hides the real complexity. The system had to connect data across different layers to deduplicate leads matching specific criteria. That meant designing a proper data structure, defining relationships between entities, choosing the right data types, and writing SQL queries to efficiently match and merge records across tables.
AI could generate the scraping logic. AI could draft the DM templates. But AI couldn’t decide how to structure the data so that deduplication actually worked at scale. That required understanding database fundamentals: relations, normalisation, and query optimisation.
Without that foundation, you get an automation that “works” on 50 records and collapses on 5,000.
The judgment test
Before accepting AI-generated code into your pipeline, ask three questions:
Can I explain why this approach was chosen?
Do I understand what will break if X condition changes?
Would I be comfortable defending this in a design review?
If the answer to any of these is no, you’re not using AI. AI is using you. And the next production incident will expose that gap.
Asking these questions is the starting point. But you need a system that makes them automatic.
Practical Workflows That Preserve Fundamentals
The solution isn’t to avoid AI. It’s to structure your AI usage in ways that build understanding rather than bypass it.
Top-Down prompting for pipeline design
After months of trial and error, I landed on a structured approach that prevents the “beautiful demo, broken production” problem.
Instead of typing “build me a pipeline that does X,” I use a conversation that forces AI to expose its assumptions, and forces me to validate them.
Round 1: Problem Statement
“I need to build a pipeline that ingests data from [source], transforms it for [use case], and loads it into [destination]. What questions do you need answered before we design the architecture?”
This flips the dynamic. Instead of AI guessing at your requirements, it surfaces the questions you should be answering.
Round 2: Requirements Interview
Let AI ask you about:
Volume and velocity expectations.
Latency requirements.
Failure tolerance.
Data quality thresholds.
Access patterns.
Answer each one. If you can’t answer, that’s a signal that you need to clarify the requirements before building anything.
Round 3: Architecture Design
“Given my answers, propose the data flow and schema. Call out tradeoffs. Be specific about failure modes.”
Now, AI generates a design constrained by your requirements. Review it critically. Push back on decisions you don’t understand. This is where you build mental models of the system.
Round 4: Master Initialisation
“Generate a complete summary of this system, goal, stack, schema, workflow, that I’ll paste into my AI coding tool. Tell it not to start coding yet, just acknowledge comprehension.”
This is the crucial step most people skip. AI IDEs lose context fast. Without this master prompt, you end up re-explaining requirements mid-build. Table names don’t match between phases. API routes expect different data shapes than the database provides. This round documents everything in one place so the AI has full context before writing a single line of code.
Round 5: Phased Implementation
“Break this into phases. Each phase should have clear success criteria I can verify before moving to the next.”
This creates checkpoints. You’re not debugging 500 lines of interconnected code; you’re validating 50 lines at each phase boundary.
Bottom-Up debugging that builds understanding
When things break, they will use AI to teach you, not just fix the problem.
Raw output prompting: Paste the full error logs. Let AI trace the execution path and explain what’s happening.
But then follow up:
“Walk me through what this transformation is actually doing, step by step.”
“Why would this fail under high load? What assumptions is this code making about input data?”
“What would a senior data engineer change about this approach?”
You’re not just getting a fix. You’re building the intuition to prevent similar issues next time.
The Rules File Approach
Every time I catch AI generating problematic patterns, I add it to a rules file that the AI reads before generating code:
# Data Engineering Guardrails
- Never generate N+1 query patterns
- Always include retry logic for external API calls
- Validate input data types before transformation
- Document assumptions about data freshness in code comments
- Flag when creating duplicate logic — ask if a shared utility exists
- Default to idempotent operations
- Include data quality checks at stage boundaries
This creates guardrails. AI learns your standards. And you think through what those standards should be, which reinforces your own fundamentals.
Cursor refuses to modify .env files with safety guardrails
Building the Habits That Grow Over Time
Knowing the theory isn’t enough. You need practices that keep your fundamentals sharp while you leverage AI’s speed.
The weekly trace-through
Pick one pipeline AI helped you build this week. Remove the AI. Open the code. Trace through it manually, step by step.
Ask yourself: Do I understand every transformation? Can I predict what happens with edge cases? Could I debug this at 2 AM without AI assistance?
If the answer is no, that’s not failure , that’s your learning target for the week.
The “Explain It Back” technique
After AI generates something substantial, ask it to explain the tradeoffs it made. Then evaluate: does the explanation match what you’d want in production?
This catches AI’s “confident but wrong” patterns. AI writes with authority regardless of whether it’s correct. Your job is to develop the judgment to distinguish solid engineering from plausible-sounding nonsense.
Stay close to the data
Never let AI be your only interface with the actual data.
Run regular query exploration. Sample your data. Profile distributions. Understand what the data actually looks like, not what AI assumes it looks like.
The engineers who understand their data deeply catch problems that AI-first engineers miss entirely. Edge cases. Data drift. Quality degradation. These only surface when you’re hands-on with the actual records.
These habits aren’t just good engineering hygiene. They’re what separates you in an increasingly crowded market.
Your Competitive Advantage
The market is starting to sort itself.
On one side: engineers who let AI do their thinking. They ship fast but can’t explain their systems. They’re increasingly replaceable. If AI wrote the code, why do you need them specifically?
On the other side: engineers whose fundamentals are so strong that AI becomes leverage, not a substitute. They ship fast and understand what they shipped. They can debug, optimise, and evolve systems because they’ve built genuine expertise.
This matches what Dan observed in that same Pipeline to Insights interview: the engineers getting hired aren’t the ones who prompt best, they’re the ones whose judgment AI can’t replicate.
The fundamentals didn’t become less important. They became rarer. And rare skills command premium value.
Where to Start
If you’re already using AI tools, audit your recent work:
Pick a pipeline you built with AI assistance. Can you explain every transformation without reading the code?
Review your failure handling. Did you design it, or did AI? Do you understand every retry, every fallback, every alerting condition?
Check your testing. Are you verifying AI-generated code works, or verifying you understand why it works?
If those questions reveal gaps, you’re not behind; you’re aware. Awareness is the starting point.
If you want to go deeper on AI-assisted building practices, I write about this extensively at Build to Launch. And if you’re focused on data engineering fundamentals, Pipeline to Insights has been publishing essential content on exactly this topic.
The tools will keep getting better. The question is whether you’ll use them to amplify your expertise or replace it.
Hi Jenny - This is a remarkably useful document. Apart from the similarities to previous practices Í really appreciate how you laid out the steps and prompts that you use as you go through your project. Much appreciated!
Thank you for the opportunity to contribute to Pipe to Insights!
Hi Jenny - This is a remarkably useful document. Apart from the similarities to previous practices Í really appreciate how you laid out the steps and prompts that you use as you go through your project. Much appreciated!