Survival Tips for Data Engineers in the Age of Generative AI

Industry Experts’ Insights on AI’s Impact in Data Engineering, Highlights from Melbourne DataEngBytes 2025 Conference.

Jul 26, 2025

We’ve all heard it: maybe not every day, but often enough to feel the pressure:

You’ve probably heard those words from your friends, colleagues, or the news. Like many of you, I try to keep up, following headlines, updates, and thinking deeply about how I can stay relevant and continue growing in this fast-moving field.

This week, I attended the DataEngBytes1 Conference 2025 in Melbourne, the largest independent data engineering event in Oceania. My immediate takeaway?

“AI will replace data engineers” not only is that wrong, it completely misses the point.

Generative AI isn’t killing data engineering. It’s elevating it. This new era is redefining our role, not replacing it.

The landscape is changing rapidly, but by focusing on fundamentals, maintaining control over AI systems, and developing strong evaluation frameworks, data engineers can not only survive but also lead the transformation of their organisations.

Let’s keep learning, adapting, and building the future, not fearing it.

Some of the most insightful sessions at the conference explored how generative AI is reshaping the data engineering landscape. From

Lena Hall

’s keynote “A New Era of Intelligence: Why Data and AI Engineering is More Critical Than Ever” to the insightful panel discussion “Survival Tips for Data Engineers in the Age of Generative AI” featuring Olena Kutsenko, Lena Hall and

Juan Sequeda

the message was clear: our field is evolving fast.

Other great talks, such as “The Great Data Engineering Reset: From Pipelines to Agents” by

Joe Reis

, reminded us that the basics of data engineering have always been important and will remain that way. But we also need to adapt to fast, automated, and smarter systems that do things with data, not just move it.

I’ve gathered some of the most valuable lessons from those sessions. Here are essential survival tips to help you not only keep up but truly thrive in this new era.

Hope you find it helpful!🙂

1. Stay in Control: Don’t Run on Autopilot

The biggest risk isn't AI replacing us, it's losing control of AI.

While everyone rushes to integrate AI into their workflows, the number one survival tip is simple: be cautious with autopilot mode. When we give AI access to our files, databases, or development workflows, it's dangerously easy to click "yes, yes, yes, allow all" without thinking through the consequences. (Lena)

A small mistake in an autonomous AI system can lead to multimillion-dollar errors. Whether we're using Model Context Protocol(MCP)2 tools in our environment or agents with various capabilities, always stay in the loop and maintain control over critical decisions. (Lena)

2. Commit to learning and mastering the fundamentals

In the chaos of new AI tools, frameworks, and solutions emerging daily, focus on the philosophies and fundamentals rather than getting distracted by every new release.

Ask ourselves: Is the tool that was relevant six months ago still there? How relevant is it now?

Understanding the underlying reasoning and trade-offs of different approaches will make us valuable regardless of which specific tools come and go. When we grasp the fundamentals, we can master any tool and guide our company down the right path. (Lena)

Besides data and AI fundamentals, engineers need to build strong foundations in:

Systems thinking approach to understand trade-offs and limitations.
Problem-solving and algorithmic thinking.

In his talk, Joe Reis emphasised that regardless of how AI evolves, the data engineering lifecycle isn’t going away anytime soon. He stressed the increasing importance of system design, noting that data is the input to all AI systems, and if it’s not handled properly, the output will be nothing more than garbage.

Joe also recommended several insightful books covering technology, strategy, system design, data modelling, and more for those looking to deepen their understanding:

Kill It with Fire: Manage Aging Computer Systems (and Future-Proof Modern Ones)3 by Marianne Bellotti.
Normal Accidents: Living with High-Risk Technologies4 by Charles Perrow.
The Data Warehouse Toolkit: The Complete Guide to Dimensional Modelling5 by Ralph Kimball, Margy Ross.
Building the Data Warehouse6 by W. H. Inmon.
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems7 by Martin Kleppmann.

What New Engineers Need:

Lena emphasised that for those early in their careers, mastering the fundamentals is the most important investment. If we're just starting and can only focus on one thing during our evenings or weekends, prioritise:

Computer science principles
Distributed systems fundamentals
Distributed algorithms ( especially relevant for agent-to-agent communication)

While expectations for junior engineers are rising, it’s also never been a better time to break into the industry. Thanks to modern tools and learning platforms, it’s easier than ever to skip outdated materials and build a focused, hands-on learning path. The key is to learn by building, not just by reading. (Lena)

Juan encouraged beginners to avoid chasing the hype of becoming a “10x engineer” overnight with AI. Instead, he urged new engineers to find a sustainable, personal learning rhythm and use this moment of change to grow thoughtfully. AI can be helpful, but strong problem-solving and critical thinking skills are still irreplaceable.

Olena also shared two frustrations that resonate with many:

An overwhelming flood of often AI-generated information that’s hard to trust.
Smart tools that guess your needs incorrectly, disrupting deep work with pop-ups and interruptions.

For engineering work, deep focus is essential, and protecting that focus might be one of the most critical survival skills of all.(Olena)

3. AI is just another tool in our toolbox

Don't fall into the "everything looks like a nail" trap.

AI offers powerful automation capabilities, but it's not the only tool available. Sometimes when we get a new hammer, we try to treat everything as a nail. Many organisations are so focused on adding AI features that they forget to fix obvious product issues.

Both using AI and not using AI are valid choices. The key is conscious decision-making about when and how to deploy these capabilities.

4. Embrace the Shift from Linear to non-deterministic work

Traditional software development has been predominantly deterministic, write code once, and it behaves the same way every time. In contrast, AI systems introduce non-determinism, where outcomes can vary even with the same input. This shift demands a fundamentally new mindset in how we design, test, and maintain systems.

Tips for handling non-deterministic systems:

If we can go deterministic, do it: If traditional code solves our problem reliably, stick with that approach. (Lena)
Assess the cost of mistakes: Before using non-deterministic AI, understand what happens if it gets something wrong. (Lena)
Use AI for ambiguity, not certainty: AI excels at handling natural language and ambiguous inputs. (Juan)

5. Focus on problem-solving, not just code writing

The days of just learning Python coding alone won’t be enough anymore. The real value lies in breaking down complex problems into smaller, manageable pieces and understanding algorithms and trade-offs. (Juan)

AI can generate code implementations, but it takes human expertise to:

Decide what to prioritise in system design.
Understand trade-offs between different approaches.
Make critical business and technical decisions.
Evaluate whether AI systems are actually solving business problems.

6. Develop strong evaluation frameworks

Since AI systems are non-deterministic, evaluation becomes critical. Only humans who understand business goals can determine if an AI agent is doing its job effectively. (Olena)

Key evaluation considerations: (Juan)

Mathematical accuracy scores are just the starting point.
Business impact and problem-solving effectiveness matter most.
Build realistic benchmarks that represent real-world challenges.
Create comprehensive test cases for expected inputs and outputs.
Monitor systems continuously for unexpected behaviours.

7. Bridge the Business-Technology gap

main ideas from Lena and Juan

There's a huge opportunity to use AI to fill the communication gap between business stakeholders and technical teams. Instead of time-consuming meetings to clarify requirements, AI can help extract and clarify business needs, turning ambiguous requests into specific technical requirements.

This is particularly valuable when:

Subject matter expertise is limited.
Stakeholders don't have time for extensive technical discussions.
You need to scale business knowledge across teams.

8. Manage decision fatigue and context switching

As more processes become automated, the real bottleneck becomes human decision-making. Our role is evolving from linear, focused work to becoming a "micromanager of agents," constantly context-switching between different systems and decisions.(Lena)

This new workflow requires developing better attention management skills and learning to handle the cognitive load of frequent decision-making.

Context Beats Code

"Only we know that the legacy API returns milliseconds, but only before 2019."

AI can write code, but it lacks your contextual judgment. Your value lies in understanding edge cases, data quirks, and institutional knowledge. Don’t delegate that.

9. Become a Knowledge Engineer, Not Just a Data Engineer

The most valuable data professionals are the ones who help their companies make or save money, and a big untapped opportunity lies in business metadata. Don’t just treat metadata as documentation. Make it dynamic, accessible, and usable for decision-making.(Lena)

This is your chance to evolve from data engineering to knowledge engineering, helping AI systems truly understand your business context so they stop hallucinating and start delivering real value.(Juan)

10. Judgment is our superpower

AI tools are force multipliers, not replacements. They boost productivity but expect us to validate their outputs. AI doesn’t understand our org’s politics, our KPIs, or why that ugly ETL script still matters. We do.

As generative AI tools become widespread, the differentiator is judgment. Those who know which code is correct, why an approach will break at scale, and how to design for failure will become the most sought-after professionals. (Olena)

11. Move up the stack

Think of your work in three layers: (Lena)

Commodity layer: Routine tasks like writing SQL or building dashboards—already automated.
Collaboration layer: AI-augmented but human-judged tasks like optimisation, architecture, and debugging.
Strategic layer: Pure human expertise, diagnosing multi-system failures, navigating business logic, designing resilient systems.

AI is eating the "easy stuff", dashboards, boilerplate code, and simple ETL scripts. That’s the commodity layer. Survival depends on moving into the second and third layers.

The Bigger Picture

Remember that we're in the early adoption phase of AI technology. Like every major technological shift (electricity, automobiles, the internet), there's initial chaos and inflated expectations before things settle into more practical, sustainable patterns.

Productivity often decreases initially when new technology is introduced because we try to adapt it to old ways of working. The real breakthroughs come when we redesign processes from the ground up with the new technology in mind. (Olena)

Our survival strategy includes:

Climb the abstraction ladder: Use co-pilots and agents, but deeply understand the systems we build with them.
Get good at context translation: Between business needs, AI limitations, and technical constraints.
Design for failure: Build with chaos in mind, because it’s inevitable.
Be the voice of trust: When things go wrong, AI won’t face the CEO. We will.

Conclusion: Focus on solving problems

Don't get lost in the hype. AI is only part of the bigger picture. There are still many other building blocks that are just as important as AI in building robust, scalable systems.

The fundamental skill that will never go out of style is problem-solving: breaking down complex challenges into smaller, manageable pieces and understanding the trade-offs involved in different solutions.

Stay curious, stay in control, and remember that our value as a data engineer lies not in our ability to use the latest tools, but in our capacity to understand problems deeply and architect solutions that truly serve business needs.

The more capable AI becomes, the more valuable deep human expertise becomes. Focus on becoming the person with the best judgment and deepest expertise in our field.

If you found this post helpful, you might also enjoy these related reads on Data Engineering concepts: