Why I Moved from Data Science to Data Engineering
How my background in Data Science shaped my journey into Data Engineering
I want to start with the lesson I’ve learned from my journey.
If you’re a Data Scientist considering a shift, know that your skills aren’t wasted, they give you a huge advantage in building data systems that actually serve the business. And if you’re a Data Engineer, working closely with the stakeholders, and understanding their pain points can make you an even better engineer.
My data journey began as a data scientist at a startup, where I was the only member of the data team. There were no data engineers or data architects, just me, trying to extract insights and build machine-learning models from very messy data mostly in separate Excel sheets.
Like many Data Scientists in small teams, my main focus was on models and insights, but I quickly realised that before I could analyse anything, I needed to build the data pipelines and models to support my tasks. While I enjoyed the challenge, I often wished I had a Data Engineer colleague, someone who could listen to my data needs and build systems, so I could focus purely on science and analytics.
Fast forward to today: I became that Data Engineer. In this post, I’ll walk you through my transition from Data Scientist to Data Engineer, covering:
My experience as a Data Scientist
Why I decided to transition to Data Engineering
The challenges I faced
The advantages of being a former Data Scientist
If you’re a Data Scientist considering a shift, or a Data Engineer working closely with analysts and scientists, I hope my story gives you valuable insights into how these two worlds connect.
Please tell us about your current career point, we’d love to hear from you!
Note: Later we will also share the experience of the other co-author of Pipeline To Insights who transitioned from Data Analytics to Data Engineering. Don’t forget to subscribe, if you are interested!
My Experience as a Data Scientist
My journey started with a fascinating challenge, developing an AI-powered Yoga Instructor. It was an exciting, high-impact project where I worked closely with yoga instructors to create an intelligent system that could guide users through personalised yoga sessions.
At first, I was expecting to spend my time designing and training AI models, fine-tuning parameters, and experimenting with data. However, reality quickly set in. The first month wasn’t about building models, it was about defining requirements:
What data do we need?
What kind of data model suits the business goals?
Which tools and infrastructure should we use?
Once we had a plan, I expected to dive into machine learning, but instead, I found myself deep in:
Cleaning massive Excel sheets and making sense of a bunch of text data
Creating and managing databases to store training data
Ingesting data into the database
Automating data ingestion and transformation pipelines
Deploying on the cloud while implementing monitoring and security best practices
By the time I finally got to the model-building stage, about 2 to 2.5 months had passed.
The project was successful. But reflecting on it now, I’d say my time was split roughly like this:
20% Data Modelling
50% Data Engineering
30% Data Science
I hadn’t expected that breakdown. However, it was the same with the second and the third projects too. I was hired as a Data Scientist but spent more time building data pipelines and infrastructure than doing any science. This realisation pushed me to rethink my career path. I started reading more about data systems, architectures, and data engineering best practices.
Why I Decided to Transition to Data Engineering
After spending two years as a Data Scientist, I took a step back to reflect on my journey. I reviewed my projects, my daily tasks, and what truly excited me. That’s when I realised I found more fulfilment in building data infrastructure, designing data pipelines, and solving data quality challenges.
One of the biggest lessons I learned was that no data project can succeed without solid Data Engineering. Regardless of whether it’s done by a Data Engineer, Data Scientist, or Data Analyst, the data must be collected, cleaned, transformed, and structured properly, otherwise, analytics and machine learning efforts fall apart.
As a Data Scientist, I often wished for better-engineered data. I knew exactly what kind of infrastructure would make my life easier. Then it hit me, what if I became the Data Engineer I always wished I had?
That realisation changed everything. I saw an opportunity to bridge the gap between Data Science and Data Engineering, ensuring that my future downstream stakeholders (Data Scientists, Analysts, and Business Users) could work without struggling with poor data quality or broken pipelines. This mindset shift was the turning point. I no longer just wanted to build models, I wanted to build the foundation that enables great models and insights. And with that, my transition into Data Engineering truly began.
Challenges I Face in the Transition
Transitioning from Data Science to Data Engineering was both exciting and demanding. While I understood the theoretical concepts, there were several key challenges I had to overcome:
1. Bridging the Skill Gaps
Even though I had a solid understanding of data, I needed to level up in several technical areas:
SQL & Data Modeling – Practicing daily, completing certification programs (e.g., Meta Database Engineer), and applying knowledge in real projects.
Software Engineering – Improving coding practices, CI/CD, and version control, ensuring I wrote scalable and maintainable code.
Cloud Platforms & Best Practices – Moving beyond the "just enough to get the job done" knowledge and diving deeper into cloud services, security, and scalability.
Tools & Technologies – Learning industry-relevant tools like Spark, Kafka, dbt, and Airflow—which seem to be listed in every job description!
2. Starting with Fundamentals
To build a strong foundation, I began with the basics. Reading Fundamentals of Data Engineering by
and Matt Housley was a game-changer. It helped me understand the core principles, architectural patterns, and best practices that every Data Engineer should know.3. Adapting from a Scientist’s Mindset to an Engineer’s Mindset
Moving from experiment-driven work to infrastructure-focused thinking.
Prioritising scalability, efficiency, and reliability.
Ensuring pipelines were automated and resilient, rather than just solving problems ad hoc.
4. Overcoming Imposter Syndrome
Career transitions can be intimidating. It’s easy to feel behind others who have been Data Engineers for years. But I chose to see this as an advantage, my Data Science background gave me a unique perspective on how to design data pipelines that truly serve data needs.
The Advantages of Being a Former Data Scientist
Transitioning from Data Science to Data Engineering gave me a unique perspective on how data flows through an organisation. While the shift required learning new skills, my experience as a Data Scientist became a strong advantage in my new role. Here’s why:
1. Deep Understanding of Downstream Needs
As a Data Scientist, I knew first-hand the struggles of working with incomplete, unreliable, or inefficient data pipelines. Now, as a Data Engineer, I build pipelines and infrastructure with the end user in mind.
2. Strong Communication Between Data Teams
One of the biggest challenges in any data-driven company is the disconnect between Data Engineers, Analysts, and Scientists. Since I’ve been on both sides, I can speak both languages. This helps in:
Defining data requirements more effectively
Aligning expectations between teams
Ensuring that pipelines serve real business needs rather than just technical goals
3. A More Holistic Approach to Data Quality
Bad data can break models. As a former Data Scientist, I understand what "bad data" looks like in a predictive model or analysis. This helps me:
Implement better validation checks and data quality controls
Design pipelines that minimise biases and inconsistencies
Anticipate the kinds of data issues that could impact analysis and ML models
Would I do it again? Absolutely.
Final Thoughts
This has been my experience transitioning from Data Science to Data Engineering, a journey of challenges, learning, and growth. Looking back, I can see how each step, from building AI models to designing scalable pipelines, shaped my understanding of the data lifecycle.
If you enjoyed this post and want more insights on Data Engineering, career transitions, and best practices, consider subscribing to Pipeline to Insights.
But this is just one path in the vast world of data, there are countless career journeys, challenges, and insights waiting to be shared.
Check out the below post for my whole career journey starting from Industrial Engineering.
Also, if you want to strengthen your foundational knowledge, you can check out our posts about the Data Engineering Lifecycle: [Post List]1
For more reading about transitioning from Data Science to Data Engineering, we highly suggest below articles:
Transition: Data Scientist to Data Engineer by
Is Data Engineering For You? Maybe you’re a recovering data scientist? by
We’d love to hear about your experience too!
Have you transitioned between data roles?
Are you considering a shift from Data Science, Analytics, or Software Engineering to Data Engineering?
What challenges have you faced in your career journey?
https://pipeline2insights.substack.com/t/data-engineering-life-cycle
Method payment
Great insights! Implementing the right Customer Data Platform is crucial for modern businesses. At CData Insights, we specialize in delivering the best customer data platform in GTA, along with Google Cloud database integration and digital asset management solutions. Check us out!
https://cdatainsights.com/services/data-engineering