How to Transition from Data Analytics to Data Engineering
A Strategic plan to help you navigate your career shift into Data Engineering
In our previous post, one of the co-founders of Pipeline To Insights shared his reason for transitioning into data engineering. We also conducted a poll, and based on the results, we've decided to write this post specifically for data analysts and professionals in similar roles looking to switch to data engineering.
For those who missed it, you can check out the post here:
In this post, another co-founder of Pipeline2Insights shares the key strategies that helped him transition from a data analyst to a data engineering role.
You'll find a clear, actionable plan that you can follow or adapt to create your own roadmap to success.
In this post, we’ll explore:
How to leverage your current role to facilitate a smooth transition within your team or company.
The value of the data engineering community in supporting your journey.
How to build a structured plan for your transition into data engineering.
Why analysing job postings from top platforms is key to identifying the critical skills you need.
Materials, Books, and Courses
How preparing for interviews can highlight knowledge gaps and help you improve.
Whether you’re just starting out or already diving into data engineering, this guide will help you take actionable steps toward achieving your goal.
Leverage Your Current Job & Data Team
Start with Your Data Team
As a data analyst, you're often the consumer of the data that data engineers prepare. So why not start learning from them?
The first step in transitioning to data engineering is understanding how your team’s data is sourced, processed, and managed. Ask the colleagues and Data Engineers in the team:
How is the data prepared before it reaches you?
What does the overall data platform or architecture look like?
If you have access to a data warehouse or data lake or so, how is the data ingested?
Who is responsible for automating data pipelines, and what tools do they use?
How do they ensure data quality and reliability?
If your company uses cloud services, try to gain a general understanding services they use and why they use them.
A great way to start is by getting closer to your data engineers. Ask for their support where possible, ask if they have any repositories or documentation you can review, and take notes on what you learn.
Then, reinforce your learning by:
Watching tutorials on YouTube.
Reading relevant documentation in the company.
Trying to get hands-on experience with the tools they use.
Apply What You Learn in Your own Job
If your role involves writing scripts to extract, clean, or transform data, think about how you can automate these tasks. Look for opportunities to:
Streamline data extraction processes.
Automate repetitive data-cleaning tasks.
Implement data quality checks.
Find a way to apply something that you learn from your data team in your current role.
By taking these initiatives, you not only develop data engineering skills but also add value to your team. Over time, this experience could open doors for you. If your team ever needs a data engineer, your proactive approach and strong connections with the data team could make you their top choice.
Being actively involved increases the chances that they’ll think of you when the right opportunity arises. Since you already know the data and the business context, stepping closer to data engineering can be a win-win for both you and your company.
Analysing Data Engineering Job Descriptions
In addition to learning from the data engineers in your company, you can explore job postings to understand the skills required for a data engineering role.
Platforms like SEEK and LinkedIn (especially in Australia) are great places to research job descriptions. Look for:
Key responsibilities and daily tasks of data engineers.
Required skills and qualifications.
Common tools, platforms, and technologies are mentioned.
Make a list of these skills and compare them with what you already know. Identify your gaps and create a learning plan to bridge them. This learning plan may include some courses, certificates or hands-on projects to be included in your GitHub profile. We will talk about the plan next.
To gain a clearer understanding, familiarise yourself with the different skill categories within the data engineering lifecycle. This will help you see how each skill fits into the broader workflow and its role in the data pipeline.
For more information about the data engineering lifecycle, you can check out this series: [Data Engineering Lifecycle].
Engaging with the Data Engineering Community
Meetups & Conferences
Another great way to dive into the data engineering field is by attending local meetups or conferences in your city, either virtually or in person. These events offer valuable opportunities to network with others, discover the latest tools, and gain insights into real-world problems being solved in data engineering. Presentations often showcase the tools, challenges, and data architectures in use, providing you with essential exposure to the field.
Social Platforms
Platforms like LinkedIn, Substack, and others are perfect for following pioneers in the data engineering space. By connecting with experts and learning from their experiences, you’ll gain insights into emerging tools, industry trends, and the technologies companies are adopting.
Personally, I found that following thought leaders on LinkedIn and Substack really helped me organise my thoughts and deepen my learning. Here are a few influential voices in the data engineering world you can follow:
- ( Substack, Discord, Linkedin )
- ( Substack, Discord, YouTube, LinkedIn)
Benjamin Rogojan -
(Substack, YouTube, Discord, Linkedin)Data Quality Camp (
& )( Slack , Substack)- (Substack, Linkedin)
- (Substack)
Data Engineering Communities
Reddit: https://lnkd.in/gawjdJsS
Discord: https://lnkd.in/gezJK_U5
Plan Implementation
Based on what you’ve learned and understood from previous stages, it’s now time to organise your thoughts and create a concrete plan with actionable steps. Here's a template I use to assess my own skills, determine my proficiency level, and identify the gaps between my current skills and those required for a data engineering role.
Remember, only you truly understand the level of your skills, so base your assessment on what you know and what you need to learn.
The transition plan we provided earlier is meant to be iterative. As you gain insights from interviews, update your learning plan to address any weaknesses. Waiting until you feel "fully ready" isn’t practical, growth comes through experience.
Exclusive Offer for Our Founding Members: Personalised Career Consultations
We’re excited to offer 45-minute consultation sessions exclusively for our Founding Members.
✅ Get guidance on:
Refining your career transition strategy
Improving your CV and cover letter
Discussing any other career-related topic with both Pipeline To Insights Co-Founders
These one-on-one (in our case two-on-one 😊) online sessions are designed to provide personalised advice and actionable insights to accelerate your journey.
Learning materials
Start with the Fundamentals
We always recommend starting with the fundamentals and then getting familiar with the tools used in the industry.
A great approach is to begin with open-source tools, as they provide a strong foundation, and then gradually expand your knowledge to enterprise-level tools. Additionally, some companies require certifications, so it’s beneficial to research the job market in your country and identify the most in-demand certifications.
Recommended Learning Materials
There are countless learning resources available for data engineering, but based on our experience, these materials might be particularly helpful.
Book
Fundamentals of Data Engineering: Plan and Build Robust Data Systems
Courses:
DataExpert.io Academy
Specialisation:
Professional Data Engineer by Google
Data engineering handbook: This is a repository with links to everything you'd ever want to learn about data engineering
Learn by Doing
As you learn, it's highly recommended to apply your knowledge through hands-on projects. Implementing what you learn not only strengthens your skills but also helps you build a portfolio that can showcase your expertise when applying for roles.
Prepare Your CV and Learn from Interviews
Prepare Your CV
As you transition into data engineering, it's important to continuously update your CV to reflect your progress. Include new skills, certifications, and projects that showcase your expertise. A well-structured CV increases your chances of landing interviews, so invest time in crafting one that highlights your strengths and relevant experience.
Use Interviews to Improve and Adapt
You don’t need to master everything before applying. Many job requirements are flexible, and some listed qualifications may be negotiable. Apply even if you don’t meet every criterion, interviews are a valuable learning experience. They help you identify gaps in your knowledge and refine your strategy for the next opportunity.
Both authors of Pipeline to Insights have gone through multiple interviews during our transitions from data science and data analytics. Based on our experiences, we’ve created a series to help you prepare for real-world interview questions and scenarios. You can explore the series here:[Link]
Conclusion
In conclusion, transitioning from a data analyst to a data engineer is an achievable and exciting journey that requires a combination of proactive learning, hands-on experience, and engagement with the data engineering community. By leveraging your current role, closely collaborating with your data engineering team, and continuously refining your skills, you can build a solid foundation for your career shift.
Don’t be afraid to dive into job descriptions to identify areas for improvement, and use interviews as a tool to further sharpen your knowledge. The road to becoming a data engineer might be challenging, but with persistence and the right approach, you’ll be well-equipped to take on this new and rewarding career path. Keep learning, stay engaged, and take consistent action to move closer to your goal.
We’d love to hear about your experience too!
Have you transitioned between data roles?
Are you considering a shift from Data Science, Analytics, or Software Engineering to Data Engineering?
What challenges have you faced in your career journey?
Check out the below post for my whole career journey.
Also if you are interested in more learning about the transition from data analyst to Data Engineer check out;
I am interested how transition data engineering to analytics engineer
Spot on, I think this what many data analysts are thinking of.