From Analytics to Data Engineering: A Career Journey

How curiosity and continuous learning drive the transition from Analytics to Data Engineering

Nov 24, 2024

Hello everyone, whether you're into data or just curious! 🙂👋

Pipeline to Insights was founded by two data engineers who transitioned from analytics and industrial backgrounds. In this post, we share Erfan’s story, one of the co-founders, and his journey into data engineering, a path that was not always easy but one he now loves.

From Software Engineering to Data Passion

My inspiration to pursue a career in data originated from the subjects I studied during my bachelor’s degree. At university, I was introduced to topics like Data Structures, Algorithms, SQL Fundamentals, Software Engineering, and AI Fundamentals, all of which I enjoyed deeply. However, after graduation, my career options felt limited to roles such as Web Developer, Software Engineer, or Network Engineer. I decided to start as a junior software engineer, focusing on application development.

While developing applications, I became increasingly curious about the data stored behind the scenes, how it could solve business problems, the tools and techniques we could use, and the insights it could reveal. This curiosity led me to explore the world of data and find my passion in the field.

From Software Engineering to Data Analytics

After working as a Software Engineer for around two years, I decided to further my education in data by pursuing a master’s degree in Business Analytics with a specialisation in Data Science in Australia. As I progressed through the program, my enthusiasm grew. During my studies, I developed a strong passion for exploring, analysing, and experimenting with data to uncover valuable insights. This growing interest ultimately led to my first data role as a Data Analyst at Allianz.

Throughout my data career, my curiosity kept me moving forward. I frequently asked myself how the data I worked with was gathered, ingested, and stored. These questions pushed me to seek answers from my team and expand my knowledge. This curiosity grew even stronger when I joined Xero as an Analytics Engineer, where I had the chance to explore these processes more deeply.

From Analytics to Data Engineering

At Xero's XSBI team, I worked closely with the data team responsible for gathering, ingesting, and storing data in the data lake. As a stakeholder of the data team, I used dbt to apply business and accounting logic, making the data ready for other downstream users. However, this didn’t stop me from wanting to learn more about how the data was gathered and how the data platform operated. During my meetings with the data teams, I made it a point to ask as many questions as I could and learn about the behind-the-scenes processes.

This helped me understand the data engineering side of things, which ultimately led me to consider taking the next step closer to the full data lifecycle.

Based on my insights and research, I created a personal plan to bridge the gap between my skills and the requirements of a data engineering role. I focused on:

Improving key areas
Pursuing certifications
Reading a great book Fundamentals of Data Engineering by
Joe Reis
and Matt Housley

to deepen my knowledge and to prepare for interviews. Throughout the interview process, I identified my weaknesses, worked on them, and applied my learnings to each new opportunity. This approach ultimately helped me land my first role as a Data Engineer at AFL.

After my time at AFL, I joined Airmaster to work on greenfield AI projects, where I contributed to building an AI-ready data platform. I continue to explore new tools and technologies in data engineering, attend industry events, and work on personal projects to stay ahead in this dynamic field.

Continuous Learning and Growth

I believe that learning never stops, especially in Data Engineering. Here are the actions I’ve taken to keep growing:

Joined DataEngBytes: Connected with the data engineering community through meetups and conferences in Australia and New Zealand.
Attended Local Meetups: Engaged with industry leaders at local meetups and on social media.
Expanded Knowledge in AI: Joined AICamp as an organiser to stay updated on data engineering and AI.
Presented at Meetups: Presented on "Data Quality in the Age of AI" (Aug 2023) and "Introduction to dlt" (Nov 2023).

I’m committed to continuous learning and contributing to the community as I continue to grow in this field.

Lessons I’ve Learned

Here are the key lessons I've learned:

Overlapping Roles: Data Analyst and Data Engineering roles can overlap, especially in smaller companies. Ensuring the data is accurate and relevant when collecting it helps bridge the gap between analysis and engineering. This ensures the data will be valuable to others down the line.
Prioritise Fundamentals and Problem-Solving: From
Joe Reis
,
Zach Wilson
and my colleague Adebayo, I’ve learned to focus on foundational skills and critical thinking when solving problems. Approaching challenges with a mindset of generating business value is more important than jumping straight to tools.
Flexibility in Problem-Solving: Not every problem has a one-size-fits-all solution. Adebayo emphasised the importance of using multiple steps or approaches when necessary.

Final words

I’m grateful to my mentors and managers for their constant support and guidance. I’ve always believed that no task is too small to ask for help, as every learning opportunity counts. Sharing knowledge and supporting others creates meaningful impact, which is why Hasan and I founded Pipeline to Insights because when we help each other, we all succeed.

If you have a similar story or unique path into data engineering, we'd love to hear about it. Please feel free to share your experiences in the comments below!

Thank you 🙂