Learning from other people’s experiences is one of the best ways to grow. It shows us what choices they made, what worked, and what they would do differently. In fast-moving fields like data, real stories often give more clarity than any roadmap or checklist.
Today, I’m excited to sit down with Nizra Nilufer, who has built a thoughtful and inspiring path into Data Engineering. Nizra’s first role in Australia was as a Test Analyst, where she developed a strong foundation in understanding systems, requirements, and quality. She then moved into an Application Support Developer role, working closely with SQL and .NET while supporting live production systems across multiple teams.
Through this experience, Nizra realised that much of what she was already doing sat very close to data engineering. That insight became the catalyst for her transition. Today, she works as a Data Engineer at Experian1, where she is part of a greenfield project involving a large-scale data migration from Azure to AWS and a ground-up rebuild of a data product.
Alongside her role, Nizra is also building DataGuild2, a platform focused on production-grade data engineering projects for people who want real-world experience, not just demos. In this conversation, she shares her journey, the lessons she learned along the way, and the advice she would give to anyone considering a similar path.
You can read/watch Brian’s journey from Data Analyst to Data Engineer here:
If you’re new to data engineering or preparing for a role in the field, I highly recommend checking these out:
How to Succeed in Data Engineering Interviews
How to Succeed in Data Engineering: Interviews, Careers, and Market Realities
Let’s start!
Interview Questions
Transition to Data Engineering
Why did you shift to Data Engineering? Was there a specific moment or project that made you say, “I want to be a Data Engineer”?
In my role as Application Support Developer, I spent a lot of my time troubleshooting issues after something had already gone wrong. While the experience was valuable, I realised I enjoyed the parts of the job where I was designing solutions more than fixing defects. So I wanted to get into a role that is more build-focused. I also genuinely enjoyed working with SQL and data, so it felt only natural to make the move into data engineering.
Misconceptions and Early Thoughts
What misconceptions did you have about Data Engineering before switching?
I thought data engineering was very similar to database administration, mainly about managing databases and writing queries. But after learning more about the role, I realised a lot of the work actually lies outside the database. It’s really about managing the end-to-end flow of data across systems, not just maintaining where data is stored.
How did you find your first Data Engineering role? Did you face any challenges during the process? How are you working on those challenges? What part of Data Engineering was the hardest for you at the start?
My first data engineering role actually came through a referral. The interview process itself was fairly smooth and quick, mainly because I had spent many months preparing. I had worked hard on my fundamentals, my projects, and my ability to explain what I had built.
The more challenging part was the job-hunting phase. My resume didn’t have the title “Data Engineer”, and at the same time, I wasn’t really entry-level, as I had plenty of industry experience with software systems dealing with data. That made positioning myself a bit tricky. To work around that, I focused heavily on building side projects, creating a portfolio that showcased my skills and highlighting my qualifications.
I also started sharing what I was building and learning on LinkedIn, which helped me build visibility and confidence.
The hardest part at the start was dealing with the many tutorials and courses without a clear path. Everything seemed useful, but there was no simple roadmap.
I eventually had to build my own learning path. Building my own projects was also challenging. Learning theory is one thing, but applying it in real projects is a completely different experience. And at the time, I felt there weren’t enough realistic data engineering projects available to practise those skills properly.
Looking back, what would you have done differently in your early days?
Looking back, I think I would have focused on building projects earlier before sitting my Azure Data Engineer Associate exam. At the time, I was in a hurry to complete the certification because my leave was ending, and I didn’t want to be studying for an exam after I had already started work. So the timing made sense then. But now I realise that hands-on project experience would have accelerated my learning even more.
Certifications are valuable, but real projects help everything click much faster. What I would do differently is build projects first, then take the exam.
Learning, Skills, and Resources
What was your learning strategy? Did you follow a structured roadmap or learn through real problems at your previous role?
I followed a structured roadmap, but it was one I created myself. I built it based on advice from experienced data engineers on YouTube and by looking at real job descriptions. I started by reading Joe Reis ’ book Fundamentals of Data Engineering4, then focused on SQL and Python, moved into big data concepts and Azure, built demo projects from YouTube, and eventually focused on building my own projects.
Before the shift, what skills did you already have, and what skills did you need to acquire? How did you learn them, through certifications, tools, projects, mentorship, or other methods?
Before making the shift:
I already had a strong foundation in SQL and good exposure to object-oriented programming languages. I had also worked with tools like SQL Server, SSIS and SSRS, so I was comfortable around data and reporting systems.
What I needed to acquire:
were Python, Pandas and Spark. I mainly learned those through structured courses, combined with hands-on practice.
What kind of personal projects did you work on and why?
I focused on building both batch and streaming data pipelines, mainly using Azure, Databricks, and Azure Stream Analytics. I chose them so that I could experience both types of workloads, batch for large historical processing and streaming for real-time data.
How did your previous roles help you in making the shift to Data Engineering?
My experience with SQL, databases, SSIS, SSRS and object-oriented programming languages gave me a strong technical foundation. I was very familiar with data structures and data flows.
My App Support Developer role helped me understand how applications interact with data in real production environments and how small data issues can have big business impacts. The combination of database knowledge and real production system exposure helped me get started on my shift to Data Engineering.
Did AI or any modern tools play a role in making your transition smoother? If yes, how?
I didn’t use AI much at the very beginning of my transition, but I started using it more in the later projects I built.
I mainly used tools like ChatGPT and DeepSeek to help generate scripts, explore transformation logic, and understand Azure Data Factory workflows. They also helped me with:
interview preparation
revising concepts
and practising how to explain my projects properly.
It reduced friction and helped me focus on the bigger picture rather than getting stuck with syntax. But I still relied heavily on fundamentals to judge whether what AI suggested actually makes sense. It can tend to hallucinate at times.
Advice for Others
What advice do you have for people who want to change roles or are in the process of transitioning?
My first piece of advice would be:
to really understand whether the role suits your personality. A lot of data engineering work happens in the background, but at the same time, you play a key role in translating business language into technical solutions and technical constraints back into business terms.
So it’s both technical and collaborative. And once you’ve decided it’s the right path for you, accept that it will take consistent time and effort to break in.
Transitioning won’t happen overnight, and having a perseverant mindset is important. I would also say that you don’t rush the process. Building new skills takes time. But the longer you stick with it, the faster your learning becomes. That’s not just personal experience; that’s real research on how people learn and how exceptional performers are developed.
So have a solid pathway, stick to it, and do a little bit every day. If you stay consistent, the progress compounds in a massive way.
Are there specific skills or concepts you recommend learning first?
a) Solidify SQL and Python first, because those two skills are the foundation of almost everything in data engineering.
b) Then learn big data fundamentals, things like distributed processing, batch vs streaming, and how data systems scale. I would also suggest reading the book Fundamentals of Data Engineering, as it really helps with understanding the role, responsibilities and big-picture thinking behind data engineering.
c) And only after that, pick a cloud provider and start learning the tools.
Are there common mistakes to avoid or habits to build for beginners?
Yes, the biggest mistake is trying to rush the process.
We all want results quickly, but meaningful growth takes time. It’s important to give yourself that time and not overwhelm yourself. The most important habit I’d recommend is simply showing up every day, even if it’s just for 30 minutes.
Consistency matters more than intensity.
You can take breaks, but try to keep learning or building something every day so you don’t lose momentum. We’re creatures of habit. If you make learning and building a habit, it will eventually take you where you want to go. And finally, never compare yourself to others. We all bring different experiences, backgrounds, and strengths. Data engineering is a very broad field, and every experience you have adds value differently.
Are there communities, forums, or mentors that you recommend beginners engage with? Can you also talk about the work you have done?
Yes, there are a few people and communities I’d definitely recommend:
On YouTube, I really like Dharshil Parmar5 and Sumit Mittal6. Their project-based videos are practical and help turn theory into real skills.
I also highly recommend Zach Wilson. His content is clear, detailed, and great for both beginners and advanced learners.
If you’re in Australia, follow DataEngBytes7 on LinkedIn8. They run data engineering conferences and create great opportunities to connect with others and stay up to date.
Personally, building and documenting my own projects on my portfolio website has been one of the most valuable ways I’ve learned.
If you could give your past self one piece of advice before switching roles, what would it be?
It would be to utilise AI to create a personalised learning pathway that leverages my existing skills and experience. I spent a lot of time building my own roadmap and searching for the right courses. While that process taught me a lot, I think I could have saved time and reduced confusion by using AI to help structure my learning path more efficiently.
Current Role, Stakeholders, Tools, and Technologies
What technologies and tools are you currently working with in your current role? Could you give an example of a workflow or project where a specific tool made a difference?
In my current role, I work with:
Azure SQL Managed Instance
Azure Data Factory
SSIS packages
ADLS Gen2
AWS S3 and AWS Aurora
We’re on a greenfield project to:
Migrate databases from Azure SQL Managed Instance to AWS Aurora
Redesign the data architecture from the ground up
Right now:
Many transformations still live in legacy SSIS packages
These are orchestrated using Azure Data Factory
As part of the migration, we are:
Moving platforms
Modernising the overall architecture
We are evaluating tools like Dagster9 Orchestration and dbt10 for Orchestration:
Even though we haven’t implemented them yet, this process has already changed how we think about Pipeline design, Modularity, Testing and Lineage
Who are your key stakeholders, and how do you collaborate with them (e.g., data scientists, analysts, product teams)?
I work closely with product, software developers, SRE, testers, data scientists, solution architects, and cloud engineers.
We collaborate through daily scrums, fortnightly demos, and ad-hoc meetings for stakeholders who don’t attend the daily, to keep everyone aligned and ensure our data solutions meet real business needs.
How do you use AI in your current role, if at all?
I use AI regularly in my current role to speed things up.
Read and explain unfamiliar code
Understand legacy logic
Brainstorm solutions
Generate scripts
Explore different ways to solve a problem
It’s especially helpful when working with older systems and complex logic.
However, whenever AI suggests or builds something, I always review and validate it myself.
Future of Data Engineering and AI
What’s your view on the future of Data Engineering in the age of AI? How do you think the role will evolve in the next 2–5 years?
I think AI will play a huge role in the future of data engineering, but I don’t believe it will replace data engineers. AI relies entirely on accurate, well-structured, and reliable data, and that’s exactly what data engineers are responsible for.
Where AI will really make a difference is in speeding up parts of the workflow, such as
writing transformation logic,
generating tests,
and helping debug pipeline failures.
Tasks that used to take hours will take minutes. I also think future data platforms will have AI agents embedded into them, offering things like chat-based access to data, intelligent monitoring, automatic pipeline fixes, and proactive recommendations. Because of that, the role will evolve.
Data engineers will spend less time on repetitive implementation and more time on system design, data quality, governance, and architecture. At the same time, they’ll need to adapt and learn how to work with AI effectively to stay ahead.
What skills do you think will be essential for Data Engineers to thrive in the AI era?
I think the fundamentals will always be essential. Data engineers still need strong skills in SQL, Python, and data modelling. Without those, it’s very hard to judge data quality or trust what any tool produces.
But in the AI era, it’s equally important to know how to work with AI effectively. That includes prompt-engineering, using AI as a co-pilot rather than a replacement, and learning how to automate repetitive workflows with AI.
I also think understanding how to integrate AI agents into data engineering systems will become important, whether that’s for monitoring pipelines or helping with troubleshooting.
So for me, the key skills are:
Strong fundamentals
Systems thinking
And the ability to work with AI to amplify your impact
Conclusion
Nizra’s story is a powerful reminder that careers are rarely linear and that growth often comes from recognising the value in what you already know and building forward from there. Her journey shows that transitioning into data engineering is not about starting from zero, but about connecting the dots between your existing skills and where you want to go. With clarity, consistency, and a willingness to build in the open, what once feels overwhelming becomes achievable. Whether you are just beginning or already partway through your transition, the message is simple: take ownership of your path, stay patient with the process, and keep showing up. Over time, those small, steady steps compound into a career you can be proud of.
Want to learn more about Nizra?
DataGuild (for learners looking for hands-on, real-world data engineering projects):
Feel free to reach out to her if you’re interested in collaborating, and explore DataGuild if you’re looking to learn through hands-on, real-world data engineering projects.
We value your feedback
If you have any feedback, suggestions, or additional topics you’d like us to cover, please share them with us. We’d love to hear from you!
Refer
3 friends: a 1-month free.
10 friends: a 3-month free.
25 friends: a 6-month free subscription.
Our way of saying thanks for helping grow the P2I community!
https://www.experian.com.au/
https://stan.store/DataGuild
https://pipeline2insights.substack.com/t/interview-preperation
https://www.amazon.com.au/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302/ref=asc_df_1098108302?mcid=5718e8b5a90339628795b7a7f9504749&tag=googleshopdsk-22&linkCode=df0&hvadid=712272731659&hvpos=&hvnetw=g&hvrand=13285504488312079700&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9071335&hvtargid=pla-1643937444435&psc=1&gad_source=1
https://www.youtube.com/@DarshilParmar
https://www.youtube.com/results?search_query=sumit+mittal
https://dataengbytes.com/
https://www.linkedin.com/company/dataengbytes/
https://dagster.io/
https://www.getdbt.com/product/what-is-dbt
