Reflecting on 2024: Milestones, Insights, and What’s Next for Pipeline to Insights
Celebrating a Year of Achievements, Community, and Learning
Hello everyone 👋🙂
As we approach the end of the year, we want to take a moment to thank each and every one of you for your incredible support throughout this journey. Since our start in October 2024, we've reached the amazing milestone of 3,000+ subscribers. This wouldn’t have been possible without your support.🙏
This year has been filled with more milestones and achievements that we’re excited to share with you. Here’s a look back at what Pipeline to Insights accomplished.
Our Key Highlights in 2024
Presentations
Accepted Research Paper2 at the 40th IEEE International Conference on Data Engineering in Utrecht, Netherlands.
Introduction to data load tool ( dlt) at the Data Engineering Meetup in Melbourne3 (hosted by Mantel Group and ThoughtWorks).
Volunteering
At DataEngBytes4, the premier independent and vendor-free Data Engineering conference in Australia and Oceania.
With La Trobe Industry Mentoring Program6, guiding and mentoring students.
Consultation
We kicked off our first consultation project with Flexlore, marking a new chapter in our journey.
We’ve started offering individual consultations to support professionals in their job search within the data domain.
Writing
We started our Substack journey by naming it Pipeline to Insights, designing our logo, and building a great community. Along the way, we shared our personal stories.
If you haven’t seen them yet, check out these stories!
Holiday Reads by Us
During the holiday season, we’d love to suggest you check some of our top posts.
Here are a few highlights:
The Data Engineering Lifecycle
We explored the data engineering lifecycle in detail, covering:
Ingestion
Transformation
Serving
Storage
Undercurrents ( Security)
Dive into this series here:
👉 Data Engineering Lifecycle Series7
SQL Optimisation
We're posting a SQL optimisation daily as part of our 100 Days of SQL series! So far, we've completed the first 51 days, with useful insights shared along the way. Here's the breakdown:
Days 1-35: Focused on PostgreSQL, exploring foundational optimisations and advanced techniques.
Days 36-70: Currently diving into DuckDB, uncovering its unique capabilities and performance benefits.
Days 71-100: We’ll transition to Spark SQL, tackling large-scale data challenges with efficient query practices.
Catch up on our recaps and summaries here:
👉 SQL Optimisation Series8
Data Engineering Interview Guide and Data Ingestion Series
About a month ago, we asked you what topics you’d like to see, and the response inspired two exciting new series:
The Data Engineering Interview Preparation Guide: Weekly insights to help you ace the data engineering interviews. See the series here:
👉 Data Engineering Interview Preparation Guide9
Data Ingestion Best Practices: Learn how to ingest data from various sources using tools like dlt effectively. Related posts;
To celebrate this journey and welcome 2025, we’re excited to offer our valued readers an exclusive end-of-year discount.
🎁 Here’s a 25% discount on our annual plan to invite everyone to join our learning journey in 2025!
What’s Next?
Looking ahead, we’re excited to bring you even more valuable content in 2025. Here’s what we’re planning:
High-Performance Data Engineering Series: Tips and techniques to optimise your performance as a Data Engineer.
Skills Every Data Engineer Should Master: Especially when collaborating with ML engineers.
Data Quality in the Age of AI: Core concepts, impacts, and strategies for improvements.
Beginner Series on Large Language Models: A deep dive into LLMs for aspiring data engineers.
Continuation of the 100 Days of SQL Optimisation Series: Building on our daily insights to complete this comprehensive journey.
Data Engineering Interview Preparation Series: Extending for 7 more months, packed with a plan specifically designed for interviews.
Once again, thank you for being part of this amazing community.
We look forward to continuing this journey together, learning, and growing as we explore the fascinating world of Data Engineering.
Happy holidays and here’s to an even brighter 2025!
Warm regards,
Pipeline To Insights Authors
Erfan & Hasan
https://www.aicamp.ai/
https://ieeexplore.ieee.org/abstract/document/10598161/
https://www.linkedin.com/posts/dlthub_introduction-to-dlt-simplifying-data-pipelines-activity-7264572159287595008-Vanl
https://dataengconf.com.au/
https://www.beerops.io/
https://mentoring.latrobe.edu.au/
https://pipeline2insights.substack.com/t/data-engineering-life-cycle
https://pipeline2insights.substack.com/t/sql-optimisation
https://pipeline2insights.substack.com/t/interview-preperation