How to Gather Requirements Effectively as a Data Engineer

A practical guide for data engineers and related roles to apply effective requirements gathering in everyday work

In any profession, one of the first skills we develop is the ability to gather and interpret requirements. Whether it’s a small task, a project, or a large-scale initiative, understanding what needs to be done and how it connects to the bigger picture is fundamental, and this applies well beyond data engineering.

Unlike school or university, where goals, requirements, and grading criteria are handed to us, the real world rarely provides a checklist. Instead, we need to uncover the objectives, ask the right questions, and bridge the gap between technical and non-technical perspectives.

A successful data engineer zooms out to see the big picture and focuses on delivering outsized value for the business. This skill is largely shaped by experience. The more projects we take on, the more stakeholders we engage with, each bringing different roles, expectations, and communication styles, the better we become at translating unclear needs into actionable outcomes that align with business goals. Mastering this process not only helps us deliver successful results but also positions us for senior roles, where seeing the bigger picture becomes just as important as solving the immediate problem.

In this post, we provide a practical guide for data engineers and similar roles to understand this concept and apply it in everyday work.

We will cover :

• Different Types of Requirements

• Key Elements of Requirement Gathering

• Tailoring Conversations with Stakeholders

• How to Initiate Conversations by Role

• Trade-Offs in Requirements Gathering

• Thinking Like a Data Engineer Framework

• Best Practices for Requirement Gathering

The post is based on insights from

Joe Reis

’ Data Engineering Professional Course, combined with our experience.

If you’re new to data engineering, you can explore:

• Data Engineering Lifecycle Series¹

• Data Engineering Interview Preparation Series

---

Different Types of Requirements

The best way to ensure we deliver real value through data systems is first to understand the requirements and then translate them into effective solutions. Before writing code or spinning up cloud resources, it’s important to pause and clarify what “requirements” actually mean at different levels:

1. Business Requirements

These capture the high-level goals of the organisation.

They answer the question: What does success look like for the business as a whole?
Examples:

• Increasing revenue.

• Expanding the customer base.

• Improving customer experience.

2. Stakeholder Requirements

These reflect the needs of specific teams or individuals within the business. Each stakeholder contributes to achieving the broader business objectives, but has their own priorities.

For example, marketing teams may need timely customer insights, while finance teams may require accurate cost reporting.

In a data engineering context, stakeholders often rely on robust data systems, reliable access to data, and the right tools to do their jobs effectively.

3. System Requirements

System requirements describe what the data system must do to satisfy both business and stakeholder needs. They are usually divided into two categories:

• Functional requirements (the what)
  Define the specific tasks or behaviours the system must perform.
  Example: A fraud detection system must immediately flag and report suspicious transactions.

• Non-functional requirements (the how)
  Define the qualities and constraints that shape how the system operates.
  Example: A streaming pipeline must scale to handle data from 10,000 concurrent users while maintaining low latency and compliance with security standards.

---

Key Elements of Requirements Gathering

At a high level, any requirements-gathering conversation should touch on four essential elements:

1. Understand existing systems and solutions

Start by exploring what tools, reports, or processes are already in place.
Example: Technicians’ overtime reports are available as an SSRS report, but the data is shown as raw numbers without any context.

2. Identify pain points and limitations

Dig into what isn’t working well with the existing setup.
Example: The report requires too much effort to interpret, so most people avoid using it.

3. Clarify intended actions and outcomes

Ask what stakeholders actually want to do with the data. This links requirements to real business decisions.
Example: We want to understand why technicians work overtime, how much it costs the business, and whether it impacts profitability or leads to losses.

Note: Always repeat back what you’ve heard to confirm your understanding and avoid misalignment.

4. Identify Other Stakeholders

If gaps remain, ask who else you should involve to fill in the picture.
Example: The business analyst is familiar with the related tables and can provide more detailed context on the data.

---

Tailoring Conversations with Stakeholders

The requirement gathering process always begins with conversations, and how we approach those discussions depends heavily on the stakeholder’s role and technical background.

Here are some practical tips shared by

Sol Rashidi

for data engineers on how technical to be when speaking with stakeholders:

1. Know our audience
   Identify whether the stakeholder is functional, techno-functional, or purely technical. Match our level of detail and language to their background.

2. Start with business language
   With non-technical leaders, avoid jargon. Emphasise outcomes, value, and business impact rather than system details.

3. Use light technical detail for techno-functional leaders
   These stakeholders understand some technical concepts but don’t live in the details. Keep explanations simple, clear, and focused on how the technology supports business goals.

4. Go deep with technical leaders
   With CTOs, technical VPs, or data platform owners, don’t shy away from specifics. They expect depth and expertise.

5. Adapt to the company context
   In startups, leaders may wear multiple hats and lean more technical. In large enterprises, leaders are often more functional; adjust your style accordingly.

6. Consider titles and reporting lines
   A stakeholder reporting into a CIO, CTO, or CDO may be more open to technical discussions, but always assess their personal background first.

7. Test their knowledge early
   Ask a few probing questions to gauge how familiar they are with technical concepts. Use this to calibrate your communication style.

8. Stay Flexible
   Begin with functional framing, but be ready to pivot if the conversation turns technical. Great communicators adjust in real time.

---

How to Initiate Conversations by Role

Before diving into role-specific guidance, it’s helpful to understand the purpose of these conversations. Each stakeholder comes with unique priorities, responsibilities, and perspectives. The way we approach a CEO will differ from how we engage a software engineer or a data scientist.

1. CEO / Executives

Ideally, start at the top by speaking with leadership. At smaller companies, we may have direct access to the CEO; in larger organisations, we might only reach VPs or directors. Either way, take every opportunity to understand the company’s strategic goals.

Tips for engaging executives:

1. Always tie data work to business goals. Frame everything in terms of outcomes: revenue growth, cost reduction, customer satisfaction.

2. Position ourselves as a partner, not just support. Show that data engineering is part of strategy, not just a back-office function.

3. Ask high-impact questions.

   • Which markets, products, or customer segments are most critical to focus on?

   • What are the company’s top priorities for the next 1–3 years?

We suggest requesting a 15-minute catch-up with executives or their direct reports. Even a short conversation can give valuable direction.

2. Software Engineers

Software engineers care most about operational stability. Their top priority is not to break production. Show that we understand their constraints and that we’re here to make their lives easier, not harder.

1. Respect production constraints:

   • Explicitly acknowledge: “I know direct prod access is risky”.

   • Suggest safe alternatives: read replicas, message queues, APIs.

2. Establish reliable communication:

   • Define clear SLAs (e.g., “data may be delayed, but alerts will go out immediately”).

   • Ask about schema change processes early: “What’s your process for schema changes”?

   • Maintain a change log, schema registry, or shared Slack/Teams channel.

   • Set up recurring catch-ups (even monthly) to stay aligned.

3. Build trust:

   • Provide feedback on which data is most useful for downstream teams.

   • Emphasise that data quality should be addressed at the source system, and software engineers are key partners in achieving this.

   • Create a feedback loop using dashboards, alerts, or visualisations to show how upstream issues impact downstream stakeholders and the business.

Keen to learn more about data quality? Check out our Data Quality Series².

3. Data Scientists

Data scientists thrive on usable, high-quality, and accessible data. The best way to support them is by clarifying needs around quality, freshness, and delivery.

Key areas to cover with data scientists:

• Data quality and format

  • What checks matter most (missing values, duplicates, outliers)?

  • What formats are needed (structured tables, JSON, Parquet, images, text)?

  • What freshness is acceptable (hourly, daily, weekly)?

• Collaboration and delivery

  • Clarify workflow needs: Do they prefer ad-hoc queries, or would scheduled pipelines add more value?

  • Use visuals to align: Mockups, diagrams, and proof-of-concepts help surface assumptions, spark useful debates, and get everyone on the same page more quickly.

• Success criteria

  • Is success defined as a clean dataset ready for analysis?

  • A feature store for ML models?

  • A reproducible pipeline that supports experimentation and scaling?

As data engineers, our job is to simplify stakeholders’ work so they can focus on their core responsibilities without worrying about data problems. By adapting our approach to each stakeholder, we not only gather clearer requirements but also strengthen trust and relationships across the organisation..

---

Trade-Offs in Requirements Gathering

Deeplearning.ai Data Engineering Professional

Gathering requirements isn’t just about capturing what stakeholders want; it’s also about balancing those needs against what’s realistically achievable within time, budget, and scope. These three constraints are often referred to as the Iron Triangle³:

• Scope: The features and functionality included.

• Timeline: How quickly the system can be delivered.

• Cost: The budget and resources available.

These factors are interdependent:

• Expanding the scope usually means more time or budget.

• Accelerating timelines may require more resources or fewer features.

• Strict budgets often limit what can be delivered.

The most important step is having open, honest conversations with stakeholders about priorities:

• Is speed more critical than cost?

• Is the budget fixed, meaning the scope must adjust?

• Or is full functionality non-negotiable, even if delivery takes longer?

Trade-offs are inevitable, but they can be managed effectively by applying solid data architecture and engineering practices:

• Build flexible systems that can evolve over time.

• Make reversible decisions to reduce risk.

• Align closely with business goals so trade-offs support what matters most.

If you want to learn more about data architecture, you might find this post helpful.

What is Data Architecture and why Data Engineers should consider it

---

Thinking Like a Data Engineer Framework

Deeplearning.ai Data Engineering Professional

In data engineering, our role spans the entire process, from requirements gathering through to full system implementation.

Joe Reis

introduced a framework called thinking like a data engineer. It’s a structured way of approaching any project, regardless of scale.

Stage 1: Clarify Business Goals and Stakeholder Needs
The first step is uncovering the business objectives driving the project. Then, identify stakeholders, understand their needs, and connect those needs back to business goals.

Stage 2: Define system requirements
Here, we document both:

• Functional requirements

• Non-functional requirements

After writing down the requirements, check them with stakeholders to make sure everyone agrees and the project stays on track.

Stage 3: Select Tools and Technologies
Next, it’s time to design the solution. Identify which tools and technologies could meet the requirements, then evaluate trade-offs through a cost-benefit lens. Consider factors like:

• Licensing and subscription fees.

• Cloud resource costs.

• Development and maintenance overhead.

Build a prototype to validate assumptions before committing to a full implementation. Share it with stakeholders, gather feedback, and refine until confident in your design.

For a detailed guide on selecting the right tools, explore this post:

7 key factors every Data Engineer Should Consider When Choosing Tools

Stage 4: Build, Deploy, and Evolve
Once our design is validated, it’s time to build and deploy the system. But the work doesn’t stop there; continuous monitoring and iteration are essential. Data systems are living systems: they evolve as business needs change or as new technologies create opportunities for improvement.

---

Best Practices for Requirement Gathering

1. Take an Iterative, Incremental Approach

• Don’t try to capture every requirement upfront.

• Deliver MVPs or prototypes early, then refine based on feedback.

• Use small examples or sample data to validate understanding before full integration.

2. Focus on Outcomes, Not Inputs

Anchor discussions on business objectives (your “elevator pitch”), not just specific data points.

• Ensure business logic is correct—but don’t over-invest in cleaning imperfect source data too early.

3. Strengthen Collaboration and Communication

• Secure buy-in from PMs and stakeholders on requirements and project scope.

• Involve subject matter experts and teams with domain knowledge.

• Make end-users part of testing cycles and encourage their feedback on prototypes.

4. Document and Visualise Clearly

• Capture what’s in-scope vs. out-of-scope in documentation.

• Use diagrams and flowcharts to illustrate data flow or model design.

• Require approvals for key artifacts to ensure accountability.

5. Set Boundaries and Follow Process

• Funnel extra requests through change management, not ad-hoc asks.

• Prioritise “must-haves” before “nice-to-haves”.

6. Shift Your Mindset

• Accept that requirements will never be perfect.

• Focus on delivering valuable outputs, not pleasing everyone.

• Treat requirement gathering as a collaboration, not a confrontation.

7. Manage Changing Requirements

• Establish a process for:

  • Requesting changes or new features.

  • Prioritising with stakeholders.

  • Communicating delivery timelines.

• Educate end-users on this process so expectations are clear.

---

Conclusion

Gathering requirements is one of the most important skills for data engineers. Tools and technologies may change, but turning unclear needs into clear, actionable plans is always valuable.

In the real world, we rarely get a step-by-step guide. We need to ask the right questions, understand both technical and business perspectives, and balance time, cost, and scope. Bridging the gap between tech and business adds value and creates growth opportunities. Understanding why a project is needed can help suggest smarter, more efficient, and cost-effective solutions.

The more projects you work on and the more people you collaborate with, the better you get at seeing the big picture while handling the details. This skill helps you make a real impact and prepares you for senior roles.

We value your feedback

If you have any feedback, suggestions, or additional topics you’d like us to cover, please share them with us. We’d love to hear from you!

1. https://pipeline2insights.substack.com/t/data-engineering-life-cycle

2. https://pipeline2insights.substack.com/t/data-quality

3. https://www.coursera.org/articles/project-management-triangle