Composing choices: How data engineering teams make decisions

As a data engineer, I’ve spent the past 15 years building knowledge, skills, and expertise in data, learning a trick or two along the way. However, my career didn’t start out in this industry. I actually have a degree in music, specialising in the electric guitar. But it’s not as unconventional a career path as you might think – if you can read music, an array of logical notations, you can certainly learn to code. Your coding language is your instrument, the platforms your amplifier, and the code itself are your notes. You can even add in your own creative flair.

At its surface, playing in rock bands does seem to be at odds with the perceived solitary role of a data engineer. We imagine data engineers sitting in a dark room headphones firmly pressed in, screens glowing, and endless lines of code filling the silence. But a career in digital and data is anything but isolated. Just like a band, the decisions we make shape outcomes, which rely on collaboration across team members and disciplines.

As consultants, we rely on the expertise and knowledge of our teams to bounce ideas off and help us make decisions. The vast array of decisions can span anything from data pipeline design and platform vendor considerations, to deciding which client needs to see each line of our code. So how exactly do we, as data engineers, make decisions?

1. Bring your stakeholders along for the ride

Our stakeholders and audience are at the heart of everything we do. Clear stakeholder engagement ensures the solutions we build are robust, purposeful, and aligned with real business needs. It’s also important to communicate clearly and effectively so our stakeholders can be invested in the process. By capturing functional and non-functional requirements proposed by stakeholders, we can collaborate with our technical experts to ensure we design or architect our best solution.

2. Artificial Intelligence augmenting rather than replacing

With the exponential boom of artificial intelligence (AI), gone are the days of memorising complex syntax structures and meticulously evaluating how to improve code block efficiency. Our peer support is still essential for seeking advice and best practice, but we can also leverage AI frameworks to accelerate code development and implement solutions efficiently which can significantly improve consistency and productivity. Embedding AI frameworks into Integrated Development Environments is incredibly useful for producing code quickly. However, it should not be used as a total replacement for good old-fashioned teamwork and peer review.

3. Using the right tools in the right context

If a client has a preferred technology stack, I always architect with this in mind. However, it’s important to be mindful of the benefits other technologies could offer that the client may not have considered. By clearly understanding the functional and non-functional requirements and applying a strong depth of knowledge of the client’s existing technology landscape, we can work as a team to best assess how to build the right engineering solution.

4. Agile vs waterfall: The great debate

As technologists, Agile is often our preferred way of working. However, there may be instances where a waterfall approach is more appropriate. I’ve experienced both approaches on projects. If timelines, scope, and functional requirements are all clear, waterfall may be the way to go. And whilst Agile is generally ‘best practice’, a waterfall approach worked very well for a recent nuclear client because they wanted the assurance that the deliverable would be met within a set timeframe. Data engineers should work closely with project managers to determine the best approach, and tailor the outputs accordingly.

5. Let your creativity flow

Data engineers will often work collaboratively with other engineers on products. Just like when playing in a band or orchestra, clear guidance around project responsibilities is essential. Afterall, you would not want a drummer playing the melody. Assigning tickets as part of sprint planning is an effective way to establish ownership and responsibility. This enables us to be creative – but within set boundaries. Utilising version controlling tools (such as Git) will provide senior data engineers with the ability to leverage creativity before pushing any changes into the main branch.

6. Data is a team sport

The best data engineers work together. We can constantly learn new trends, tools, practices, and techniques from those around us, keeping our minds open to new ideas, across different seniorities in an ever-evolving technology landscape. Working together with other disciplines is also key to both organisational resilience and innovation, and ensures we arrive at the most suitable, fit for purpose, and effective data solution for our clients

Just like playing in a band, data engineering is a collaborative craft. It brings together diverse disciplines, creative thinking and problem solving, while providing unique opportunities to leverage growing AI tools to accelerate tech solutions that are reliable and scalable. As a musician turned data engineer, I always aspire to find creative solutions to some of the most complex technology challenges – looking at new ways to make code efficient, trying different programming languages (#TeamPySpark), and experimenting with different platforms. That’s what keeps me motivated and excited every day.