Senior Data Engineer

Job description:

WHAT TO EXPECT

We are looking for a Senior Data Engineer to support the design, management, checking, and engineering of a diverse range of data using a variety of tools to achieve consistent data sharing across the organisation.

As a Senior Data Engineer, you will undertake Data Engineering for digital projects ensuring data is trusted, accurate and timely. You will be responsible for the structural elements of data, including data storage, data pipelines and interfacing with digital platforms, manage all data aspects related to data layers of the digital product, participate in data requirements modelling and testing and define the flow of data from input through to storage, including interfaces with analytics tools or end-user software.

Key Accountabilities and Responsibilities

* Design, create, test, and maintain data pipelines to support Digital projects
* Assemble complex datasets for a range of applications and teams across the business
* Identify, design, and implement process improvements in data pipelines. Automating manual processes, increase scalability, reduce run time, and optimise cost.
* Build the infrastructure required to support the running of data pipelines that are cost-effective, robust, and scalable
* Build analytic tools and solutions to support the business in data-driven decision making and actionable insights


WHAT YOU WILL NEED

You will be capable of designing, creating, testing, and maintaining data pipelines to support analytics projects. You will possess the ability to identify, design and implement process improvements in data pipelines, automate manual processes, increase scalability, reduce run time, and optimise cost.

You will possess a strong understanding and knowledge of how to resolve raw data questions using a storytelling approach to provide business clients with detailed, actionable solutions. Provide advisory service to business domain analysts and citizen data engineers to assist with technical work in agreed territories.

Knowledge, Skills and Experience

Essential -

* Previous experience working in a team in one of the following functions: data analytics, software or application development, database development
* Experience in designing data models and data architecture
* Experience in creating tests; unit testing, data quality tests, assumption testing
* Excellent problem solving and troubleshooting skills
* Previous experience and knowledge of modern programming languages and scripts
* Strong exposure to Python programming experience
* SQL development and experience in relational or no-SQL databases
* The ability to communicate complex data or technical concepts to non-technical internal clients
* Proven ability to lead small teams in technical assignments


Desirable -

* Previous experience in Google Cloud Platform, Kubernetes, GIT, Docker
* Experience with BigQuery and Airflow
* Previously worked in Agile project management environment using Scrum
* Process-oriented with great documentation skills
* Data exploration, visualisation in Tableau, Looker, Power BI or similar