Senior Data Engineer

This job is brought to you by Jobs/Redefined, the UK's leading over-50s age inclusive jobs board.

Senior Data Engineer

Department: Tech Hub

Employment Type: Permanent

Location: London

Reporting To: Lead Data Engineer

Description

Contract type: Permanent
Hours: Full Time, 37.5 hours per week

Salary: circa GBP 70,000 depending on experience

Location: Canary Wharf

WFH policy: Employees are required to attend the office 2 days/week

Flexible working: Variety of flexible work patterns subject to line manager discretion e.g. Compressed 9-day fortnight.

Reports to: Lead Data Engineer
Deadline Note: We reserve the right to close the advert before the advertised deadline if there are a high volume of applications.

Role Summary:The Senior data engineer role will have a specific focus on data and integration, ensuring that they drive single version of the truth by developing and maintaining reliable solutions to store and manage key data.

They will take ownership for the design, development, and maintenance of LCCCs databases and data warehouse, as well as ensure all data systems conform to the data architecture and strategy expectations.

The Senior Data Engineer will drive better data governance through the creation and embedding of principles and processes. They will also build and maintain key data governance and management artefacts such as the data dictionary and KPI catalogue.

Key Responsibilities

• Own, design, develop and maintain LCCCs databases and DataWarehouse ensuring reliable data across the organisation
• Source, model and provide reliable data in a form that is ready for report / dashboard /forecast model building
• Drive better data governance through the creation and embedding of principles and processes
• Build and maintain key data governance and management artefacts e.g., Flow Diagram, Data Dictionary, KPI Catalogue, data quality standards, metadata management, and data lineage documentation
• Define overall approach and data flow for Extract, Transform and Load; and application of this for a given deployment/project
• Identify patterns, anomalies, and structure of data in preparation for Extract Transformation and Load
• Design and implement data models within a data warehouse based on business requirements
• Write ETL data validation and data reconciliation queries
• Define high level and low-level design of a data warehouse to ensure robustness e.g., restorability, traceability, ease to support, etc.
• Define, embed, and drive master data management approaches, principles, and processes
• Identify data quality issues through data profiling, analysis, and stakeholder engagement
• Design and maintain the Microsoft Azure modern DataWarehouse architecture, implementation, administration, and support
• Stay updated with emerging technologies and best practices in data engineering and Azure
• Ensuring adherence to best practices for cloud security and governance
• Propose and implement innovative solutions to enhance data processing efficiency and scalability
• Collaborate all teams and business stakeholders to gather requirements and deliver data-driven solutions
• Mentor data engineers, promoting best practices in coding, design, and operational processes

Skills Knowledge and Expertise

Essential:

• At least 4 years' experience in Microsoft Azure Datawarehouse architecture as well as Data Warehousing tools and database systems
• Minimum 3 years' experience with Python, preferably PySpark
• Experience of ETL and ELT processes, working with database architecture and business intelligence tools
• Experience with database administration
• Able to develop and optimise queries in SQL
• Experience in creating data pipelines using Azure Data Factory
• Experience with Azure: ADLS, Databricks, Stream Analytics, SQL DW, Synapse, Databricks, Azure Functions, Serverless Architecture, ARM Templates
• Experience in managing DevOps Pipelines
• Strong background in deployments
• Experience in writing PowerShell scripts
• Able to work with large datasets and extrapolate conclusions
• Strong analytical and critical thinking abilities

Highly desirable:

• Experience in building and maintaining reliable and scalable ETL on big data platforms as well as experience working with varied forms of data infrastructure inclusive of databases such as SQL, Synapse
• Data Engineering experience in Microsoft stack

Employee Benefits

As if contributing to and supporting work that makes life better for millions wasn't rewarding enough, we offer a full range of benefits too. Key benefits that may be available depending on the role include:

• 25 days' annual leave and bank holidays
• Recognition schemes allowing colleagues to say thanks
• Company contribution to your pension scheme
• Family friendly policies, including enhanced company maternity/paternity and shared parental benefits
• Employee assistance programme for free, confidential support for your professional or personal life, including financial management and family care
• Special leave such as study leave, sabbatical or public duties
• Three days paid leave a year for volunteering to support your local community
• Season ticket loan scheme to support your commute
• Access to "Work Perks" offering deals, discounts and cash back on your purchases
• Family savings on days out and English Heritage or gym discounts through our partners.