Senior Data Engineer
Lead the design and delivery of reliable data platforms for leading brands, mentor colleagues and turn complex data into practical solutions that make a difference.
Who we are
Optima Partners is a VC-backed, scaling AI and data consultancy headquartered in Edinburgh. We bring together expertise in strategy, design, data science and engineering to help leading brands solve real business challenges.
We turn data into better customer experiences, smarter decisions and lasting business value.
Our clients include Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Eon Next, OVO and GSK. We work alongside their teams to deliver practical solutions and share the skills to build on them.
Within our Data practice, you’ll work in mixed-capability pods alongside data engineers, analysts, data scientists and ML/AI engineers. You’ll bring technical depth, guide colleagues and turn data into practical solutions, clear insights and better customer experiences.
The opportunity
As a Senior Data Engineer within our Data practice, reporting to a Technical Lead, you will lead the design, build, testing and maintenance of reliable, scalable data pipelines and platforms for clients across diverse sectors. Working in multi-disciplinary teams, you’ll take ownership of technical delivery and help shape solutions from requirements through to production.
This role suits an experienced data engineer who combines strong hands-on delivery with sound technical judgement, client-facing confidence and a commitment to developing others.
You will work alongside engineers, data scientists, architects and product specialists to make pragmatic design decisions, resolve complex challenges and improve engineering standards. You’ll mentor colleagues, share knowledge and continue to develop your technical and consulting expertise.
What you will be doing
Lead the design, build, testing and operation of scalable data pipelines using platforms such as Databricks, taking ownership of delivery quality and production readiness.
Design and develop data warehouses, data lakes and lakehouse solutions, making pragmatic architecture and data modelling decisions with architects and technical leads.
Build and optimise robust ETL/ELT workflows, embedding data quality checks, monitoring and controls to deliver reliable, accessible data.
Work directly with clients and multi-disciplinary colleagues to translate business requirements into technical designs, estimate work and communicate trade-offs, risks and progress.
Deliver well-modelled, documented datasets that support analytics, dashboards and visualisations in tools such as Power BI, Looker and QlikView, collaborating with analysts to meet reporting needs.
Provide hands-on technical leadership through design and code reviews, mentoring and knowledge sharing, helping colleagues adopt consistent engineering practices.
Troubleshoot complex production issues and improve performance, reliability and cost efficiency through automation, testing and CI/CD. Maintain clear documentation and support effective client handover.
What skills we would like you to have
Strong hands-on Python and SQL skills, with experience writing maintainable, tested code for production data workloads.
Practical experience building and optimising distributed data processing workloads using Spark and Databricks, or comparable platforms.
Experience with Delta Lake or comparable lakehouse technologies, including notebooks, workflows and reliable production deployment.
Experience delivering and operating data solutions on at least one major cloud platform, such as AWS, Azure or GCP, including appropriate access and security controls.
Strong understanding of relational databases, advanced SQL and query optimisation, with the ability to diagnose and resolve performance bottlenecks.
Experience implementing orchestration, scheduling, dependencies and failure handling using tools such as Airflow, Apache NiFi, Talend or platform-native workflows.
Strong data warehousing, modelling and schema design skills, with experience creating curated datasets for analytics and an understanding of how tools such as Power BI, QlikView and Looker consume data.
Experience using Git, automated testing and CI/CD to support collaborative development, controlled releases and maintainable data solutions.
Proven experience delivering production data engineering solutions end to end, including technical design, implementation, deployment and ongoing support.
Experience working with clients or business stakeholders to clarify requirements, explain technical choices and manage delivery expectations, alongside mentoring engineers and reviewing their work.
Experience delivering iteratively within Agile or Scrum teams, breaking down work, estimating effort and identifying dependencies and technical risks.
What you will bring
Strong analytical and problem-solving skills, with sound judgement when balancing technical quality, delivery timelines and client priorities.
An inquisitive mindset and a commitment to continuous improvement, with the ability to evaluate new approaches and share practical learning across the Data practice.
Clear, confident communication, with the ability to explain complex technical ideas, challenge assumptions constructively and build trusted client relationships.
A collaborative, accountable approach, with the initiative to lead technical work, support colleagues’ development and give and receive constructive feedback.
As technology partners with leading companies such as AWS and Databricks, we’ll invest in your training and development, alongside hands-on project experience with some of the UK’s leading organisations.
What we offer
Competitive base salary.
Inclusion in our annual discretionary bonus plan with an on-target performance bonus of up to 15%.
Up to 37 days holiday inclusive of personal and public allocations. Seven are fixed days for Christmas and New Year, and 30 are floating days taken at your discretion, subject to client scheduling and line manager approval.
Private medical insurance, single cover.
Group life and income protection insurance.
Salary sacrifice pension scheme after three months’ employment.
Access to over 1,000 perks and discounts via an employee discount portal.
Dedicated development time plus tools and funding to support personal and professional development.
- Department
- Engineering
- Role
- Senior Data Engineer
- Location
- Holborn Circus, London
- Remote status
- Hybrid