SC Cleared Senior Data Engineer

Cabinet Office - Digital and Technology

Senior Data Engineer (SC cleared)

Location: Bristol/London - on an as and when needed basis, potentially 2 days per week at times

Contract: End of financial Year

Role Description

You will be joining a team that develops and manages a cross-government data platform.

The platform lets departments store, analyse and distribute data, primarily for insight generation and data sharing. You will be working on data pipelines, to ingest, transform and distribute data needed by different stakeholders. You will support data analysts and data scientists who work to generate insights from critical government data.

Skills Required
. Proficient in Python and PySpark for building data pipelines (ETL) and processing data
. Experience with AWS cloud platform including AWS services: Glue, S3, Athena, IAM and Apatchi Iceberg
. Awareness of storage layer technology such as Delta Lake and Parquet
. Awareness of data visualisation technologies such as Tableau and Plotly
. Strong SQL skills and experience with Database Technology
. Experience with Data Lakehouse concepts such as star schema design, change over time, partitioning and snapshots
. Proficiency with CI/CD pipelines, pull requests, automated tests, scripted deployment and infrastructure as code
. Testing experience: unit testing, integration testing, performance testing, user acceptance testing - automated where time and cost effective
. Code reviews, working in the open, proactively pulling work through
. Give consideration to observability, whole life costs and sustainability
. Able to take security requirements into the design. Authentication and Authorisation. Field level, row level and data set level access control. Encryption at rest and in flight.
. You will want to work in a collaborative team following agile practices such as sprints, standups and definition of done.

Please be aware that this role can only be worked within the UK and not Overseas.

Disability Confident

As a member of the Disability Confident Scheme, The Cabinet Office guarantees to interview all candidates who have a disability and who meet all the essential criteria for the vacancy. In cases where we have a high volume of candidates who have a disability who meet all the essential criteria, we will interview the best candidates from within that group. This scheme encourages candidates with a disability and/or neurodivergence to apply.

Armed Forces Covenant

The Cabinet Office guarantees to interview veterans or spouses/partners of military personnel who meet all the essential criteria for the vacancy. In cases where we have a high volume of ex-military candidates/military spouses or partners, who meet all of the essential criteria, we will interview the best candidates from within that group.

In applying for this role, you acknowledge the following "this role falls in scope of the Off Payroll Working in the Public Sector legislation. Any rates of payment quoted will reflect the gross rate per day for the assignment and will be subject to appropriate taxes and statutory costs. As such the payment to the intermediary and your income resulting from this contract will be different".