Data Engineer

Role: Data Engineer

Location: UK Remote

Duration: 6 Months

Day rate: £470 inside IR35

Active SC clearance required

Role Description:

• Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc.
• Strong understanding of complex JSON manipulation.
• Experience working with Data Pipelines using a custom Python/PySpark framework.
• Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
• Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc., including experience of handling sensitive datasets.
• Strong problem-solving and analytical skills, particularly able to demonstrate these intuitively.

Required skills:

• Languages / Frameworks: JSON, YAML, Python (as a programming language, not just able to write basic scripts; Pydantic experience would be a bonus), SQL, PySpark, Delta Lake, Bash (both CLI usage and scripting), Git, Markdown, Scala (bonus, not compulsory), Azure SQL Server as a HIVE Metastore (bonus).

Technologies: Azure Databricks, Apache Spark, Delta Tables, Data processing with Python, PowerBI (Integration / Data Ingestion), JIRA.