Software Engineer - Azure Data & DevOps (SC Cleared)

Posted 14 days 6 hours ago by Scrumconnect Limited

Permanent
Not Specified
I.T. & Communications Jobs
London, United Kingdom
Job Description

London Arena, United Kingdom Posted on 12/03/2025

Scrumconnect Consulting is looking for a Software Engineer to work on a Strategic Data Platform with expertise in Azure Data Factory (ADF), Python, PySpark, Java, Terraform, and Azure DevOps. The ideal candidate will have experience in cloud-based data engineering, automation, and infrastructure provisioning while working in an Azure environment.

You will be responsible for developing scalable data pipelines, integrating cloud services, automating deployments, and supporting DevOps workflows.

Key Responsibilities:
  1. Develop, test, and deploy scalable data pipelines using Azure Data Factory (ADF), Python, PySpark, and Java.
  2. Implement data transformation, ETL/ELT workflows, and data integration solutions.
  3. Optimize data flow and performance for cloud-based data processing.
  4. Use Terraform to provision and manage Azure resources.
  5. Implement infrastructure as code (IaC) best practices for automated cloud deployments.
  6. Ensure efficient resource scaling and cost optimization.
  7. Collaborate with DevOps teams to build automated CI/CD pipelines in Azure DevOps.
  8. Deploy and manage containerized applications using Docker & Kubernetes.
  9. Monitor and troubleshoot build, deployment, and infrastructure issues.
  10. Optimize data pipeline performance in Azure.
  11. Implement cloud security best practices and ensure compliance with data governance policies.
  12. Troubleshoot data and infrastructure-related performance issues.
  13. Work closely with data engineers, cloud architects, and DevOps teams to design solutions.
  14. Participate in agile ceremonies, sprint planning, and technical discussions.
  15. Maintain technical documentation and best practices.
Required Skills & Experience:
  1. Strong experience in: Python, PySpark, Java, Azure Data Factory (ADF).
  2. Hands-on experience with Azure Data Lake, Synapse, and Azure Functions.
  3. Experience in cloud infrastructure automation and DevOps workflows.
  4. Experience with Azure DevOps, Git, YAML pipelines.
  5. Ability to work with Docker, Kubernetes, and containerized applications.
  6. Strong problem-solving and debugging skills.
  7. Experience working in Agile/Scrum environments.
  8. Excellent communication and collaboration skills.
Nice to Have:
  1. Experience with Databricks, Apache Spark, or ML workloads.
  2. Knowledge of security best practices in cloud environments.
  3. Azure, Terraform, or DevOps-related certifications.
Why Join Scrumconnect Consulting?

Work with modern cloud technologies and data platforms. Collaborate with industry experts and agile teams. Career growth opportunities with training and certifications. Competitive salary and flexible work arrangements.