Platform Engineer - Big Data

£500 - £525 per day
Croydon, Surrey

Role: Platform Engineer - Big Data
Location: Croydon

Start Date: ASAP
Duration: 3-6 months

Job Spec-

As a platform engineer you will build, integrate and deploy the infrastructure and distributed platforms used to host modern big data ecosystems. You'll have solid hands-on experience implementing and designing complex Big Data and ETL systems using multiple technologies and be comfortable working with a variety of agile methodologies

You'll have hands-on experience of new technologies, in particular Hadoop and its surrounding ecosystem. You'll understand the pros and cons of these technologies and have experience on various implementations surrounding them, or of similar tools in this technical space.

Key Responsibilities:

*Ensure scalability, reliability, security and efficiency of systems.
*Contribute to the design and build large-scale distributed big data and ETL systems.
*Build automation to prevent problem recurrence.
*Working with other engineering teams to accelerate software delivery.
*Mentoring other team members.
*Help defining processes and standards, following Open Source best practices.
*Help shaping our department's culture.

Things we seek:

*Hands-on, highly technical with strong experience in systems engineering.
*Has experience working with Hadoop/Yarn and associated ecosystems (i.e. Kafka, Samza, Spark, Flink, Ambari, Ranger, Hive, PIG).
*In-depth understanding of Linux
*Has experience in one or more of these programming languages: SQL, Java, Scala, Ruby, Python, Go or Shell.
*Has experience working with Infrastructure as Code, configuration management systems and CI/CD.
*Has operational experience working with large, highly available, highly transactional systems on private and public clouds.
*Understands how to analyse and present operational metrics and use that data to optimize systems.
*Can work independently and take initiative.


*Docker and Kubernetes.
*Configuration management using Ansible
*CI build pipelines with Jenkins, Drone, Artifactory, SonarQube
*Terraform, Consul and Vault
*Elasticsearch, Postgres, MySQL, Redis, MongoDB, Neo4j
*Monitoring with Sysdig, Prometheus and ELK
*Knowledge of HIDS, NIDS and security hardening.
*Networking experience and understanding of network protocols (e.g. TCP/IP, UDP, ICMP, etc) OSI layers, DNS, VPN and load balancing.

Please submit CVs in the first instance.