IAG Singapore, SG
#CLOSER #BRAVER #FASTER Data Engineer IAG Firemark Ventures Role based in Singapore About The Role The Data Engineer role ensures that data pipelines are scalable, repeatable and secure. It helps facilitate getting data from a variety of different sources, getting it in the right formats, adhering to data quality standards, and assuring that downstream users can get data quickly. Additional responsibilities include developing prototypes and proof of concepts for the selected solutions, and implementing complex data projects with a focus on collecting, parsing, managing, analysing and visualising large sets of data using multiple platforms. Data Engineers should also have the capability to architect highly scalable distributed systems, using open source tools. The Data Engineer role is a technical job that requires substantial expertise in a broad range of software development and programming skills. These professionals have knowledge of data analysis, end user requirements analysis, and business requirements analysis to develop a clear understanding of the business needs and to incorporate these needs into technical solutions. They have a solid understanding of physical database design principles, and the software development life cycle. Preferred Skills and Expertise Experience in building and maintaining data pipelines on Hadoop, AWS, Google Cloud, or Azure. Experience with Spark, Kafka, Hadoop, Hive to process both batch and streaming data. Experience with schedulers or distributed processing tools such as Airflow, Nifi or Oozie. Proficiency in Python or Scala. Being a polyglot is a big plus. Experience with Docker containers and container management services such as Kubernetes or Docker Swarm. Proficient understanding of code versioning tools such as git. Experience with AWS components are a huge plus. Experience in Agile methodology and TDD. Good understanding of data-structures and algorithms. Excellent written and verbal communication skills. Experience with NoSQL and RDBMS databases. At IAG, we believe that everyone has a unique point of view to share, shaped by their life experiences, cultures & passions. We celebrate and commit to #Proud to be me – we value difference, not sameness #Together – harnessing our collective wisdom enables us to be our best for our customers & each other #No boxes – it's not about labels, boxes or categories. It's about building a diverse and inclusive mindset into everything we do IAG is the largest general insurance group in Australia and New Zealand. IAG owns some of the region's most trusted brands, including NRMA Insurance, CGU, SGIO, SGIC, WFI and Swann Insurance.