Senior Big Data Engineer (Python+Apache Iceberg+AWS experience))

About The Role In this role, you will contribute to large-scale data pipeline migrations, moving trading venue data from existing HDF5-based pipelines to modern Parquet and Apache Iceberg-based storage, while ensuring uninterrupted delivery to customers and downstream systems. Our Client is a leading provider of harmonised historical market data and analytics, supporting trading and research decisions. You will work within a focused team of Python and C++ engineers, modernising data parsing pipelines and infrastructure to improve performance, reliability, and scalability. Responsibilities Develop robust Python code for Iceberg-based data storage, retrieval, migration, and validation, working closely with C++ and Python engineers. Migrate existing parsing processes to the new Parquet and Iceberg-based pipeline, focusing on simplicity, performance, and reliability. Collaborate with business stakeholders and engineering teams to gather requirements, understand constraints, design technical solutions, and validate outcomes. Build, test, and deliver production-ready functionality to support business requirements and ongoing pipeline improvements. Contribute to the development and optimisation of existing parsing pipelines and C++ parsers (feed handlers). Leverage AI coding tools to accelerate development while maintaining responsibility for code quality, consistency, and maintainability. Understand end-to-end data flows, including data ingestion, processing, storage, and delivery to customers. Communicate with internal technical teams and external data providers, including financial exchanges, to support data processing and operational requirements. Provide second-line support for daily parsing processes and data-related issues, working alongside the wider development team. Requirements Essential Professional experience in Python development, including writing reliable and maintainable production code. Hands-on experience with Apache Iceberg and pyiceberg libraries. Experience with AWS services, particularly S3 storage, and ideally AWS Glue Data Catalog. Experience with system migrations, ensuring existing end-to-end functionality is maintained. Experience working with large datasets and data-intensive applications. Professional experience developing software in Linux environments. Familiarity with industry-standard development practices, including version control, unit testing, and continuous integration. Strong foundational software engineering skills, with the ability to develop correct and maintainable code independently and with AI assistance. Ability to self-organise and deliver development tasks over projects lasting several months. Strong problem-solving and communication skills. Computer Science or another STEM-related degree. Nice to Have Experience with Apache Spark. Experience working with large-scale distributed systems. Experience delivering well-tested, production-level code as part of a larger codebase. Exposure to equities or derivatives market data from financial exchanges, including protocols such as ITCH, GTP, EOBI, iMpact, Pillar, or Optiq. Knowledge of containerisation technologies such as Docker. SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law. … mais Pesquise vagas mais rápido com Premium Acesse informações sobre empresas, como prioridades estratégicas, tendências de número de funcionários e muito mais Jonatas e outros milhões de usuários têm contas Premium Reative o Premium Aproveite o suporte 24 horas e cancele quando quiser.