Data Engineer (Azure Databricks) in the EU
Andersen is hiring a Data Engineer (Azure Databricks) in the EU for a project enhancing a cloud-based data platform and delivering scalable data solutions for drug development.
andersen · На сервисе с: 03.10.26 15:07
Andersen is hiring a Data Engineer (Azure Databricks) in the EU for a project enhancing a cloud-based data platform and delivering scalable data solutions for drug development.
The customer is an international company operating in the life sciences sector. It focuses on developing and commercializing specialized products and solutions, supported by research, technology, and partnerships across multiple markets.
The project is focused on developing and enhancing data products for scientific and business stakeholders within the drug development domain. It aims to strengthen the cloud-based data platform and delivery capabilities, enabling scalable data processing, reliable analytics, and effective use of data across the organization.
- Building and maintaining multi-layer pipelines (Landing → Raw → Enriched → Curated) on Azure Databricks (Spark, Delta Lake),
- Orchestrating pipelines via Databricks Jobs / Workflows: dependencies, retries, alerting.
- Implementing batch and API-based data ingestion from multiple source systems.
- Applying schema enforcement and handle schema drift; run data quality checks at ingestion.
- Working hands-on with dbt for pipeline observability and data governance tasks.
- Monitoring job failures, data freshness, and pipeline health against SLA-based delivery targets.
- Setting up alerts, logging, and retry strategies for production pipelines.
- Working within a controlled Dev/Test/Prod environment: access control, audit trails, pipeline documentation, traceability to requirements.
- Contributing to CI/CD and MLOps/AI pipelines on Databricks where the project needs it.
- Commercial experience in Data Engineering for 4+ years.
- Strong hands-on experience with Azure Databricks, including Spark and Delta Lake.
- Experience building and maintaining multi-layer data pipelines following the Landing → Raw → Enriched → Curated architecture.
- Experience with schema enforcement and schema-drift handling.
- Ability to implement data quality checks at ingestion.
- Experience orchestrating pipelines using Databricks Jobs / Workflows, including dependencies, retries, alerts, and failure handling.
- Experience with both batch and API-based data ingestion.
- Strong hands-on experience with dbt.
- Ability to monitor job failures, data freshness, and overall pipeline health.
- Experience delivering data in line with defined SLAs.
- Experience implementing alerting, logging, and retry strategies.
- Experience designing and maintaining access control models.
- Understanding of audit trails, data lineage, and end-to-end traceability.
- Experience with logging and evidence generation for audit and compliance purposes.
- Understanding of segregation of duties.
- Experience defining and maintaining data contracts.
- Ability to create and maintain clear pipeline documentation.
- Experience working across controlled Dev, Test, and Prod environments.
- Ability to ensure traceability between requirements and implemented data solutions.
- Level of English – from Upper-Intermediate and above.
- Experience with Databricks.
- Experience with SAP S/4HANA; Microsoft 365; Azure; SharePoint; Enterprise integrations.
- Experience with CI/CD and DevOps practices for data, including Git-based development and branching strategies.
- Experience building and maintaining CI/CD pipelines for data workflows.
- Experience promoting code and data solutions across Dev, Test, and Prod environments.
- Ability to integrate automated testing into deployment pipelines.
- Experience with MLOps and AI pipelines on Databricks.
- Exposure to regulated domains such as GxP, clinical trials, and pharmacovigilance.
- Strong SQL engineering skills, including the development of modular and reusable dbt models, tests, and documentation at Analytics Engineer depth.
- Experience with dimensional data modeling, including star schemas, One Big Table (OBT) approaches, and curated data marts.
- Familiarity with Tabular Editor and metadata-as-code concepts.
- Experience with Power BI dataset design, performance optimization, and maintaining KPI and semantic consistency.
- Experience in the pharmaceutical or biotechnology industry.
- Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
- The opportunity to change the project and/or develop expertise in an interesting business domain.
- Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
- Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
- The opportunity to earn up to an additional 1,000 EUR per month, depending on the level of expertise, which will be included in the annual bonus, by participating in the company's activities.
- Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
- Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
- Certification compensation (AWS, PMP, etc).
- Referral program.
- Private health insurance and sports compensation, depending on the type of employment.
Join us!
Похожие вакансии Data Engineer
Mayflower is a technology company building large-scale entertainment products for a global audience. Our platforms serve millions of users worldwide and rank among the top-50 most visited websites globally. For more than 10 years, we've been creating reliable, high-performance solutions at the intersection of product, engineering, and entertainment.
Responsibilities
Design, develop and maintain data pipelines (python, SQL, Airflow, dbt, k8s, Spark)
Support data platform in terms of data integrations, data quality, technical implementation
Conduct new data integrations
Ensure data quality
Documenting your solutions
Provide support to analytic teams
Important/challenging tasks:
build redundant data transformation pipelines across different domains and centralised Operational Data Store
You’ll thrive here if you have
At least 3 years working as a data engineer
Python programming (outside of Airflow)
Working with docker
SQL advanced
Kafka, Airflow, Spark
dbt as a plus
Column store DBs (clickhouse as a plus)
S3/iceberg as a plus
Understanding how web applications work
Qualification that can be a plus:
Working with k8s (as a plus)
Experience with data-mesh approach
Conditions
We know that great talent deserves great conditions, so here's what you can expect when joining us:
EU-based employment contract and a 3-year Cyprus work visa with full support for your relocation and visa processes, including assistance for your family.
EU passport. Opportunity to obtain an EU passport after 4 years of residence
Full relocation package: flights to Limassol for you and your family, a company-covered apartment for the first month, and full relocation support to make your move smooth and hassle-free.
Provident fund (Cypus): a long-term savings plan co-funded by you and the Company together (available after probation) that grows throughout your time with us in Cyprus.
Transparent performance reviews twice a year, with bonus opportunities and salary adjustments.
Private medical insurance for you and your family
Corporate mobile plan (unlimited in Cyprus with roaming included)
Mindfulness & well-being support, including psychological assistance with 50% coverage.
50% coverage of school and kindergarten fees for your children.
Fully covered sports benefits, and also access to in-house electric scooters and bike rentals
Professional growth support: language courses, conferences, training programs, and coaching.
Peer recognition program: receive and share kudos for great work.
A fully equipped office in Limassol’s city center, with everything you need for deep work and collaboration.
Free breakfasts and lunches in the office and an in-house coffee bar with high-quality drinks and a health bar stocked with nutritious snacks.
A strong engineering culture: international teams, corporate events, team buildings, and hackathons — because great work happens in great communities.
About the role
Responsibilities
- Foster a culture of working with data across the organization, ensuring data-driven decision-making
- Create and maintain a unified system for processing, storing, and validating data, ensuring data integrity and accessibility
- Design and build processes for processing and enriching data, participating in all stages of the data pipeline from data capture to consumer presentation
- Develop and maintain infrastructure for big data storage and processing using tools like Kubernetes (K8S) and Terraform
- Create and optimize APIs (REST, gRPC) for high-load data access services, enabling efficient data retrieval
- Write integration and unit tests, develop automation tools for data validation and alerting
- Сontribute to system design and architecture with the development team
Qualifications
- Advanced proficiency in Python 3.7+ with strong experience in developing ETL processes using PySpark
- Proven experience in developing data flows using Airflow2
- High level of expertise in SQL, including complex queries and optimization
- Extensive knowledge and industrial experience with Kubernetes (K8S)
- Strong understanding of data processing algorithms and principles, with experience in Spark/Flink
- Solid understanding of general programming concepts, including design patterns, OOP, modularity, and pure architecture
- Demonstrated ability to take ownership of technologies or services and proactively contribute ideas to the team
Benefits
- Help us challenge injustice by creating fair choices for millions of people across 47 countries
- Develop your professional skills with access to mentoring, career consulting, and learning programs
- Collaborate with teams around the world and gain international experience through our Global Talent Exchange Program
- Engage in company-wide challenges, awards, sports activities, employee-led social impact and volunteering projects
- Work alongside people who take initiative, speak openly, and challenge themselves to grow
- Improve your language skills through co-financed courses and internal speaking clubs
- Annual salary from 49 100 EUR gross
About the role
You will be working with cutting edge cloud technologies (GCP, AWS, BigQuery, Databricks, K8s) and building a large scale data infrastructure for analytics, machine learning, and streaming/CDC data delivery.
Responsibilities
- Build and operate batch and streaming ingestion into a layered BigQuery DWH (raw → ODS → data marts) using Airflow, Debezium CDC over Kafka with protobuf, Pub/Sub, and Dataflow
- Integrate external data sources end-to-end — marketing platforms (GA4, AppsFlyer, TikTok/Meta/Google Ads), payment providers, S3 buckets, and third-party APIs — including schema contracts, backfills, and reconciliation
- Engineer the data platform itself in Python: custom Airflow operators and connectors in a shared ETL framework, Kafka Connect on Strimzi (K8s), Cloud Functions, and API integrations with external providers
- Build CI/CD and change-management tooling for BigQuery: GitHub-based test-and-approval flows, SQL migration engines (Liquibase/Flyway/Bytebase), sandbox validation, backup and rollback
- Own reliability and correctness of pipelines: idempotency, deduplication, late-data handling, backfill and replay, freshness monitoring and alerting; write integration and unit tests
- Drive data governance and compliance: ITGC-compliant change management for BigQuery, IAM and least-privilege access, PII policy tags and DLP, Unity Catalog on Databricks, column-level lineage (OpenMetadata/Dataplex), and disaster-recovery planning
- Build internal data tools and platform services for agentic workflows with data — Streamlit apps, Slack bots, LLM-based agents and MCP servers that help teams find and use data
- Support analysts and business teams with data requests, fostering data-driven decision-making across the company
- Contribute to system design and architecture with the development team
Qualifications
- Strong practical Python: clean, well-structured, and tested code for services, tooling, and data pipelines
- Solid software design skills (OOP, modularity, design patterns) — we build platform tools for agentic workflows with data and plan to develop data-related backend services, so well-designed code is highly valued
- Experience building and operating services in a cloud environment (GCP, AWS or similar): CI/CD, containerization, monitoring and alerting
- Familiarity with Kubernetes and Terraform — our infrastructure runs on GCP/K8s
- Hands-on experience with DWH-related tasks (BigQuery or another cloud warehouse) and confident working SQL
- Clear communication with non-engineering stakeholders — a meaningful share of the work is data requests from analysts and business teams
- Demonstrated ability to take ownership of technologies or services and proactively contribute ideas to the team
- Advanced SQL: complex queries, window functions, partitioning, clustering, and cost optimization
- Experience building reliable pipelines around CDC (e.g., Debezium): idempotency, schema evolution, backfills, and reconciliation
- Analytical data modeling skills: table grain, facts vs dimensions, slowly changing dimensions, and metric definitions
- Experience with stream processing frameworks such as Flink or Apache Beam/Dataflow
- Exposure to data governance and audit compliance (ITGC/SOX), Databricks Unity Catalog, or lineage/catalog tooling (OpenMetadata, Dataplex)
- Interest in building LLM-based agents and AI tooling for data
Benefits
- Help us challenge injustice by creating fair choices for millions of people across 47 countries.
- Develop your professional skills with access to mentoring, career consulting, and learning programs.
- Collaborate with teams around the world and gain international experience through our Global Talent Exchange Program.
- Engage in company-wide challenges, awards, sports activities, employee-led social impact and volunteering projects.
- Work alongside people who take initiative, speak openly, and challenge themselves to grow.
- Improve your language skills through co-financed courses and internal speaking clubs.
HireSeeker собирает вакансии со всех площадок и присылает только релевантные. Бесплатно.