Data Engineer Spark (Офис/Банк) (Python, Java)

About the Role

We are seeking exceptional Senior Engineer- Hyperscale Analytics to design and scale next-generation data processing and analytics platforms that power OLTP, OLAP, and large-scale distributed data systems. You will build and optimize pipelines and services that handle billions of records daily, enabling real-time transactions, analytical insights, and AI-driven decisioning.

What You’ll Do

• Transactional & Analytical Systems: Design and implement highly scalable OLTP systems for real-time workloads and OLAP systems for complex analytical queries on massive datasets
• Distributed Processing: Build, optimize, and maintain large-scale batch and streaming pipelines using frameworks such as Apache Spark, Flink, Presto/Trino, or Kafka Streams
• System Performance & Scale: Optimize systems for low-latency queries, high-throughput ingestion, and interactive analytics, ensuring seamless performance as data volumes scale to petabytes
• Data Infrastructure: Develop and integrate with modern storage and processing systems (e.g., Snowflake, BigQuery, Redshift, Cassandra, HDFS, Delta Lake, Iceberg) to support hybrid analytical/transactional workloads
• Reliability & Observability: Ensure high availability, reliability, and monitoring across large compute and storage clusters with automated failover and recovery
• Collaboration: Partner with data scientists, ML engineers, and product teams to build unified, secure, and cost-efficient data platforms

What You’ll Bring

• 6+ years of professional software engineering experience, with a significant portion focused on data infrastructure, distributed systems, or large-scale analytics platforms
• Deep experience with distributed data processing frameworks (e.g., Spark, Flink, or similar)
• Strong background in cloud-scale data architecture (AWS, Azure, or GCP) — data lakes, warehouses, streaming platforms
• Proficiency in one or more of: Python, Java, Scala, or Go
• Experience with modern data warehouse/lakehouse technologies (e.g., Snowflake, Databricks, BigQuery, Redshift)
• Solid understanding of data modeling, ETL/ELT design, and pipeline orchestration (e.g., Airflow, dbt)
• Track record of designing for scale, reliability, and cost efficiency in production systems

Bonus Skills

• Experience with HTAP (Hybrid Transactional/Analytical Processing) systems or real-time analytics platforms
• Familiarity with data lakehouse architectures and formats like Parquet, ORC, Delta, Iceberg, Hudi
• Knowledge of containerized deployments (Docker, Kubernetes) and cloud-native data architectures (AWS Redshift, GCP BigQuery, Azure Synapse)
• Background in query engine development or contributing to open-source OLAP/OLTP frameworks

About the Role

We are seeking an exceptional Senior Engineer – Access Entitlements to design and scale the systems that discover, normalize, and reason about who has access to what across our customers' entire data estate — on-prem file shares, Active Directory, and SaaS platforms like SharePoint, Google Workspace, and Microsoft 365. You will build the connectors, pipelines, and graph models that turn millions of raw permission grants into an accurate, queryable picture of access across billions of files and identities, powering least-privilege analysis, access certification, and risk detection for enterprise customers

What You'll Do

• Entitlement Scanning: Build and extend connectors that enumerate identities (users, groups, service accounts, computers) and resource-level permissions (ACLs, role assignments, sharing links) across on-prem and SaaS data sources, handling per-connector quirks like SID resolution, inheritance breakage, and nested group membership
• Scale & Concurrency: Design entitlement pipelines that safely parallelize across large tenants — correctly handling shared state, batching, and checkpointing so a scan of hundreds of thousands of principals and files can pause, resume, and recover without data loss or duplication
• Data Modeling: Own the mapping from raw connector output to normalized entitlement records, and from those records into our identity graph — designing node/edge structures that represent principals, resources, and access grants (including sharing links and group-inherited access) in a way that supports fast traversal at scale
• Multi-Store Architecture: Work across the full data path — Elasticsearch for search-driven access, Databricks/Delta for large-scale analytical joins, and a Neptune-backed graph for relationship queries — making deliberate tradeoffs about what gets synced where, and keeping those stores consistent as data volume grows
• Entitlement-Activity Correlation: Build the pipelines that join static entitlements against observed activity logs to answer "who can access this, and who actually does" — the core signal behind over-permission detection and access-risk scoring
• Reliability & Correctness: Instrument and test for the failure modes unique to entitlement data — partial scans, malformed ACLs, race conditions in concurrent principal writes, and inconsistent state between graph and source-of-truth stores

What You'll Bring

• 6+ years of professional software engineering experience, with meaningful time spent on identity/access systems, security data pipelines, or large-scale distributed data processing
• Strong Go, Java, or Python skills, with direct experience writing concurrent/parallel data pipelines (goroutines, worker pools, or equivalent) and reasoning about race conditions and shared state
• Experience with graph databases or graph query languages (Gremlin, Cypher, or similar) and modeling relationship-heavy data
• Familiarity with Elasticsearch/OpenSearch and at least one large-scale analytical store (Databricks, Snowflake, BigQuery, Redshift)
• Understanding of identity and access concepts — RBAC, ACLs, group membership

Bonus Skills

• Experience with Microsoft Graph API, SharePoint REST APIs, or ActiveDirectory/LDAP-based identity systems
• Familiarity with AWS Neptune, TinkerPop/Gremlin, or other graph-native databases at production scale
• Background in DSPM, CIEM, IAM governance, or data security posture tooling
• Experience designing multi-tenant systems with per-tenant data isolation across search, analytical, and graph store

About the Team

Here at Avride, we're building the future with self-driving vehicles and delivery robots. As you can imagine, this creates a massive amount of data, and we need someone to help us connect the dots. This is a great opportunity to join a growing team and have a real, tangible impact on our technology and our success.

About the Role

We’re looking for a Data Analytics Engineer who is skilled in both data engineering and data analysis. Your main goal will be to take the messy, raw data coming from our vehicles, APIs, and databases, and transform it into clean, reliable datasets that everyone from our engineers to our CEO can actually use. You'll be responsible for building the data pipelines that make this happen, and the dashboards that bring the insights to life.

What You'll Do

• You'll tame our raw data streams, building the ETL pipelines that pull information from all corners of the company into our data warehouse.
• You'll work with some truly unique datasets—everything from vehicle sensor telemetry and logistics data to the results of our internal system tests.
• You'll write the smart, efficient SQL queries needed to shape our data and prepare it for analysis.
• You'll create and manage the Grafana dashboards that our teams rely on to track performance and spot issues.
• You'll work closely with other teams to figure out what data they need, and then you'll deliver it.
• You'll also get to experiment with our internal AI and LLM-based tools to find new ways to analyze data and automate insights.

What You'll Need

• Strong, practical experience with Python and its data libraries (like Pandas, Polars, etc.).
• Expert-level SQL. You should be very comfortable with complex joins, window functions, and query optimization.
• A solid background in building and maintaining ETL pipelines using modern tools.
• Hands-on experience with a BI tool like Grafana, Tableau, or Looker.
• Familiarity with workflow orchestrators like Airflow, Dagster, or Prefect.
• A good high-level understanding of how Large Language Models (LLMs) work and an interest in applying them to data problems.
• A strong sense of ownership and a passion for making sure the data is right.

Nice to Have

• You've worked with ClickHouse or other modern analytical databases (Snowflake, BigQuery, Redshift).
• You have experience with vehicle, sensor, or logistics data.
• You've worked in the autonomous vehicle or robotics industry before.

#LI-MS1