Overview

We are looking for a Senior Data Engineer to support a long-term engagement focused on simulation data processing for Autonomous Vehicle (AV) development. This role sits at the intersection of large-scale data engineering, real-world sensor analysis, and safety-critical automotive systems.

You will work with high-volume multimodal sensor data collected from a test AV fleet and help transform raw inputs into simulation-ready datasets used to develop and validate advanced autonomous driving features. The work directly contributes to next-generation vehicle safety in collaboration with a leading global automotive OEM.

What You’ll Work On

You’ll handle complex real-world driving data and scenarios such as:

• Obstacle detection

• Path planning

• Complex traffic environments (e.g., tunnels, unusual vehicles, temporary network issues)

• Edge cases critical for safe autonomous driving

Sensor inputs include 8–12 cameras, LiDAR, and radar, generating up to ~1TB of data per hour.

Key Responsibilities

• Analyze large-scale real-world sensor datasets to identify edge cases (e.g., hard braking, close-proximity vehicles, unusual road behavior)

• Design and write advanced SQL, Python, and Spark/PySpark queries for data filtering, transformation, and preparation

• Work with internal platforms for data search, labeling, and auto-labeling workflows

• Process structured and semi-structured data, including object detection and perception outputs

• Select and prepare relevant data for AV simulation environments and ML pipelines

• Contribute to improvements in data discovery and curation processes

• Build and maintain data mining scripts and ETL pipelines

• Develop internal tools to enhance analytics capabilities and streamline engineering workflows

• Collaborate closely with engineers and researchers to support development and validation of safety-critical AV features

Requirements

• Advanced SQL

• Advanced Python

• Advanced Spark / PySpark

• Hands-on experience with Databricks

• Strong understanding of data pipelines and large-scale data processing

• Familiarity with machine learning workflows (data preparation for training/validation; this is not an ML engineer role)

• Experience working with complex or high-volume datasets

• 4+ years of commercial experience in data engineering or a related field

Nice to Have

• Experience in the Autonomous Vehicle (AV) or ADAS domain

• Exposure to sensor data (camera, LiDAR, radar)

• Understanding of real-world driving edge cases

• University degree in Computer Science or a related technical field