Observability Architect - #2180641

Dev/Null Security is seeking an Observability Architect with a strong engineering background in production Databricks platforms and Apache Spark pipelines.

You will own practical designs, guide implementation and lead platform transformation. You should be an experienced engineer who has moved into architecture, or can demonstrate readiness to take that step.

Essential technical experience:

• Essential: hands-on production Databricks deployment and operation, covering platform configuration, pipelines, governance, performance and cost.
• Strong Apache Spark, PySpark and SQL skills, including partitioning, joins, skew, execution plans and memory or shuffle troubleshooting.
• Spark Structured Streaming experience covering checkpoints, event time, late data, replay, recovery and end-to-end delivery guarantees.
• Practical Delta Lake, Auto Loader, job orchestration and Unity Catalog experience, including permissions, lineage and environment separation.
• Security telemetry ingestion and correlation across cloud, identity, endpoint, network and application sources, with schema and data quality controls.
• Understanding of logs, metrics, traces and OpenTelemetry; ability to distinguish observability, analytics, SIEM and lakehouse responsibilities.
• Experience with a major cloud platform, infrastructure as code, CI/CD, APIs and secure integration; evidence of enterprise platform migration.

Architecture design and leadership:

• Translate business needs, threats and functional and non-functional requirements into practical architectures and clear trade-offs.
• Own target architectures, high-level designs, patterns and decisions; guide detailed design and assure implementation.
• Build transformation roadmaps with dependencies, delivery risks, acceptance criteria and operational handover.
• Present and defend designs at design authorities and executive forums; explain complex concepts clearly to different audiences.
• Provide thought leadership through technical challenge, emerging technology assessment and recommendations grounded in delivery experience.
• Influence stakeholders, facilitate workshops, build consensus and mentor engineers across security, technology and business teams.
• Apply peer review and governance; document deviations and risks, balancing security, resilience, user experience, cost and supportability.

Accountabilities and responsibilities

Databricks platform architecture

• Design workspace, compute, storage and networking patterns against security, scale, availability and data residency requirements.
• Define Unity Catalog ownership, permissions and lineage; integrate enterprise identity and least privilege for users and automated workloads.
• Automate environment deployment and configuration; establish capacity, consumption monitoring, recovery requirements and operational support.

Security data engineering

• Design and validate batch and streaming pipelines using Spark and Databricks, with reusable security data models and enrichment.
• Define schema evolution, deduplication, checkpointing, replay and quarantine controls for missing, malformed and late events.
• Benchmark and tune processing, queries and storage using execution evidence, preserving required data fidelity.

Observability and security integration

• Define telemetry and service measures covering freshness, completeness, latency, source health and pipeline failure.
• Integrate Databricks with SIEM and observability services; validate data placement, end-to-end latency and analyst investigation workflows.
• Apply retention, encryption, masking and access controls; prevent unnecessary personal data, credentials and secrets entering analytical datasets.

Migration and operational improvement

• Deliver phased migrations with data reconciliation, parallel validation, rollback, service continuity and retirement of obsolete services.
• Measure reliability, coverage, query performance, analyst usability and cost; use results to prioritise engineering improvements.

Qualifications and certifications

Relevant degree or equivalent experience. Databricks, cloud, CISSP, SABSA or TOGAF qualifications are desirable. Production Databricks delivery is essential.

Working at DevNull Security

DevNull Security is remote-first. Client-site travel may be required up to 3 times per week, depending on project and customer needs.

We welcome applicants from all backgrounds and are committed to an inclusive, equitable and accessible hiring experience. We offer adjustments throughout application and assessment; tell us what you need.