Senior Data Platform Engineer

About the role

Help us make a big green dent in the universe. We're on a mission to drive the global renewable energy revolution, and we need many more people to help us make our vision a reality. Octopus Energy Generation (OEGEN) is a leader in sourcing, investing in, and managing a wide range of utility-scale renewable energy assets. As we continue to expand our investment base, we are dedicated to building a diverse team that reflects the communities we serve. Currently, we manage a global portfolio of over £8 billion in renewable energy assets. About the role: As part of our mission to drive smarter energy decisions, we are building a production-grade, governed platform that empowers our teams to deploy AI-enabled tools, agents, and data products safely and at scale. Sitting within the Systems and Data / Transformation team, you will turn existing platform architectures, governance specs, and prototype patterns into production-grade infrastructure. You’ll work alongside our Tech and Transformation Lead, data engineers, application builders, and business stakeholders across Finance, Asset Management, and Fund Operations. This is not a general DevOps or blank-page architecture role. The primary focus is designing and implementing enterprise-grade identity, access, and governance controls across human users, systems, and non-human agents. Our Tech & Ecosystem: Databricks, Lakebase, Unity Catalog, Entra ID / Azure Identity, Okta, ZScaler, GitHub Actions, MCP patterns, and Terraform. What you'll do Platform Engineering & CI/CD Build and maintain Infrastructure-as-Code (IaC) using Terraform across development, test, and production environments. Strengthen CI/CD practices in GitHub Actions, establishing repeatable deployment patterns, environment promotions, automated rollbacks, and release controls. Support architecture decisions by converting designs into maintainable, well-documented engineering patterns. Identity, Access & Security Design and implement federated identity architecture across Okta, Entra ID, and ZScaler. Build identity, token exchange, and permission propagation models that work for both human users and non-human AI agents. Implement SCIM provisioning/deprovisioning and access models with single-revoke capability and full auditability. AI Gateway & Governance Harden existing AI gateway and agent orchestration layers (built on MCP patterns) into secure, production-ready capabilities. Implement API gateway controls: authentication, routing, rate limiting, token budgets, policy-as-code, and action logging. Build platform-, tool-, and agent-level kill switches, alongside dry-run/safe-testing modes for agent workflows. Observability & FinOps Implement AI system observability, including prompt logging, output monitoring, quality scoring, and drift detection. Establish operational monitoring for gateway usage, latency, error rates, and model routing logic. Enforce token budgets, cost alerts, and usage/cost attribution by team and use case. Enablement & Integration Deploy vector database and retrieval infrastructure (e.g., Databricks AI Search) to support Document AI and RAG use cases. Align platform controls with Databricks and Unity Catalog governance patterns. Provide clear documentation, reusable patterns, and self-service onboarding guidance so delivery squads can deploy tools without bespoke manual intervention. What you'll need Federated Identity Mastery: Proven experience designing and implementing enterprise-grade federated identity, token exchange, permission propagation, SCIM provisioning, and access lifecycle management. Platform Foundations: Strong hands-on experience with IaC (Terraform) and production CI/CD pipelines (environment promotion, rollback, release controls). API Gateways: Experience hardening API gateway patterns (authentication, routing, rate limiting, access policy enforcement). Cloud & Data Environments: Technical comfort working across Azure, AWS, or Databr

Skills