Sr. Software Engineer, Observability and Telemetry

Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance expectations, ease of use, and cost efficiency. With AI redefining the computing paradigm, solutions must evolve to unify innovations in software models, compilers, platforms, networking, and semiconductors. Our diverse team of technologists have developed a high performance RISC-V CPU from scratch, and share a passion for AI and a deep desire to build the best AI platform possible. We value collaboration, curiosity, and a commitment to solving hard problems. We are growing our team and looking for contributors of all seniorities. Tenstorrent is building the world’s fastest, most efficient AI compute clusters. Our modular RISC-V and AI processors can snap together into a single, massively parallel distributed supercomputer consisting of thousands of compute nodes. As we scale, the volume and complexity of operational data grows by orders of magnitude. Observability and telemetry are key to ensuring our customers can resolve problems in minutes rather than hours. The telemetry team owns our proprietary telemetry infrastructure, spanning from the device level to the infrastructure needed to drive dashboards, monitoring systems, and orchestration. This role is hybrid, based out of Santa Clara, CA; Austin, TX; or Toronto, ON. We welcome candidates at various experience levels for this role. During the interview process, candidates will be assessed for the appropriate level, and offers will align with that level, which may differ from the one in this posting. Who You Are Strong C++ engineer and comfortable working in both low-level environments and distributed systems design. Experience building atop observability platforms such as Prometheus, OpenTelemetry, Grafana, ClickHouse, or similar technologies. Solid understanding of data structures for manipulating large volumes of data. Familiarity with SQL databases, with time-series databases a plus. Curious about networking and communication across large clusters and comfortable reasoning from first principles while challenging industry conventions. What We Need Architect, implement, and maintain TT-Telemetry, our C++-based service for collecting and exporting device-level metrics. Interface with internal engineering teams to build a deep understanding of Tenstorrent’s architecture and identify and surface useful metrics. Design efficient built-in web GUIs for observing device- and cluster-level state, diagnosing problems, and monitoring utilization. Design ingestion pipelines for industry standard telemetry systems (e.g., Prometheus). Help define the long-term architecture of Tenstorrent’s distributed telemetry stack. What You Will Learn How large-scale AI clusters are architected from the networking layer up. The performance characteristics of custom AI hardware and RISC-V processors at scale. How telemetry and observability considerations impact the design of next-gen AI accelerators. How to design and architect a world-class telemetry and observability platform from the ground up. Compensation for all engineers at Tenstorrent ranges from $100k - $500k including base and variable compensation targets. Experience, skills, education, background and location all impact the actual offer made. Tenstorrent offers a highly competitive compensation package and benefits, and we are an equal opportunity employer. This offer of employment is contingent upon the applicant being eligible to access U.S. export-controlled technology. Due to U.S. export laws, including those codified in the U.S. Export Administration Regulations (EAR), the Company is required to ensure compliance with these laws when transferring technology to nationals of certain countries (such as EAR Country Groups D:1, E1, and E2). These requirements apply to persons located in the U.S. and all countries outside the U.S. As the position offered will have direct and/or indirect access to information, systems, o ... (truncated, view full listing at source)