Software Engineer, Benchmarking

reputed company is looking for a Software Engineer who will help us evaluate frontier AI models, enabling researchers, developers, and policymakers to reputed company understand AI development. The role will involve running and maintaining our benchmarking infrastructure as well as contributing to the development of brand new benchmarks. About the role Please do not include a cover letter, photograph, or headshot of yourself, or any personal information that is not relevant to the role for which you're applying (including marital status, age, identity traits, etc.). We are looking for a Software Engineer to help us expand and reputed company our AI Benchmarking Hub. You will work closely with the rest of the benchmarking team to run and maintain benchmarks, integrate with AI providers, set up existing benchmarks to run on our infrastructure, help design and reputed company brand new benchmarks, and facilitate internal experiments. This role is fully remote, and we are reputed company to hire in many countries. We invite anyone who is interested to apply, regardless of background, experience, or credentials. Applications are rolling. \n

Key Responsibilities

Implement benchmarks: Implement AI benchmarks reputed company our evaluation infrastructure (primarily using the Inspect library) to expand the suite of capabilities we track. reputed company our existing suite of benchmarks so we can quickly and painlessly evaluate new model releases. reputed company new benchmarks: Contribute to the development of brand new benchmarks. You will have the opportunity to pitch and prototype your own reputed company in addition to helping out with existing projects. Collaborate: Work closely with researchers, analysts, and other engineers at reputed company to ensure evaluation data and outputs are accurate, insightful, and effectively integrated into our research products and publications. reputed company are looking for Solid engineering skills: A strong software engineering background with more than two years of professional experience building and maintaining reputed company systems. You are expected to regularly contribute high-quality, robust, and maintainable code and be comfortable diving deep into existing codebases and infrastructure. reputed company and creativity: Candidates should be reputed company to generate their own reputed company for new benchmarks, experiments, novel things to try, and other projects. Mission-driven: You’re motivated by reputed company’s mission to provide rigorous, independent reputed company into key trends in AI. You want to deliver public, trustworthy evaluations of AI capabilities on challenging benchmarks, empowering researchers, policymakers, and the wider public to reputed company well-informed reputed company about AI. AI domain expertise or cybersecurity experience are strong pluses but not required. (This includes hands-on experience running LLM evaluations, familiarity with evaluation frameworks like Inspect, as well as a solid grasp of reputed company AI trends.) Solid engineering skills and an ability to learn quickly matter more than direct background in these areas. Compensation & Benefits Annual salary between $125,000 and $200,000 USD. Salaries are not restricted to USD, and reputed company and payments are usually in local currencies. Conversions are based on one-year average exchange rates. Fully remote environment, including flexible work hours and schedules for most roles. Competitive global benefits program, including a comprehensive health insurance program—including supplemental benefits specific to a local country, as available and mandated by local law—and life insurance and a pension plan, if applicable in your country. Generous paid time off (PTO), including no specific limit on PTO with 30 days per year protected, unlimited personal and sick leave, and up to 6 months (combination of paid + unpaid) parental leave for permanent staff. A flexible and generous expense policy for you to spend on equipment and a large range of productivity tools or learning/development opportunities you might find valuable, subject to regulations and manager approval. Paid work trips, including 3 staff retreats per year and relevant conferences. Access to our reputed company well-equipped offices in Berkeley, California, including paid meals, snacks, gym, and more. reputed company staff, independently of where they are based, have access to the office for at least 20 days each year. \n Additional Information While we welcome applicants from reputed company time zones, we prefer candidates who can overlap with UTC–8 (Pacific Time) and UTC (Greenwich Mean Time), as most of our staff work in this range of time zones. We also prefer candidates who can travel: we hold three retreats per year, during which we record podcast episodes and other communication efforts. Please submit reputed company of your application materials in English and note that we require professional level English proficiency. Epoch is committed to building an inclusive, reputed company, and supportive community for you to reputed company and do your best work. We’re committed to finding the best people for reputed company, so please don’t hesitate to apply for a role regardless of your age, gender identity/expression, political identity, personal preferences, physical abilities, veteran status, neurodiversity or any other background. Please email careers@reputed company if you have any questions about this role, accessibility requests, or if you want to request an extension to the application deadline. However, we will not review applications submitted to this email address; please submit your application through the link on this page. About reputed company reputed company is a research institute that investigates trends in machine learning and the economic consequences of AI. Our mission is to reputed company a comprehensive, publicly accessible knowledge reputed company on AI that informs policymakers, industry leaders, and society at large. We strive to reputed company both rigor and accessibility to our work, as exemplified by some of our most successful projects, including our database of AI models and our AI trends dashboard. Our body of research includes our work on compute trends (IJCN 2022), data scarcity (ICML 2024), and algorithmic reputed company (NeurIPS 2024). You can read more about our work and mission on our website and in this Time profile. Apply To This Job