widget factoryMadison, WI
Problem Design Engineer
$40/hour
Apply (opens in a new tab)Problem Design Engineer
Problem Design Engineer
widget factoryMadison, WI
Apply for this role (opens in a new tab)today
$40/hour
What We’re Building We create high-quality coding problems for training AI models — drawn from real production repositories, not synthetic puzzles. Your role is to ensure these problems are fair, clear, and technically sound.
The Role:
- Part-Time Code Quality Reviewer | Remote | $40/hr | 0-25 hrs/week
- We’re looking for engineers who can trace the intent behind a code change, understand what changes were made and why, and express that clearly in natural language.
- This is judgment work, not mechanical testing. You'll make philosophical decisions about fairness and clarity in technical contexts.
- What You'll Actually DoReview code changes across unfamiliar repositories and tech stacks
- Assess problem fairness: Could a skilled engineer solve this with only the given context?
- Evaluate test quality: Do tests properly validate the intended solution?
- Write clear justifications for your assessments (2-3 sentences per decision)
- Flag coherence issues: Inconsistent artifacts, unclear requirements, missing context
- Time commitment: 1-4 hours per problem, fully async — work whenever fits your schedule.
- Who Should ApplyIdeal candidates:
- PhD students in CS/computational fields seeking long-term flexible income
- Open source contributors who care deeply about code quality
- Undergrad with at least 1 year in CS, ECE, Computational Biology, or related technical programs
- Professional engineers seeking intellectually interesting part-time work
- You'll thrive if you:
- Have strong software engineering fundamentals Can work independently with minimal supervision
- Write clearly and precisely
- Enjoy making judgment calls in ambiguous situations
- Stay focused during 1-3 hour review sessions
Requirements:
- 1+ years of software engineering experience (internships, research projects, or professional work)
- Fluent English; strong writing is a must
- Responsive, independent, and attentive to detail0–25 hours per week, flexible
- Compensation & Growth$40/hour for all review work
- Performance-based advancement: Top contributors can receive promotions and bonuses
- Long-term engagement: We're building a scalable team, not a temporary workforce
- Application Process15-minute async video interview — answer 4 questions about your technical background and work style
- Sample problem review — demonstrate your assessment skills on a real coding problem
Onboarding — join our active problem design engineering community and start reviewingFAQQ: Is this suitable for PhD students?
Yes — many of our top contributors are PhD students. The work is fully async and fits around research schedules.
Q: Do I need to be available during specific hours?
No. Work whenever you want. Some problem design engineers work weekends, others weekday evenings.
Q: What if I'm not familiar with the programming language in a problem?
That's expected. You're assessing problem fairness and clarity, not implementing solutions. Familiarity with general software engineering principles is what matters.
Q: How quickly can I start?
If you pass the video interview and sample review, you can start within 1-2 weeks.
Q: Is this a path to full-time employment?
This is a long-term contract role, not a pipeline to full-time positions. We value contributors who excel at this specific type of work.
About the Widget Factory:
The Widget Factory is the quality assurance division of Idler (YC S25), where we ensure coding problems meet research-grade standards for AI training. Our mission is to build the most reliable QA system for AI coding benchmarks.
Also on the board Same function, level within a rung
Level
Senior
Salary
$40/hour
Location
Madison, WI
Occupation
Software Developers
Industry
Custom Computer Programming Services
Posted
today