Who we are

Contra Labs is a human-centered research lab studying how creatives actually use AI, how tools can serve creative work better, and how to protect the originality that gets flattened as everything starts to look the same.

We connect a global network of top creative talent with frontier AI labs and product companies. By grounding our research in real creative workflows, we help define what "good" looks like across design, video, imagery, and beyond. Our work powers benchmarking, evaluation, and post-training for leading models and applications. Built on Contra, the professional network for independent creatives, we turn human judgment into the training and evaluation layer that helps these tools serve the people who use them.

Why this role exists

We are scaling across client engagements: post-training dataset creation, benchmarking programs, and ongoing evaluation partnerships. Each project spans a different mix of AI labs, creative-tool companies, and research teams, with different requirements, timelines, and quality bars. We need someone to own this work from end to end.

Think of yourself as the operator for a project or vertical. You sit between three groups that do not naturally talk to each other: the client who needs the data, the creative experts producing it, and the internal engineering team building the tooling. Your job is to make all three feel like they are working on the same thing, and to ship results that are both rigorous and genuinely useful.

What you'll own

• The methodology. Annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs.
• Delivery. Scope, evaluator staffing, quality routing, timelines, and the client relationship, from kickoff to final readout.
• Quality. Data quality, inter-rater reliability, and real-time course correction when something feels off. You hold the bar, and you hold the levers to fix it.
• Analysis and storytelling. Qualitative and quantitative analysis turned into clear findings, recommendations, reports, and workshops that clients act on.
• The systems behind it. You turn repeated work into clean, measurable process, and you partner with engineering on the tooling that makes that process faster and more reliable.
• The narrative. Case studies and benchmark publications that sharpen how Contra Labs is understood in the market.

About you

• You design studies for a living. You can scope a question, run the study, analyze the data, and translate it into a direction people can act on.
• You understand good creative, and you can explain it. You recognize creative quality and can say why, and you can tell the difference between personal preference and something closer to ground truth. You build frameworks that capture that signal faithfully.
• You care more about people than about technology for its own sake. You understand how humans judge creative work and design systems around that.
• You turn messy work into machines. You make process clear, predictable, and measurable, and you run a tight ship across several projects at once.
• High ownership, low ego, bias to action. You ship a v1, learn, and do what it takes to deliver.

Experience

• 3+ years designing studies, running evaluations, analyzing data, and presenting findings.
• Experience with data labeling, annotation, human-in-the-loop systems, or AI and ML evaluation workflows.
• A track record managing several concurrent projects, including annotation, evaluation, or QA work.
• A real point of view on creative quality and what "good" looks like.

Bonus

• A strong research background, qualitative and quantitative. A Master's or PhD in HCI, cognitive science, psychology, design research, or a related field is a plus, and so is equivalent depth from industry.
• Statistical methods for evaluation work: inter-rater reliability, sampling, regression, significance testing.
• RLHF, LLM evaluation, or benchmark design.
• Ethnography, UX research, or design-study methods.
• Startup or 0-to-1 experience.

Requirements

• Based in San Francisco, California

Total comp

• Salary: $150,000 to $200,000 USD plus equity
• Medical, dental, vision
• 401k matching
• Company laptop on your start date

Interview process

1. Recruiting team (20 minutes)
2. CEO and co-founder (30 minutes)
3. Technical interview, research and methods focus (45 minutes)
4. VP of Product (45 minutes)
5. Paid case study and presentation (60 minutes)

Note: Contra communicates with applicants through contra.com domains only. We never ask candidates for money. For current postings, visit Contra Careers.

The Detection team at HUMAN is dedicated to protecting the internet by disrupting the economics of cybercrime. On Threat Research, we are responsible for identifying bot traffic pretending to be humans viewing ads, understanding how the bad actors are monetizing the invalid traffic, and launching mitigations to stop the bots.

We are looking for an Analyst to join our team; someone who is curious, efficient, and possesses a relentless drive for improvement and will use those tools to manage the full-cycle of fraud fighting: from the early stages of exploratory research to the development and launching of statistical detection techniques.

What you'll do:

• Identify novel methodologies to detect invalid and bot traffic

• Act with speed to deploy stop-gap mitigations for emerging threats

• Deploy robust defenses based on statistical analysis of supply chain data, network & JS signals, and fraud actor behavior

• Deliver new customer-facing features through collaboration with the Product and Engineering teams

• Build complex queries and notebooks to analyze historical data, including the use of techniques such as: sub-queries, temp tables, complex joins, variables

• Participate in week-long rotations with the primary responsibility of Triaging customer and internal inquiries about fraud efficacy by acting as the first line of analysis

• Continuously identify opportunities for process and product improvement (e.g. internal tooling, mitigation processes, etc) that reduce the time needed to identify and action a threat

• Rigorously document research, analyses, and threat mitigations to build visibility and auditability of work

• Experiment! Try new methods of identifying threats. Test out new tooling. If you have ideas, we want you to test them out.

Who you are:

• You have a good sense of ownership which drives you to find ways to do things better, faster, and leaner; you hold yourself accountable for the quality and accuracy of your work

• You love puzzles and trying to think one step ahead of a fraudster; you look to find new and innovative ways to tackle complex problems through meticulous experimentation and application of emerging technologies

• You are open and transparent and work in tight collaboration with other team members on data projects and cross-departmentally

• You are communicative and go out of your way to support your team

• Proficiency at writing performant SQL, bonus points for Snowflake, on TB to PB of data

• Experience improving processes and implementing tooling to accelerate analysis

• Python experience or exposure preferred

• AdTech experience or exposure preferred

About HUMAN Security

HUMAN was founded in a Brooklyn sci-fi bookstore with a hacker mindset and the lofty mission of making the internet safer by putting humans first. Today, our Human Defense Platform safeguards enterprises from sophisticated bots, fraud, and account abuse, verifying the humanity of more than 20 trillion interactions per week for the world’s biggest brands and internet platforms.

HUMAN is an equal opportunity workplace dedicated to protecting the internet's integrity for everyone. We believe in putting people first, embracing diversity of thought from our global teams, and welcoming all individuals to share their unique experiences as we fight cybercrime together. We support our Humans with a comprehensive total rewards package for personal and professional development, including well-being and learning stipends, flexible work options, and dedicated time off. While our HQ is in NYC, we have teams worldwide.

HUMAN is growing fast, and there’s never been a more meaningful time to join us. If you thrive on solving complex problems and want to help shape the future of cybersecurity, join us as we build a safer, more human internet.

If you are an individual with a disability or special need that requires accommodation, please contact us directly.

What you'll do:

• Lead the development of product features and detection capabilities, such as payload signal collection and launch ad fraud detection techniques utilizing those signals in order to identify different forms of Invalid Traffic (IVT) in mobile and desktop environments.

• Facilitate research into new and emerging threat models within the Ad Tech Ecosystem, as well as new environments and types of advertising that are potential vectors of abuse. For example, by finding new days to detect tools like Selenium, Playwright, or Puppeteer.

• Answer the technical questions, such as: How is an actor obfuscating the redirect between one page to another? How is a data center pretending to be a desktop browser? Which page elements are being tampered with and why? What javascript evidence can we find that automation was occurring?

• Keep up on industry standards, techniques, cybersecurity trends, and tools adversaries might use to circumvent detection or mitigation.

• Red team, experiment, and develop new tactics for various kinds of fraud and to bypass our detection, no need to wait for an attack to be discovered and used by adversaries first.

• Regularly up-level the broader team’s knowledge of signals and techniques useful for identification of invalid traffic, ad fraud, or automation, such as by leading signal sharing sessions with the rest of the team

Who you are:

• You’re passionate about finding bad actors and disrupting their techniques :)

• You are knowledgeable on the mechanics of fraud and abuse within Ad Tech and the programmatic ecosystem OR are knowledgeable on the cybersecurity threat landscape

• You have extensive knowledge of how the internet operates, a clear understanding of DNS, TCP/HTTP/TLS network protocols, and the capabilities of javascript signal collection. You’re willing to dig into the details of source code

• Proven experience in scripting languages (Python and JavaScript). Bonus Points for SQL

• You are a curious, adaptable person who is always thinking out of the box that wants to apply your knowledge to always do the right thing

• You are intrigued by the adversarial game, discovering new threats, and rapid experimentation

• You have a strong sense of ownership which drives you to find ways to do things better, faster, and leaner; you hold yourself accountable for the quality and accuracy of your work

• You are passionate about learning and growing your skillset and the skillset of those around you.

• You have knowledge on how to use AI tools effectively or improve, automate, and “agent-ize” existing processes

• You are comfortable juggling different projects as priorities change

About HUMAN Security

HUMAN was founded in a Brooklyn sci-fi bookstore with a hacker mindset and the lofty mission of making the internet safer by putting humans first. Today, our Human Defense Platform safeguards enterprises from sophisticated bots, fraud, and account abuse, verifying the humanity of more than 20 trillion interactions per week for the world’s biggest brands and internet platforms.

HUMAN is an equal opportunity workplace dedicated to protecting the internet's integrity for everyone. We believe in putting people first, embracing diversity of thought from our global teams, and welcoming all individuals to share their unique experiences as we fight cybercrime together. We support our Humans with a comprehensive total rewards package for personal and professional development, including well-being and learning stipends, flexible work options, and dedicated time off. While our HQ is in NYC, we have teams worldwide.

HUMAN is growing fast, and there’s never been a more meaningful time to join us. If you thrive on solving complex problems and want to help shape the future of cybersecurity, join us as we build a safer, more human internet.

If you are an individual with a disability or special need that requires accommodation, please contact us directly.