AI Quality Specialist

  • Krakow| Hybrid
AI Quality Specialist, 1. imageAI Quality Specialist, 2. imageAI Quality Specialist, 3. image

We are seeking a hands-on IT Operations Generalist to serve as the ultimate judge of quality for our AI-driven root-cause analysis engines. In this role, you will evaluate whether AI-generated diagnostics and troubleshooting guidance deliver genuine, actionable value to system administrators facing real-world network and infrastructure outages.

You will sit directly within the Product team, bridging the gap between deep IT operations reality and cutting-edge AI engineering. Rather than managing timelines, your primary focus will be practical validation: inspecting telemetry from real customer alarms, establishing ground-truth diagnoses, and building automated, break-fix lab environments to stress-test new AI models before they reach production.

If you have spent years managing mid-sized IT environments, know what a true root-cause diagnosis looks like under pressure, and enjoy breaking infrastructure on purpose, this role offers a unique opportunity to shape the future of AI-assisted IT monitoring.

YOUR TASKS

Daily AI Diagnostic Evaluation & Telemetry Audit

  • Review AI-generated incident analyses daily, leveraging real product telemetry and raw customer alarm streams.
  • Evaluate diagnostic accuracy: determine whether the identified root cause is correct, whether symptom alarms are grouped logically, and whether recommended remediation steps are practical and effective.
  • Maintain rigorous standards for quality, identifying hallucinations, redundant alert advice, and unhelpful diagnostic loops.

 

Engineering Feedback Loop & Quality Metrics

  • Identify recurring AI failure modes (e.g., repeating alarm text without diagnosis, missing dependency links,generic fixes) and translate them into actionable engineering tickets and prompt/model requirements.
  • Develop, track, and report baseline quality metrics for AI accuracy, relevance, and actionability.
  • Help design systems to capture and integrate real customer feedback and resolution outcomes as our AI beta program expands.

 

Lab Environment Construction & Fault Injection

  • Design, deploy, and maintain realistic hybrid lab environments (virtualized and cloud-based) that simulate real-world enterprise IT infrastructures.
  • Safely engineer and execute fault-injection scenarios—reproducing real outages such as storage failures, expired SSL certificates, saturated links, DNS misconfigurations, and cascading service dependencies.
  • Ensure lab scenarios are fully repeatable and documented for reliable regression testing.

 

Test Scenario Library & Regression Testing

  • Build and curate a comprehensive benchmark library of golden test cases with verified ground-truth answers.
  • Execute automated and manual regression test suites against candidate AI models prior to every release, ensuring zero degradation in diagnostic precision.

YOUR PROFILE

Must-Have Technical Experience

  • System Administration Foundation: 5+ years of hands-on experience as an IT Generalist, Systems Administrator, or Infrastructure Engineer in a mid-sized enterprise environment.
  • OS & Infrastructure Mastery: Deep proficiency with Windows Server administration combined with practical working knowledge of Linux distributions.
  • Networking Essentials: Strong understanding of core networking protocols and infrastructure services,including DNS, DHCP, VLANs, subnet routing, SNMP, and SSH.
  • Cloud Platform Experience: Hands-on operational experience with at least one major cloud platform (AWS,Azure, or GCP).

 

Automation & Lab Building Skills

  • Command-Line & Scripting: Strong command-line fluency in PowerShell and Bash, with sufficient scripting ability (Python strongly preferred) to automate lab deployment, state resetting, and fault injection.
  • Virtualization & Lab Automation: Proven experience building test environments using hypervisors (Hyper-V,VMware ESXi, Proxmox) or Cloud VMs.

 

Troubleshooting Judgment & Soft Skills

  • Root-Cause Intuition: Superior technical troubleshooting judgment—the ability to look at an alert storm and immediately separate primary root causes from secondary symptoms.
  • Technical Communication: Exceptional written communication skills for authoring clear, concise bug reports,lab reproduction steps, and diagnostic reviews.
  • Pragmatic Mindset: Driven by real-world utility rather than theoretical perfection; passionate about making tools genuinely helpful for sysadmins.

 

Nice-to-Have Qualifications

  • Monitoring Tool Experience: Prior hands-on experience configuring, managing, or querying enterprise monitoring platforms—such as PRTG Network Monitor, SolarWinds, Datadog, LogicMonitor, or open-source solutions (Zabbix, Nagios/Icinga, Prometheus/Grafana, Checkmk).
  • AI Tooling Awareness: Basic familiarity with prompt engineering concepts, LLM benchmarking, or evaluation frameworks.
     

FACTS FOR YOU

  • Many possibilities to further develop your individual skillset
  • Innovative and creative work environment with plenty of room for new ideas
  • A structured onboarding phase with experienced colleagues
  • Flexible working hours and a family-friendly environment
  • Permanent employment contract 
  • A great working atmosphere in an enthusiastic international team with flat hierarchies
  • Paessler is a fast-growing midsized company that has been in the market for 20 years. The company is focused on long-term goals and is currently comprised of 400+ employees

INTERESTED?

Make it easy for yourself and use the application button below.
 
Your contact person for this great job is Annalena Klein.
 
And at the end, a very important note:
For us, diversity is more than just a word — it's a commitment. Our job posts are geared towards all people, regardless of their gender, sexual orientation, background, faith, or physical abilities. We value diversity and strive to create a work environment that prioritizes equal opportunities and inclusion. In line with this objective, please note that candidates with severe disabilities will be given preferential consideration over applicants with equal qualifications. At Paessler, we're determined to break down barriers and provide everyone with the chance to develop their full potential. Please feel free to let us know if you believe there are ways we can improve the wording of our jobs posts or our overall recruitment approach. Let's work together to create a more colorful and inclusive corporate culture!