Ai Automation Cgi Chennai Bengaluru
Heavy focus on testing LLMs, RAG, vector DBs and LangChain—so it's directly about working with AI dev tools and prompt/output validation.
About the Role
Automation Quality Engineer responsible for designing, developing, and maintaining automation test frameworks in Java and/or Python to validate AI-enabled applications. The role focuses on testing LLM-based systems, RAG pipelines, vector databases, OpenAI APIs, and integrating AI-specific test strategies into CI/CD pipelines to ensure quality, performance, and reliability.
Job Description
Role
Seeking an Automation Quality Engineer with strong hands-on experience in Java and/or Python automation frameworks and practical exposure to AI-driven testing. The role focuses on validating intelligent applications and ensuring quality, reliability, performance, and security of AI-enabled systems.
Key Responsibilities
- Design, develop, and maintain automation test frameworks using Java and/or Python.
- Implement UI, API, and backend automation using tools such as Selenium, Playwright, REST Assured, PyTest, or similar.
- Validate LLM-based applications for prompt accuracy, response quality, hallucination detection, and consistency.
- Test and evaluate RAG pipelines, including document ingestion, embeddings, retrieval accuracy, and response relevance.
- Work with vector databases (FAISS, Pinecone, Chroma, Weaviate) to validate indexing, retrieval, and performance.
- Integrate and test applications using OpenAI APIs and similar LLM providers.
- Leverage LangChain (or equivalent frameworks) to test chains, agents, tools, and workflows.
- Create automated test cases for AI workflows, model outputs, prompt versions, and inference pipelines.
- Implement data-driven and AI-assisted testing strategies and participate in CI/CD pipelines to enforce quality gates.
- Collaborate with Data Scientists, Developers, and Product teams to define quality metrics and ensure compliance with security, performance, and scalability standards.
Requirements
- Strong programming skills in Python (and/or Java).
- Hands-on experience with automation testing frameworks such as Selenium, Playwright, REST Assured, PyTest, etc.
- Practical experience with LLMs (OpenAI GPT, Azure OpenAI, or similar).
- Solid understanding and exposure to RAG architectures.
- Experience working with vector databases for embeddings and semantic search (FAISS, Pinecone, Chroma, Weaviate).
- Hands-on usage of OpenAI APIs (chat/completions, embeddings, etc.).
- Experience with LangChain for prompt chaining, memory, and tool usage.
- Strong knowledge of API testing, SQL, and data validation.
- Familiarity with CI/CD tools such as Jenkins, GitHub Actions, Azure DevOps.
- Good understanding of SDLC, STLC, and Agile methodologies.
Good-to-Have
- Certifications in LLMs and/or Python.
- Exposure to AI-assisted testing strategies and emerging AI QA tools.
- Experience with performance testing of AI-enabled systems.
- Knowledge of cloud-based AI platforms and orchestration frameworks.