AI Review Agent

Overview

The AI Review Agent is a review layer built into DiligenceVault's rules engine. It is designed for analysts on the requestor side who need to evaluate unstructured content within responder DDQ submissions, such as long text paragraphs, document attachments, text grids, and Yes/No answers that include free-text explanations.

While the existing rules engine handles structured data such as integers, dates, dropdowns, and booleans, it cannot interpret meaning from unstructured content. The AI Review Agent fills that gap. It reads responder submissions, identifies risks, evaluates completeness, and surfaces insights, so analysts spend less time searching through documents and more time on the judgment calls that matter.

Key benefits include:

• Significant reduction in first-pass review time per responder
• Consistent, repeatable evaluation applied across all responders and responses
• Automated highlighting of the specific text or row that triggered a rule, removing the need to manually search through lengthy documents
• A full audit trail of AI results, overrides, and timestamps

How It Fits Your Review Workflow

The AI Review Agent sits between the responder's submission and the analyst's review. Once a responder submits their DDQ and attachments and the requestor accepts the project, the agent runs automatically and performs a first pass across all applicable responses.

Results from the agent feed directly into downstream workflows, including memos and recommendations. Analysts retain full control and can accept, override, or annotate results before finalising their output.

What You See in a Project

Once the AI Review Agent completes its evaluation, results appear directly below each response in the project questionnaire. For every question that has an AI Mode rule applied, you will see an AI Review summary bar showing the combined output, followed by expandable detail for each result type.

The AI Review Summary Bar

At the bottom of each response, an AI Review bar displays the combined status across all rule types applied to that question. This gives a quick read on the response before the analyst opens any details.

• Rating: Shows the assigned rating value, for example: Low, Medium, or High
• Flagged: Appears when the flagging rule has identified issues in the response
• Incomplete: Appears when the completeness rule has found one or more missing items

Each label is clickable and expands to show the full details for that result type, including the AI Reasoning and Assessment Criteria.

Rating 

When a rating rule fires, the response displays a Rating row showing the assigned value (High, Medium, or Low) alongside a confidence tag indicating how certain the agent is about its assessment. Expanding the row reveals the AI Reasoning, which explains why that rating was assigned, and an Assessment Criteria panel listing the specific factors the agent evaluated.

Rating result showing the assigned value, confidence tag, AI reasoning, and assessment criteria

Flag

When a flagging rule fires, the response displays a Flag row showing Issues Found alongside a confidence tag. Expanding the row reveals the AI Reasoning explaining why the flag was triggered, and an Assessment Criteria panel listing the specific conditions that matched the prompt.

Flag result showing issues found, confidence tag, AI reasoning, and assessment criteria

Completeness

When a completeness rule fires, the response displays a Completeness row showing Incomplete alongside a confidence tag. Expanding the row reveals the AI Reasoning, which states exactly which checklist item was missing, and an Assessment Criteria panel that separates items into Incomplete and Pass categories so analysts can see at a glance which elements were found and which were not.

Completeness result showing missing and passing checklist items
 

Confidence Tag

Each result type includes a confidence tag (for example, High Confidence) placed alongside the result value. This indicates how certain the agent is about the result it produced, not about the quality of the responder's submission. A result marked Incomplete with a High Confidence tag means the agent is confident that the checklist item is missing.

Automated Highlighting

For lengthy text grids and large document attachments, the agent automatically highlights the specific row or text passage that triggered the rule. Analysts do not need to manually search through the document to find the relevant section.

Overriding AI Results

Analysts always retain full control over results. If you disagree with the agent's rating or flag, you can override the result directly from the response view using the Override button. Hovering over the button displays the tooltip: Replace the AI's review with your own assessment.

The Override button replaces the AI result with the analyst's own assessment

• Rating: Select a new rating from the override options to replace the AI-assigned value
• Flag: Override the flag to remove it from the response if the AI's assessment does not apply

Audit trail: Overriding a result removes the flag or updates the rating in the active view, but the system logs the original AI result, the override reason, the user who made the change, and the timestamp. The original AI result and its timestamp are strictly preserved in the history for full auditability.

How to Configure AI Mode

AI Mode rules are configured at the template level, before a project begins. Each rule is linked to a specific question and tells the agent what to evaluate when a responder answers that question.

Adding Automations

• From the left navigation panel, click Diligence
• Go to Templates and select the template you want to configure
• In the template preview, click More in the top right action bar
• Select Set Flag / Score from the dropdown menu

Access Set Flag / Score from the More menu in the template preview

• In the Set Flag / Score panel, click + Add to open the Create New Rule modal
• Select the AI Mode tab (alongside Standard and Advanced)

Select the AI Mode tab in the Create New Rule modal

What Types of Content Does It Evaluate?

AI Mode is designed for unstructured question types that standard rules cannot interpret:

• Text paragraph responses
• Attached documents (PDFs and other files)
• Text grids
• Yes/No questions that include a lengthy explanation

How AI Mode Differs from Standard Rules

Standard and Advanced rules evaluate structured data using fixed conditions: if a specific condition is true, a rule fires. AI Mode works differently. You write a prompt in plain English describing what to look for, and the agent evaluates each responder's submission against that instruction. This allows the agent to detect sentiment, identify missing context, and flag nuanced issues that fixed rule conditions cannot capture.

AI Mode Prompt Types

When creating a rule in AI Mode, you select one of three prompt types. Each type performs a distinct action on the responder's submission.

Rating / Scoring

Write a prompt that defines what a High, Medium, or Low response looks like. The agent evaluates the responder's submission against your description and assigns a rating accordingly.

Flagging

Write a prompt that describes the conditions or risks you want the agent to surface. If those conditions apply to the responder's submission, the agent fires a flag.

Completeness

Write a checklist of required items that the responder's submission must include. The agent reads the response or attached document, checks it against your list, and flags it as incomplete if any specific element is missing. The agent also provides a reason stating exactly which item was absent.

Creating an AI Mode Rule

Each prompt type requires a separate rule. To add multiple evaluation criteria for the same question, create multiple rules, one for each type.

• In the Create New Rule modal with AI Mode selected, use the Apply the new rule to dropdown to choose the question the rule should apply to
• Select a Type from the dropdown: Rating, Flag, or Completeness
• Enter a descriptive Name for the rule (for example: Outdated policy flag)
• Write your Prompt in plain English. Use the ghost text examples in the field as a guide for phrasing
• Click Create Rule to save, or Create & Add Another to add another rule for the same question immediately

Example Use Case

Reviewing Governance Documentation Across Responders

An analyst is reviewing submissions from multiple responders and needs to evaluate governance documentation without reading every attachment manually. They set up the following AI Mode rules on the governance question:

• Completeness rule: Checklist includes: named board members, meeting frequency, most recent board meeting date, and governance policy version
• Flag rule: Flag if the governance policy is undated, references a document older than 24 months, or lacks evidence of board approval
• Rating rule: Rate High if all governance checklist items are present and the policy is current. Rate Low if fewer than two items are addressed

When responders submit their DDQs, the AI Review Agent evaluates every attachment against all three rules. The analyst opens the project and sees each response already tagged with a rating, any flags, and a completeness status. For a response flagged as incomplete, the AI Reasoning states the exact missing item, for example: Board of Governance meeting date was not found in the document. The analyst can accept the result, override it with a note, or add a comment before finalising their memo.

Tips for Writing Effective Prompts

• Be specific about what a strong, adequate, or weak response looks like when writing rating prompts
• For flagging, describe the exact condition or risk language you want to surface, not just a general topic
• For completeness, list individual required items clearly rather than grouping them into broad categories
• Use the ghost text examples in the Prompt field as a starting point for structure and phrasing
• If you need to evaluate Rating, Flagging, and Completeness on the same question, create a separate rule for each type
• Review AI Reasoning alongside results, particularly for completeness flags, to validate the agent's findings before overriding