File review sampling: how it works, its limits and full coverage

Many advice firms check a sample of their files. The compliance manager picks a handful each month or quarter, reviews them properly, writes up the findings and moves on. It is a sensible way to spend limited time, and it is how many monitoring plans are written. But a sample answers a narrower question than many firms assume, and the way the files are chosen matters as much as how well they are read.

This guide explains how file review sampling works, the different ways to select files, what a sample can and cannot tell you, and what changes when you move from a sample to checking every file.

What the rules ask for, and what they leave to you

No FCA rule we know of sets a percentage of files a firm must check. What the rules do require is good records and genuine monitoring. SYSC 9.1.1R says "A firm (other than a common platform firm) must arrange for orderly records to be kept of its business and internal organisation… sufficient to enable the FCA to monitor the firm's compliance" (SYSC 9.1). Under the Consumer Duty, PRIN 2A.9.8R requires firms to "regularly monitor the outcomes retail customers receive from: (1) the products the firm manufactures or distributes; (2) the communications the firm has with retail customers; and (3) the customer support the firm provides to retail customers" (PRIN 2A.9).

How you do that monitoring is your decision, and it should be written into your compliance monitoring plan. Sampling is one method. The FCA's own training and competence page adds a useful reminder for supervisors: "We do not expect supervision to only involve file checking" (FCA training and competence). File reviews are one source of evidence among several, however many files you check.

Three ways to choose a file review sample

Random selection

Every file has an equal chance of being picked. A random sample is the fairest way to estimate how the whole book looks, because nobody's judgement has shaped which files are read. Its weakness is that it spreads attention evenly, so the riskiest files get no more scrutiny than the simplest ones.

Risk-based selection

Files are chosen because something about them raises the stakes. This concentrates review time where harm is most likely, but it produces a biased picture of the book: if you only read the risky files, your results will look worse than the whole, and you cannot use them to say how the average file looks.

Stratified selection

The book is split into groups (by adviser, advice type or service status, for example) and files are drawn from each group. Every adviser and advice type is seen, and higher-risk groups can be weighted more heavily.

One practical approach is a mix: a risk-based core, plus a small random draw so the monitoring plan is not only looking where it already expects to find problems. Whatever you choose, write down the method and why.

Building risk-based selection criteria

A good risk list draws on the rules and on the FCA's stated priorities. For advice and wealth firms, the FCA's Consumer Duty focus areas include "Risks to providing unsuitable advice" and "Tackling poor identification of clients with characteristics of vulnerability by wealth managers" (FCA Consumer Duty focus areas). Criteria that follow from the rules include:

• Advisers not yet assessed as competent. TC 2.1.3G says the level and intensity of supervision "will be significantly greater in the period before the firm has assessed the employee as competent, than after" (TC 2.1).
• Pension transfers, conversions and opt-outs. COBS 9.5 requires these records to be kept "indefinitely", which reflects how long the consequences can last (COBS 9.5).
• Clients with characteristics of vulnerability. FG21/1 notes that firms advising on investments and pensions "have an older customer base, so common characteristics of vulnerability may involve health and life events associated with old age" (FG21/1).
• Ongoing service clients with no recent review. Where the firm provides a periodic suitability assessment, COBS 9A.3.9R requires a review "at least annually" (COBS 9A.3).

The limits of file review sampling

A well-designed sample is useful. It still has limits that are worth stating plainly in your monitoring records.

• It says nothing about the files it skipped. A sample can suggest how the book looks. It cannot confirm that any particular unread file is sound, and a complaint or a regulatory review may pick exactly that file.
• Small numbers hide patterns. If you read three files per adviser a quarter, a gap that appears in one file in ten may never be seen, or may be seen once and dismissed as a one-off.
• Selection drifts. Reviewers can drift towards files they expect to be complete. A written method and an occasional random draw help.
• Remediation is harder to test. If you fix a template after a finding, a small sample may not contain enough later files to show whether the fix worked.

Regulators sample too. The FCA's 2017 Assessing Suitability Review looked at 1,142 pieces of advice from 656 firms (FCA Assessing Suitability Review, 2017). That is a sample designed to describe a market, which is a different job from a firm satisfying itself about its own clients.

Sampling vs full coverage at a glance

What full coverage changes

Firms sample because a thorough review by hand takes, by our estimate, around half a day per file. When that first read takes minutes, the economics change, and so does the role of the reviewer.

The biggest shift is the order of work. With a sample, you choose files and then read them. With full coverage, every file gets a first read and score, and the reviewer then chooses which ones need human attention. Risk-based selection does not disappear; it moves from choosing what to read to choosing what to examine closely.

It also improves your management information. The FCA's outcomes monitoring review says firms should show "What their information tells them", "How they use it to identify risks or issues", "What action they take in response" and "How they consider whether those actions have improved outcomes" (FCA outcomes monitoring review). Results across every file make each of those easier to evidence, particularly the last one.

Full coverage does not remove the need for judgement. A first score is a starting point. Reviewers still challenge findings, decide severity and agree actions with advisers, and the firm still owns the conclusions. Our article on checking every compliance file walks through where the half day goes, and preparing for a file review covers what to do with the findings.

Where Scrutineer fits

Scrutineer reads a whole client folder, every subfolder and scanned document included, and gives each file a compliance score against FCA suitability and disclosure standards, a file-quality score against your firm's own standard, and a verdict of Suitable, Unclear or Unsuitable. Every finding names the rule it engages with the passage quoted. A dashboard shows every check with the client, both scores, the verdict and what needs attention, so your reviewers can pick the files that need a human. It flags any review that could only read part of a folder. By our estimate it saves about three and a half hours per file. The decisions, and the responsibility, stay with your compliance team.

Frequently asked questions

How many files should an advice firm sample?

The FCA rules we cite do not set a number. Base the size and method on the risks in your business, write them into your compliance monitoring plan and revisit them when the risks change.

Is risk-based sampling better than random sampling?

They answer different questions. Risk-based selection finds problems where they are most likely; random selection tells you how the whole book looks. A mix of both, with the method recorded, covers more ground than either alone.

Does checking every file mean reviewing every file in depth?

No. Full coverage gives every file a first read and score. Reviewers then spend their time on the files the scores and findings point to, which is a better use of judgement than reading files chosen in advance.

See how Scrutineer gives every file a first score, or book a twenty-minute demo with Adam Booth on Microsoft Teams, run live on a demo firm with nothing to prepare.