AfterDesk
EN
Apply

Working with AI

Accuracy on a batch, not a row

Checking one row proves that one row is correct. It proves nothing about the four hundred and ninety-nine behind it. This course teaches the measurement that does prove something: a rubric written before you look, a random sample of fixed size, hand-scoring that records disagreements, and a threshold that decides whether the run ships. You finish able to state your error rate and defend it.

8
lessons
~66
minutes
12
exam questions

Free · No paid tier · No certificate fee

After this course

Everything, and what is in it.

Why one spot check proves nothing

~8 min

The row you checked is correct

You ran a batch of 500 rows and you opened row seven. The category is right, the evidence quotes real text, nothing is blank. You feel better. What you have learned is that row seven is correct. That is the whole finding. A spot check is a sample of size one, and a sample of size one carries almost no information about the 499 rows you did not open. The feeling it produces is out of all proportion to the fact it establishes. This is not a small mistake in judgement. It is the single most common way a batch job leaves your hands with dozens of wrong rows inside it.

Make the arithmetic intuitive

Put a number on it. Suppose 5 percent of your 500 rows are wrong, which is 25 broken rows sitting in the file. You open one row at random. The chance it is one of the broken ones is 5 in 100, so 95 times out of 100 you see a clean row and conclude the run is fine. Open ten rows and you still miss the problem about 60 times out of 100. Open twenty and you catch it roughly two times in three. Even twenty rows, hand-checked, leaves a real chance that 25 wrong rows walk out of the door. One row leaves a near certainty.

Errors are not spread evenly

The arithmetic above assumes errors are scattered at random. They usually are not, and that makes casual checking worse rather than better. A model handles the plain rows and fails the awkward ones: the vendor with two trading names, the date written without a year, the address that is really two addresses in one cell. Those rows are not distributed evenly through the file. They cluster wherever the source is messy, and messy sources are often sorted, so the awkward rows sit together. If you check the first twenty rows of a file sorted by date, you have checked the twenty oldest and tidiest records. You have measured the easy end and called it the run.

Fluent output hides its own errors

There is a second reason spot checking fails here, and it is not about counting. A wrong row produced by a model does not look wrong. It has the same shape, the same tone, the same confident formatting as the 480 correct rows around it. A category that is plausible but not the one the brief defined reads exactly like a category that is right. An invented quote in an evidence column reads better than a real one, because real source text is usually clumsy. When you scan a sheet for errors, you are scanning for something that looks out of place, and nothing looks out of place. Polish proves nothing.

Measure the run, not your mood

So the discipline this course teaches is a replacement, not an addition. You are not being asked to spot check more carefully. You are being asked to stop treating a feeling of confidence as evidence, and to produce a number instead: this run was scored against a written rubric on 50 randomly drawn rows, and 4 of them failed. That sentence can be checked by someone who was not there. Your confidence cannot. Everything that follows is the machinery for producing that sentence honestly, in an order that stops you from bending it: rubric first, sample second, scoring third, verdict last. Do it in that order and the number means something.

Remember

  • A spot check is a sample of size one. It proves one row.
  • Five percent bad means one row misses the problem ninety-five times in a hundred.
  • Errors cluster in the awkward rows, and awkward rows are rarely at the top.
  • A wrong row from a model looks exactly like a right one.
  • Replace confidence with a number someone else can check.

Write the rubric before you look

~8 min

Grade before you see the answer

The rubric is the written definition of what a correct row looks like, and it has one hard requirement: you write it before you look at any output. This is not ceremony. If you open the file first, you will read what the model produced and your idea of correct will quietly re-shape itself around what you see. A category you would have called wrong at nine in the evening looks defensible at two in the morning, because it is already there, it is already formatted, and the deadline is close. Written first, the rubric is a standard. Written after, it is a description of the thing you were going to deliver anyway.

A rubric is a list of checks

Keep it concrete. A rubric is not a paragraph about quality; it is a short list of tests that can be run against a single row, each answerable yes or no by a person looking at that row and its source. For a 500-row categorization task the list might read: the category is one of the seven named in the brief, the category matches the written definition of that bucket, the evidence column quotes text that actually appears in the source row, the vendor name is copied exactly and not reworded, no required field is empty. Five checks. A person can apply those to a row in under a minute, fifty times in an hour.

One row, one verdict

Score each sampled row as pass or fail, and nothing in between. The temptation is to invent partial credit: this row is mostly right, the category is fine and only the evidence is thin, call it three out of five. Partial credit destroys the measurement, because a rate built from half marks cannot be projected onto the run and cannot be defended to anyone. If a row fails one check, the row fails. That sounds harsh until you remember what you are measuring: whether this deliverable can go out. A client does not receive a row that is three fifths correct. They receive a wrong row, and a wrong row costs the same as any other.

Name the failures you expect

Before you score, write down the failure codes you expect to see, four or five of them, in plain words: wrong bucket, evidence not present in the source, name reworded, field left blank, row skipped entirely. This costs two minutes and pays for itself twice. First, it forces you to think about how this particular batch could go wrong, which sharpens the checks themselves. Second, it means that when you find failures you are recording a code rather than writing a fresh sentence each time, and codes can be counted. Four failures with three sharing one code is a finding. Four failures described in four different sentences is just a bad feeling.

Freeze the rubric, then run

Once the rubric is written, stop editing it. Save it, date it, and treat it as fixed for this run. If scoring reveals that a check was badly worded, you may rewrite it, but then you rescore every row in the sample against the new version, starting from the first one. That rule exists because the alternative is invisible: a rubric loosened at row eleven, tightened at row thirty, and quietly forgotten at row forty-eight produces a rate that measures your fatigue rather than the batch. Rescoring twelve rows is cheap. A rate nobody can reproduce is worthless. Keep the old version as well, because the operator may want to see what changed and why.

Remember

  • Write the rubric before opening the output. Otherwise correct means whatever you produced.
  • A rubric is per-row yes or no checks, not a paragraph about quality.
  • Pass or fail. Partial credit cannot be projected onto five hundred rows.
  • Pre-write four or five failure codes so failures can be counted, not described.
  • Freeze the rubric. If you change it, rescore the sample from row one.

Draw a sample you can defend

~8 min

The first twenty rows are not a sample

Almost every file you receive arrives in some order: by date, by vendor, by whatever the export tool did. That order is information, and it is working against you. The first twenty rows of a file sorted by date are the twenty oldest records, usually entered when the system was new and the data entry was careful. The first twenty rows of a vendor export are often the largest accounts, which are the ones with complete records. Checking the top of the file is comfortable because it is one scroll away, and that comfort is exactly the problem. You are not sampling the run. You are sampling the tidiest corner of it.

Interesting rows are not a sample either

The opposite habit is just as wrong and feels far more diligent. You scan the sheet, your eye stops on the strange ones, and you check those: the blank-looking cell, the row with a category you did not expect, the vendor name in all capitals. Hunting for problems this way is a useful activity and you should do it. It is not a measurement, and you must not report it as one. A hand-picked set of odd rows tells you the worst the batch can do, not how often the batch does it. Keep the two apart in your own head and in your notes: one is a sample, the other is a hunt.

Randomize with a helper column

In a spreadsheet the mechanics take one minute. Add a helper column beside your rows and put a random number in it with the RAND function, filled down the full 500 rows. Then copy that column and paste it back over itself as values, which freezes the numbers so they stop changing every time the sheet recalculates. Sort the whole sheet by that column, smallest first, and take the top 50 rows. That is your sample. Record the row identifiers somewhere separate before you start scoring, so the sample stays fixed even if the sheet gets re-sorted later. Delete the helper column from anything you deliver. It is your working tool, not the client's.

Pull fifty, and know what fifty says

Thirty is the floor for any run worth measuring, and 50 is a sensible working number for a batch of a few hundred rows. Going from 50 to 100 buys you real precision, but it also doubles the slowest part of the job, so spend that time only when the deliverable deserves it. Understand what a clean sample does and does not say. If you score 50 rows and find zero failures, you may not say the run is perfect. You may say there were no failures in 50 rows, which is consistent with a true error rate of up to roughly 6 percent, which on 500 rows is about 30 bad rows. Zero is not zero.

A new run needs a new sample

When the rate comes back bad and you change the instruction and run the batch again, you must draw a fresh random sample from the new output. Reusing the same 50 rows is the most natural mistake in this whole method, and it is fatal, because those are the rows you looked at hardest and, often, the rows your instruction change was aimed at. A second score on the same 50 rows measures how well you patched a known list. It says nothing about the 450 you have still never opened. New run, new draw, every time. Keep both samples in your notes so the improvement can be seen rather than claimed.

Remember

  • Files arrive sorted. The top of the file is the tidiest corner.
  • Hand-picked odd rows are a hunt, not a measurement. Keep them separate.
  • Random column, paste as values, sort, take the top fifty rows.
  • Fifty rows with zero failures still allows roughly six percent bad.
  • Every re-run gets a fresh draw. Rescoring old rows measures your patching.

Score by hand, record disagreements

~9 min

Score against the rubric, not memory

Open the rubric beside the sample and work check by check, row by row. It is slower than reading the row and forming an impression, and the slowness is the point. Working from the written checks, you will fail rows that look fine, which is the whole reason the rubric exists. Do not hold the checks in your head. Fifty rows times five checks is two hundred and fifty small decisions, and by decision one hundred and eighty your memory of check three has drifted toward whatever the last few rows contained. Read the check. Apply the check. Move on. A rubric consulted only at the start is a rubric you stopped using.

Check the source, not the output

For most checks you cannot decide by looking at the delivered row alone. If the check says the evidence column quotes text that actually appears in the source, then you must open the source record and look for that text. This is where invented content is caught, and it is the check people skip because it is the one that takes real time. Budget for it. A 50-row sample where you verified 50 source records is a measurement. A 50-row sample where you read 50 output rows and thought they seemed reasonable is the spot check from lesson one, wearing better clothes. Plausibility is not a check. Presence in the source is.

Record the code, not a feeling

For every failure, write three things in your scoring sheet: the row identifier, the failure code, and one short sentence naming what was wrong. Row 218, evidence not present, quoted phrase about a delayed shipment does not appear anywhere in the source note. That takes fifteen seconds, and it is the raw material for everything in lesson six, because a pile of failures with codes attached will tell you whether you have one instruction problem or five unrelated accidents. Vague records cost you later. Several rows looked off is a note you cannot act on tomorrow morning, and it is certainly not something an operator can use to target a review.

Do not resolve a disagreement silently

Some rows will not fit either verdict. The rubric says the category must match the written definition, and this row is a training company that also sells software, and the definition genuinely does not settle it. The pull in that moment, especially at four in the morning with a deadline, is to make a call and move on. Do not. Mark the row undecided, write one line about which two readings are in play, and keep scoring. Undecided rows are not failures and they are not passes; they are a third pile, and lesson seven is about why that pile is often the most valuable thing you produce. Resolved silently, it disappears.

Score every row you drew

Score all 50. The temptation to stop early comes in two flavours and both are wrong. The first is stopping because the first fifteen rows all passed and the run is obviously fine, which throws away the part of the sample where the awkward rows were waiting. The second is stopping because the first eight rows failed and you already know you have to redo it, which leaves you unable to say how bad the run was or whether your fix helped. If you truly must abandon a score partway, write down that you scored 22 of 50 and stopped, and never report the partial count as a rate. A partial sample is not a sample.

Remember

  • Read each check for each row. A rubric consulted once is abandoned.
  • Verify evidence against the source record. Plausibility is not a check.
  • Log row identifier, failure code, one sentence. Vague notes cannot be acted on.
  • Rows the rubric cannot decide go in a third pile, marked undecided.
  • Score the whole sample. A partial score is never reported as a rate.

Read the rate, set the line

~8 min

Turn the count into a rate

Four failures in a 50-row sample is 8 percent. On a run of 500 rows that projects to about 40 wrong rows in the file you were about to deliver. Do the projection out loud and in writing, because the two forms land very differently. Four failures sounds like a small tidy-up. Forty wrong rows in a client deliverable sounds like what it is. This is the single most useful arithmetic in the course and it takes ten seconds: failures divided by sample size, multiplied by the size of the run. Write the projected count in your notes beside the percentage, and use the count when you talk about the run.

The rate is an estimate, not a fact

Your 8 percent is a measurement of 50 rows, not a property of the batch. Draw a different 50 rows and you might have found two failures, or seven. With a sample that size, four failures is comfortably consistent with a true rate anywhere from about 3 percent to about 17 percent, which is 15 to 85 wrong rows. That range is not a reason to distrust the method; it is a reason to be honest about what the number is. When you are deciding whether to ship, plan against the unfavourable end of the range rather than the middle. When you are reporting, give the count and the sample size so the reader can judge the range too.

Set the threshold from the consequence

There is no universal acceptable error rate, because the cost of a wrong row is not universal. A wrong email address in a 500-contact outreach list produces a bounce, a wasted send, and possibly a message to a stranger; the tolerable rate there is effectively zero and those fields must be checked individually. A judgement column of topic tags on the same 500 rows, which the client will sort and eyeball anyway, can survive a few percent. Decide which fields in this deliverable are zero tolerance, usually money, dates, names, and anything used to contact a person, and which are judgement fields. Then write one threshold per class into your rubric, before scoring.

The threshold ends the discussion

Scoring is supposed to end in a verdict, not a conversation. You wrote 2 percent on judgement fields and zero on contact fields. You measured 8 percent. The run is not deliverable. That is the entire decision and it should take no time at all. What must not happen next is the compromise that feels reasonable at five in the morning: repairing the four rows you found, delivering the file, and mentioning in the note that a few rows needed fixing. You measured 8 percent and you fixed four rows. The other thirty-six are still sitting in the file. The rate has not moved, and the note is now misleading.

The line does not move after the score

Write the threshold into the rubric in the same sitting as the checks, before any output exists to argue with. Once you have a rate and a deadline, every threshold looks negotiable, and the negotiation always runs the same direction. Nobody has ever tightened a threshold at four in the morning. If you genuinely believe the line you set was wrong, that belongs in a note to the operator explaining why, not in a quiet edit to your own scoring sheet before delivering. The whole value of a written threshold is that it was decided by a version of you who had nothing to lose by being strict.

Remember

  • Failures over sample size, times run size. Report the projected count.
  • Four in fifty is consistent with roughly three to seventeen percent.
  • Zero tolerance on money, dates, names, contacts. Judgement fields can carry more.
  • Threshold plus rate equals a verdict, not a discussion.
  • Set the line before scoring. Nobody ever tightens it afterwards.

Fix the instruction, not the rows

~9 min

Patching your sample fixes nothing

The rate came back at 8 percent and you have the four failing rows in front of you, each one repairable in about a minute. Repairing them is the wrong move, and it is wrong in a specific way that is worth naming. Those four rows are not the problem; they are the four you happened to draw. Behind them stand roughly thirty-six rows you never opened, produced by the same instruction and failing in the same way. Fixing the sample changes your file by four rows and changes the true error rate by almost nothing, while destroying your ability to measure it again. You have edited the evidence. The run is still broken.

Read the failures for a pattern

This is where the failure codes earn their keep. Lay the four failures side by side and ask one question: do they share a cause? Three of them coded wrong bucket, all three involving companies that both train and sell software, is not four accidents. It is one instruction that never told the model what to do when a vendor spans two buckets. That is a five-line fix in the instruction, and it will move all thirty or so of the same rows across the whole run. Four failures with four unrelated codes is a different situation, usually meaning the task is a poor fit for batch work, and the last section here covers what to do then.

Change one thing, then run it all again

When you find a pattern, change the instruction to address it, and change one thing at a time. Then re-run the entire batch, not just the rows that failed. Running only the failures leaves you with a file assembled from two different instructions, which is impossible to describe honestly in a delivery note and impossible for anyone to reproduce. After the re-run, draw a new random sample and score it against the same frozen rubric. You now have two numbers, 8 percent and, say, 2 percent, and the improvement is a measurement rather than a hope. Two rounds like this is normal. Budget for them when you claim the task.

Carve out what cannot be automated

Some failures are not instruction problems at all. If 60 of the 500 rows have a missing country field, no wording will conjure the country out of a blank, and every attempt will produce a confident guess, which is worse than a blank. The right move is to carve those rows out and handle them as their own small job: fill them by hand from the source where the answer exists, and list them as unresolved where it does not. Sixty rows done by hand is an hour of honest work. Sixty rows filled by a confident guess is sixty errors that look like data, and they are the kind QC finds and the client remembers.

Know when to stop and work by hand

If two rounds of instruction changes have not brought the rate under your threshold, stop. Do not start a third round at five in the morning with three hours left. At that point the honest reading is that this task is not a batch task tonight, and you have two choices: do the work by hand at a pace you know you can hold, or, if the remaining time cannot cover it, tell the operator now rather than at the deadline. Both of those are recoverable. Delivering a run you measured at 8 percent while calling it checked is not, because it will be checked again by someone who did not build it.

Remember

  • Repairing the sampled failures edits your evidence and leaves the run broken.
  • Three failures sharing one code is one instruction problem, not three accidents.
  • Change one thing, re-run everything, draw a new sample, rescore.
  • Blank source fields cannot be automated. Carve them out and work by hand.
  • Two rounds without hitting the threshold means stop and tell the operator.

The disagreements are the findings

~8 min

Undecidable rows are not your failure

Three rows in your sample of 50 could not be scored, because the rubric genuinely did not settle them. It is easy to read that as a defect in your rubric, and sometimes it is. More often it is something better. The rubric could not decide because the brief could not decide, and the brief could not decide because the client has never had to state a rule for this case. A vendor that both trains staff and sells the software they train on has no home in a bucket list written by someone thinking about ordinary vendors. You did not fail to categorize it. You found the edge of the client's own definition.

Three in fifty means thirty in five hundred

Project the undecided pile exactly as you project the failures. Three undecided rows in a sample of 50 implies roughly 30 rows across the run sitting in the same ambiguity. That number matters more than it looks. Thirty rows is not an edge case; it is 6 percent of the deliverable resting on a rule nobody has written. Whatever you decide for those rows, you are deciding for thirty of them, silently, on behalf of a client you will never speak to. Doing that quietly is how a delivery passes review and then produces a complaint two weeks later, when someone downstream sorts by category and finds a group they do not recognize.

Group them, do not list them

Do not hand over three row numbers and a shrug. Name the class instead. A vendor that sells both training and the software it trains people on, where the bucket definitions treat those as separate, is something a person can make one decision about. Rows 218, 344 and 401 are unclear is not. Grouping also checks you: if you can state the class in one sentence, you understand the ambiguity, and if you cannot, you probably have three unrelated problems rather than one pattern. Once the class is named, filter the full file and count how many rows belong to it. A named class with a count is a decision request.

Send the question up, not sideways

You never contact the client, so the question travels through the operator, and it has to arrive complete enough to answer without a second exchange. Four parts: the class in one sentence, the count of affected rows, the default you applied in the meantime, and what would change if the decision goes the other way. If dual vendors belong in Software rather than Training, roughly 30 rows move and the category totals shift by about 6 percent. That short paragraph turns a vague worry into a question with a yes or no answer, and it is the difference between a targeted review and a task sent back for clarification.

Apply a default, and label it

Do not deliver blanks while you wait. Blanks in a category column read as unfinished work, and they force whoever opens the file to guess what happened. Choose the more defensible reading, apply it consistently to every row in the class, and label those rows so the decision can be reversed in one pass. A flag column with a single value, dual vendor default, lets an operator or a client filter all 30 rows and change them together in a minute if the call goes the other way. The rule is simple: never leave a hole, never hide a choice. A labelled default is a decision anyone can undo.

Remember

  • The rubric could not decide because the brief never had to. That is a finding.
  • Three undecided in fifty is about thirty rows across the run.
  • Name the class in one sentence. Row numbers alone cannot be answered.
  • Send the class, the count, your default, and what a decision would change.
  • Never leave blanks. Apply a labelled default that can be reversed in one pass.

Report the method with the delivery

~8 min

A checkable rate beats a confident claim

The operator reviewing your 500 rows was not there while you built them and cannot re-derive the whole file. What they can do, in five minutes, is take your rubric, pick three of your sampled rows, and see whether your scoring holds. That is why the method travels with the delivery. A note saying the file has been checked gives the reviewer nothing to act on, so they must either trust you or open rows at random, which puts them back in lesson one. A note saying which 50 rows you scored, against which checks, with which result, turns their review into a short verification. Checkable beats confident, every time.

Put six items in the note

Six items, in this order, and it fits in a short paragraph. What you produced and how, in one sentence. The rubric, attached or pasted, with its checks and thresholds. The sample: how many rows, how they were drawn, and the row identifiers. The result: the number of failures, their codes, and the rate. The verdict against your threshold. The undecided classes with their counts and the defaults you applied. Six items, perhaps 150 words, and every one of them is a fact rather than a reassurance. Write it as you go rather than at the end, because the details you need are exactly the ones that evaporate once the file is finished.

Point the reviewer at the right rows

Give the operator the row identifiers of your sample and of every failure you found and repaired. This feels like handing over the evidence against yourself. It is the opposite. A reviewer with a list of rows can check the places where problems actually were, confirm your repairs, and approve. A reviewer with no list opens rows at random, and random rows in a 500-row file mostly look fine, so the review either passes on a false impression or drags on while they hunt. Targeted review is faster for them and safer for you, and it is the only version of review that reliably catches anything. Blind review catches little and takes longer.

Say what tool did what

State plainly which parts of the work were produced with a tool and which were done by hand, because a reviewer reads a machine-produced column differently from a typed one, and they should. Two standing rules govern this. Client data does not go into third-party or consumer tools: not a chat assistant you signed up for, not an online transcription site, not a document summariser. Our own in-product assistant, available on a task you have claimed, is sanctioned for method questions; it does not see the task record, the client, the files, or the price, only what you type. A brief may grant written permission for a named external tool for a named purpose. Silence is not permission.

Understate rather than round down

When you write the rate, write the number you measured, not the impression you want to leave. A few rows needed correction is not a rate, and it will read, correctly, as an attempt to avoid one. Measured 8 percent on 50 rows, changed the instruction, re-ran, measured 2 percent on a fresh 50 is a stronger delivery note than any adjective, and it is stronger precisely because it admits the first number. Nobody is grading you on a perfect first run. You are being trusted with 500 rows on the strength of one thing: that the number you report is the number you found. Overstate your certainty once and that is gone.

Remember

  • Ship the method with the file. A checked claim gives the reviewer nothing.
  • Six items: what you ran, rubric, sample, result, verdict, undecided classes.
  • Hand over the sampled and failed row identifiers. Blind review catches little.
  • Client data stays out of outside tools. A brief may grant named written permission.
  • Report the rate you measured, including the bad first one.

Sit the exam

The courses are free. The work is real.

Twelve scenario questions. Pass at ten. Three attempts a day. The bar is the point.

No paid tier. No certificate fee. No upsell. Not now, not later.

Create a free account

You can start the first course tonight.

The curriculum

Everything, and what is in it.