Home Blog Page 5

How to Compare Prices Using the Same Unit

0
Different-sized ceramic jars and glass bottles sit on a comparison scale above small measuring cups.

The smaller pack costs less at the checkout. The larger pack claims to be better value. Neither statement tells you the price of the same quantity.

The useful habit: put competing offers into one common unit before deciding which represents better value for your needs.

Start with price divided by quantity

Unit price equals the relevant price divided by the quantity. Choose a unit that makes the comparison easy: per kilogram, per litre, per 100 grams or per item.

Consider two fictional packs of the same food:

Pack Price Quantity Price per 100 g
A 180 currency units 500 g 36
B 250 currency units 750 g About 33.33

For A: 180 ÷ 500 × 100 = 36. For B: 250 ÷ 750 × 100 ≈ 33.33. B has the lower unit price, while A requires less money today.

Make the units match

A price per kilogram cannot be compared directly with a price per 100 grams. Divide the per-kilogram price by ten to express it per 100 grams. A litre contains 1,000 millilitres, so use the same conversion discipline for liquids.

Do not convert grams to millilitres without information about the product’s density. Mass and volume measure different quantities.

Compare the quantity you can use

A cheaper bulk pack can be poor value if part of it spoils. Imagine paying 300 units for a kilogram but using only 600 grams. The effective cost of the amount you used is 50 units per 100 grams.

This does not mean small packs are always better. It means storage, shelf life and your actual consumption belong in the decision. A lower shelf unit price is one input, not a requirement to buy more.

Check whether the products do the same job

For comparable products, price per mass or volume is a useful starting point. But concentration, formulation, quality and usable yield can matter. A concentrated cleaner and a ready-to-use cleaner may need a comparison based on the prepared amount or the manufacturer’s instructed dose.

Likewise, “per serving” is only comparable if serving sizes and the relevant contents match. A manufacturer can define a smaller serving without making the package cheaper per gram.

For a fictional detergent, one bottle costs 240 units and provides 24 instructed washes; another costs 300 and provides 40. Their stated costs are 10 and 7.5 units per wash, assuming the instructions fit your use. Do not increase or reduce chemical doses beyond safety instructions just to improve the arithmetic.

Include the costs attached to the offer

Delivery, mandatory accessories, membership requirements and conditional discounts can change the comparison. Use the total relevant cost rather than comparing one seller’s delivered price with another seller’s item-only price.

For a multipack, count the total quantity: four 250 g packs contain one kilogram. Check whether a promotion requires buying more than you need or committing to a recurring purchase.

Watch for shrinking packs

If a product keeps the same price but its quantity falls, the unit price rises. A fictional 200-unit pack shrinking from 1,000 g to 900 g moves from 20 to about 22.22 units per 100 g, an increase of about 11.1%.

The packet price did not change, but the amount received did. This is one reason consistent unit prices help track value over time.

A quick shopping checklist

  1. Check the exact quantity and measurement unit.
  2. Calculate or read the unit price using the same unit.
  3. Include relevant extra charges and discount conditions.
  4. Compare quality, concentration and usable yield.
  5. Consider waste, storage and the amount you can afford today.
  6. Choose the option that fits the job, not simply the largest pack.

Note to keep: unit pricing makes quantity comparisons fairer. It does not decide your preferences, budget or needs for you.

Related: read percentage changes accurately and understand numerical summaries.

Sources

All prices above are fictional arithmetic examples, not current offers or local pricing requirements.

Relative Risk vs Absolute Risk: How to Read a Dramatic Headline

0
A magnifying lens enlarges coral pebbles while a full bowl of mostly ivory pebbles remains visible.

“Risk increases by 50%” can sound alarming. But 50% of what starting risk, over what period and in which group? Without those details, the headline leaves out information you need.

The useful habit: ask for the starting risk and the changed risk using the same denominator and time period.

Absolute risk describes the chance of an event

If an event occurs in 2 out of 1,000 people over a specified period, its observed frequency in that group is 0.2%. A risk estimate uses information about a defined group and period; it is not a prediction of exactly what will happen to a particular person.

The time period matters. A one-year risk and a lifetime risk cannot be treated as interchangeable numbers.

Relative risk compares two risks

Relative risk, also called a risk ratio, divides the risk in one group by the risk in a comparison group. A ratio of 1 means the two risks are equal. A ratio of 1.5 corresponds to a 50% relative increase.

Consider these entirely fictional examples. They do not describe a real disease, medicine or exposure.

Example Starting risk Changed risk Relative increase Absolute difference
A 2 in 1,000 3 in 1,000 50% 1 additional event per 1,000
B 200 in 1,000 300 in 1,000 50% 100 additional events per 1,000

The relative increase is identical, but the absolute differences are very different. Neither description should replace the other. Together they show the size of the comparison more clearly.

Keep percentages and percentage points separate

In Example A, risk rises from 0.2% to 0.3%. That is an increase of 0.1 percentage points and a 50% relative increase. It is not an increase of 50 percentage points.

Use the same discipline for reductions. A fictional drop from 10 in 1,000 to 5 in 1,000 is a 50% relative reduction and an absolute reduction of 5 in 1,000, or 0.5 percentage points.

Our percentage-points guide explains the arithmetic in more detail.

Use equal denominators

Comparing “1 in 100” with “5 in 1,000” is easier after rewriting the first as “10 in 1,000.” The changed risk is half the starting risk, rather than five times as large.

Natural frequencies can make a comparison easier to read, but keep the groups and periods comparable. An annual rate based on person-time is not automatically the same as a simple proportion of people over a fixed period.

Ask what the study can establish

An association between an exposure and an outcome does not, by itself, establish that the exposure caused the difference. Differences between groups, measurement problems and other limitations can affect the result.

Read whether the comparison came from an experiment or an observational study, how participants were selected and which factors researchers considered. See correlation versus causation for that distinction.

Read uncertainty and the exact measure

A reported risk ratio is an estimate. Look for uncertainty intervals and the number of events, especially when events are rare. A dramatic ratio based on very few events may be imprecise.

Also check whether the study actually reports a risk ratio, odds ratio or hazard ratio. These measures are related but not interchangeable. Do not automatically translate “twice the odds” into “twice the probability.”

A five-question headline check

  1. What is the baseline risk?
  2. What is the risk in the comparison group?
  3. What population and time period do both numbers cover?
  4. How uncertain is the estimate, and what did the study design establish?
  5. What benefits, harms or alternative outcomes are missing from the headline?

A small absolute increase can still matter when an outcome is serious or many people are exposed. A large relative change should not automatically create panic, and a small absolute change should not automatically be dismissed.

Note to keep: risk communication is most useful when it gives both the comparison and its scale. For personal treatment decisions, discuss the evidence and your circumstances with a qualified clinician.

Sources

Average vs Median: Why One Number Cannot Describe Everyone

0
Small ceramic homes and a tall glass tower sit on a balance beam, illustrating the influence of an extreme value.

“The average is 40” sounds informative until you discover that most observations are close to 10. A single summary can be correct while describing very few individual experiences.

The useful habit: ask which average is being reported and what the values look like around it.

Mean and median answer different questions

The arithmetic mean adds the values and divides by their count. The median identifies the middle of the ordered values. Both are measures of centre; neither describes the whole distribution.

Consider these fictional waiting times, in minutes: 10, 10, 15, 15, 100.

Measure Calculation Result
Mean (10 + 10 + 15 + 15 + 100) ÷ 5 30 minutes
Median The third value after ordering 15 minutes
Range 100 − 10 90 minutes

No one in this example waited exactly 30 minutes. That does not make the mean wrong. It shows that the mean is a calculated centre, not necessarily a value experienced by a typical individual.

See what an extreme value changes

Replace the 100-minute wait with 200 minutes. The mean becomes 50 minutes, while the median remains 15. The larger delay matters greatly for total waiting time, even though the middle observation does not move.

This is why neither measure should automatically win. If you are planning the total staff time needed to handle cases, the long waits matter. If you are describing the middle customer’s wait, the median answers a different useful question.

What happens with an even number of values?

For an even-sized numerical dataset, the usual median is the mean of the two middle values. In 10, 20, 30, 80, the median is (20 + 30) ÷ 2 = 25.

That median is not an observed value either. Also, when values repeat, it is too simplistic to say exactly half of observations are strictly above the median and half strictly below it.

Spread changes the meaning

Two fictional teams could both have a mean turnaround time of 20 minutes. One handles every case in about 20 minutes. The other alternates between very quick and very slow cases. The same centre hides different levels of predictability.

Ask for a range, quartiles or relevant percentiles alongside the average. For a service, a high percentile can help describe longer waits that the median alone leaves out. Read the definition and sample size rather than assuming a percentile guarantees a maximum.

Check who is included

An “average salary” may concern full-time employees, all employees, a particular occupation or a selected sample of respondents. Hourly pay and annual pay answer different questions. Gross and take-home pay are not interchangeable.

A change in the group can change its average even if no individual’s circumstances change. For example, adding more senior employees to a fictional workforce can raise its mean pay without giving existing employees a rise.

Do not casually average averages

Suppose one group of two people has a mean of 10 and another group of eight has a mean of 20. The combined mean is not 15. Weight by group size: (2 × 10 + 8 × 20) ÷ 10 = 18.

Without the group sizes, combining means can be misleading. Combining medians is even less straightforward; group medians and counts alone generally do not determine the overall median.

A checklist for the next average you read

  1. Is it the mean, median or another measure?
  2. What units, period and population does it cover?
  3. How many observations are included?
  4. Could extreme values or a changing group affect it?
  5. How widely are the values spread?
  6. Does the claim concern the whole group or an individual?

Note to keep: an average is a useful compression of information. Ask what that compression leaves out before using it to describe everyone.

Related: how charts mislead and percentages versus percentage points.

Sources

How GPS Finds You—and Why It Sometimes Gets Your Location Wrong

0
A miniature city beneath satellites has a central location pin and a light beam reflected by a glass tower.

The blue dot on a map looks precise. Yet it may jump across a road, drift beside a tall building or place you near the wrong entrance. Understanding the layers behind it helps explain what went wrong.

The useful distinction: estimating your position, drawing that position on a map and choosing a route are separate tasks.

Satellites broadcast; your receiver calculates

GPS satellites transmit signals containing timing and orbital information. A receiver uses the signals’ travel times to estimate its distance from satellites whose positions are known.

For an ordinary three-dimensional solution, measurements from at least four satellites allow the receiver to estimate position and correct its clock offset. The calculation uses distance relationships rather than a satellite taking a photograph of you.

GPS reception itself is one-way: the receiver does not have to transmit your location to those satellites. A phone app may separately share your location through an internet connection, subject to its settings and permissions.

Why timing matters so much

Radio signals travel extremely quickly. A tiny timing error can create a meaningful distance error. The system therefore depends on careful clocks, orbital information and corrections.

A helpful analogy is working out where you are from distances to several known landmarks. The real satellite calculation is more complex, but the important idea is that several measurements constrain the possible position.

The signal reaching you can be imperfect

Source of difficulty What it can do
Buildings, roofs and terrain Block or weaken useful signals
Reflections from surfaces Make a signal follow a longer path, called multipath
Satellite geometry Make measurement errors affect the position estimate more strongly
Atmospheric effects Alter signal travel time
Receiver and antenna design Change how well signals can be received and processed

GPS.gov notes that real-world accuracy depends on these factors. An open-sky result is not a guarantee of the same accuracy inside a shopping centre or between tall towers.

Your phone may use more than GPS

Modern phones can combine satellite navigation with other available information, such as Wi-Fi, cellular signals and motion sensors. Some receive several satellite constellations, not only the US GPS system.

These additional inputs can help, but their availability and quality vary. The map’s uncertainty display is worth reading when provided. A small dot symbol does not mean the underlying estimate is exact.

Sometimes the map is wrong

Imagine that your position is correctly estimated beside a building, but the map marks its entrance on the opposite side. The resulting arrival instruction can be wrong without a satellite-positioning failure.

Likewise, a road closure, misplaced business listing or outdated road geometry can produce a bad route. Report map-data problems through the relevant mapping provider instead of assuming every navigation error has the same cause.

Location is not heading

A stationary phone can know roughly where it is while displaying an uncertain direction. The orientation arrow can involve compass and motion information as well as location history.

If the dot is plausible but the arrow is wrong, check the app’s official calibration guidance and possible magnetic interference. Do not walk or drive into danger merely to see whether the arrow catches up.

Practical checks when a location looks wrong

  1. Compare the map with visible landmarks and signs.
  2. Check that the app has the location access needed for your chosen feature.
  3. Where safe, move to a place with a clearer view of the sky and allow the estimate to settle.
  4. Check whether the error concerns position, heading, the destination pin or the route.
  5. Update or report incorrect map information through the provider.
  6. For safety-critical navigation, use appropriate equipment and independent checks.

A downloaded map can help when mobile data is unavailable, but offline mapping and satellite reception are different capabilities. Downloading a map does not remove signal blockage, and receiving GPS does not automatically provide current traffic information.

Note to keep: a location estimate is evidence to interpret alongside your surroundings, not an instruction to ignore them.

Sources

How Noise Cancellation Works—and Where Its Limits Begin

0
Sculptural headphones surround a quiet garden while coloured sound ribbons flatten near the earcups.

Switch on active noise cancellation and a steady engine hum may become much quieter. A nearby conversation or sudden clatter may still reach you. That difference is part of how the technology works, not necessarily a fault.

The useful distinction: blocking sound physically and reducing it electronically are separate processes that often work together.

Passive isolation starts with the fit

An earcup or ear tip forms a physical barrier that reduces some outside sound before electronics become involved. Gaps can weaken that barrier. Glasses, hair, worn cushions or poorly fitting tips may affect the seal.

Imagine a window that closes almost completely but leaves a narrow gap. Improving the material elsewhere does not remove that opening. The analogy is imperfect, but it explains why fit can matter as much as a setting in an app.

Active noise cancellation adds a controlled sound

Microphones measure sound, and electronics generate a carefully timed signal through the headphone drivers. That signal reduces part of the unwanted pressure variation near the listener’s ears.

This is often described as producing an opposite wave. Real products continuously adapt to changing conditions; they are not deleting sound from the entire room. The reduction occurs in a limited listening region and depends on timing, fit and the system’s design.

Why steady sounds are easier

Sony’s guidance explains that its noise cancellation is especially useful for low-frequency background noise such as engines and air conditioning. Higher-frequency or rapidly changing sounds, including parts of speech, can remain more noticeable.

Performance differs across products and situations. “I can still hear a voice” does not establish that ANC is doing nothing. Compare the background sound with cancellation enabled and disabled, keeping playback volume unchanged.

Situation What to expect
Steady vehicle hum Often a noticeable reduction, depending on the product
Nearby conversation Some sound may remain intelligible
Sudden impact or clatter No guarantee of complete removal
Poor ear-tip or earcup seal Weaker overall isolation or altered cancellation
Wind over microphones Possible unwanted noise; check supported wind settings

Listening cancellation is not call-microphone processing

A headset may advertise both ANC and background-noise reduction for calls. The first concerns what you hear. The second concerns what the other person receives from your microphone. Strong performance in one does not prove strong performance in the other.

Test them separately. Listen in a safe, familiar environment, then make a sample call or recording where permitted. Check the relevant device settings rather than assuming one “noise” button controls both functions.

A simple troubleshooting sequence

  1. Confirm that ANC is enabled rather than an ambient or transparency mode.
  2. Check the physical fit and condition of cushions or tips.
  3. Use the manufacturer’s fit test if the model offers one.
  4. Check automatic mode switching and wind-related settings.
  5. Follow the approved cleaning instructions without blocking or probing microphones.
  6. Compare ANC on and off with the same sound and volume.

Use a moderate listening level. Raising music volume to overpower a noisy environment is not a good substitute for an appropriate fit or quieter setting.

Know the safety boundary

Consumer ANC headphones are not automatically certified hearing protection. For hazardous workplace or recreational noise, use protection rated and selected for that exposure. ANC can also reduce awareness of traffic, announcements or warning signals, so choose when and where to use it carefully.

Transparency features can help bring outside sound through the headphones, but they are not a guarantee that every warning will be heard. Where awareness is essential, removing the headphones may be the appropriate choice.

Can ANC work without music?

Many models can operate cancellation without playback, provided they are powered and configured appropriately. Check your model’s instructions; automatic power-off and connection behaviour differ.

Note to keep: good cancellation can reduce a tiring background without producing complete silence. Judge the technology against the sound you need to reduce and the environment in which you will use it.

Sources

Watts, Watt-Hours and Kilowatt-Hours Explained Simply

0
A glowing reservoir supplies a miniature town through a controlled spout beside an unmarked clock.

A lamp rated in watts, a battery labelled in watt-hours and an electricity bill measured in kilowatt-hours are describing related quantities. Mixing them up makes it difficult to estimate running time or compare energy use.

The useful distinction: power is the rate of using or transferring energy. Energy is the amount used or stored.

Watts describe the rate

A watt, written W, is a unit of power. A kilowatt is 1,000 watts. If an appliance draws 1,000 W continuously, it is using energy at twice the rate of one drawing 500 W.

That does not tell you the total energy used until you know how long it runs. A powerful appliance used briefly may consume less energy than a modest device left on all day.

Watt-hours include time

For constant power, energy in watt-hours equals power in watts multiplied by time in hours. Divide watt-hours by 1,000 to get kilowatt-hours.

Energy (Wh) = power (W) × time (h)

Fictional example Calculation Energy
10 W lamp for 5 hours 10 × 5 50 Wh = 0.05 kWh
1,000 W heater for 30 minutes 1,000 × 0.5 500 Wh = 0.5 kWh
100 W device for 10 hours 100 × 10 1,000 Wh = 1 kWh

Notice that half an hour is 0.5 hours, not 30 hours. Convert minutes before multiplying.

Compare a short burst with an all-day load

Suppose a fictional kettle draws 2,000 W for three minutes. Three minutes is 0.05 hours, giving 100 Wh, or 0.1 kWh. A 20 W device running for ten hours uses 200 Wh, or 0.2 kWh.

The smaller device uses more energy in that comparison because it runs much longer. This does not establish which appliance is more efficient at its job; it compares the energy in two specified uses.

Why the label is only a starting point

Many devices do not draw their rated power continuously. Refrigerators cycle, computers change demand with workload and heaters may switch under thermostat control. A maximum input rating is not necessarily the average power during your day.

For a better estimate, use an appropriate energy monitor or the manufacturer’s measured consumption information. Follow electrical safety guidance and the monitor’s rating; do not connect unsuitable equipment merely to measure it.

Use watt-hours to estimate battery runtime

In an ideal calculation, a 60 Wh battery supplying a constant 20 W load could run for three hours: 60 ÷ 20 = 3. Real runtime can be shorter because usable capacity, conversion losses, reserve limits, temperature and changing demand matter.

A charger rated at 65 W does not mean the device consumes 65 W every moment or that its battery stores 65 Wh. Charging power and stored energy are different specifications.

Similarly, milliamp-hours cannot be compared as energy without considering voltage. Watt-hours are often a clearer basis for comparing energy capacity. See why batteries lose capacity for the ageing side of runtime.

Estimate the energy portion of a bill

Multiply kWh by the applicable price per kWh. For a purely hypothetical flat rate of 8 currency units per kWh, using 0.5 kWh costs 4 units for energy.

Your actual bill may include different tariff bands, fixed charges, taxes and other components. The example is arithmetic, not a statement of any current electricity tariff.

A reusable calculation checklist

  1. Identify average power in watts, not merely an unrelated output rating.
  2. Convert running time to hours.
  3. Multiply watts by hours for Wh.
  4. Divide by 1,000 for kWh.
  5. Apply the correct tariff or use the result to compare like-for-like scenarios.
  6. Record assumptions about cycling, efficiency and changing demand.

Note to keep: W and kW tell you how fast; Wh and kWh tell you how much. A kilowatt-hour is energy, not “kilowatts per hour.”

Sources

Why Rechargeable Batteries Lose Capacity Over Time

0
An artistic glass battery greenhouse contains glowing paths, some narrowed by deposits, symbolising battery ageing.

A phone can show 100% charge and still last much less time than it did when new. The percentage describes how full the battery is relative to its current usable capacity, not a promise that its original capacity remains.

The useful distinction: charge level, battery health and device energy use are different measurements.

What changes inside a rechargeable battery?

Many phones and laptops use lithium-ion batteries. Charging and discharging move lithium ions between electrodes. Over time, unwanted chemical reactions and changes in battery materials reduce the amount that can participate effectively and make energy transfer more difficult.

The process is not perfectly reversible. Recharging restores the state of charge; it does not rebuild all the original materials. The detailed degradation mechanisms depend on chemistry, design and operating conditions.

Time and use both matter

Battery ageing includes the effects of cycling and the passage of time. A battery can age even when it is not used heavily. Temperature history and charging patterns also matter, so two devices bought on the same day may not age identically.

A charge cycle is not necessarily one visit to a charger. Manufacturers commonly count accumulated use equivalent to a full discharge. Follow the definition and service guidance for your particular device rather than applying one universal cycle limit.

Separate three possible explanations

Observation Possible explanation Useful check
Gradually shorter runtime Reduced usable capacity or changing workload Battery-health estimate and comparable usage
Poor performance in cold weather Temporary temperature-related limitation Behaviour after returning naturally to the normal operating range
Sudden drain after an app change More screen, network or processing demand Built-in battery usage records
Unexpected shutdown under load Power-delivery limitations or another fault Manufacturer diagnostics and service advice

Capacity and peak power are different

Capacity concerns how much energy the battery can provide over a discharge. Peak power concerns how quickly it can deliver energy when the device demands it. An ageing battery may struggle with a short burst of demand even when the display still shows charge remaining.

Apple explains that battery impedance, charge level and temperature can affect iPhone power delivery and performance. That is a device-specific implementation, not a description of every product’s software.

Temperature can change today’s result and long-term ageing

High temperatures can accelerate degradation. Cold conditions can temporarily reduce performance; recovery after returning to the permitted temperature range is different from restoring capacity lost through chemical ageing.

Do not try to fix a cold battery with a heater or a hot battery with a freezer. Follow the manufacturer’s operating and charging temperature instructions. Device charging protections exist for a reason.

Investigate before blaming the battery

  1. Compare similar days: screen time, brightness, mobile signal and demanding apps can change energy use.
  2. Read the built-in battery usage report to identify unusual activity.
  3. Check battery-health and service messages where provided.
  4. Consider recent updates, new apps and environmental changes.
  5. Seek authorised or qualified service if symptoms persist or the device reports a fault.

For a fictional example, a laptop that once lasted six hours editing text may last much less during video rendering even with an unchanged battery. Runtime is an interaction between stored energy and the rate at which the device uses it.

Useful habits without obsessive charging rules

Avoid unnecessary heat, use suitable charging equipment and consider built-in charge-management options recommended for your model. You do not need to force a lithium-ion battery to empty every day to “train” it.

No charging routine stops ageing entirely. A practical goal is reliable use within the device’s guidance. Battery replacement can be more sensible than replacing an otherwise suitable device.

If a battery swells, leaks, becomes unusually hot or is physically damaged, stop using and charging it and seek appropriate service guidance. Do not puncture, compress or attempt to repair the cell yourself.

Note to keep: 100% charge means full for that battery now. It does not mean the battery is chemically new.

Sources

Where Should Human Approval Sit in an Automated Workflow?

0
A miniature conveyor carries parcels through a checkpoint with a separate return track and repair station.

An automated system can prepare a useful result and still take the wrong action with it. Drafting a customer reply is different from sending it. Identifying an outdated file is different from deleting it.

The useful habit: place review before the consequential action, when the reviewer can still change the outcome.

Map preparation, decision and action

Imagine a fictional support workflow: read a request, summarise it, propose a refund, update the account and notify the customer. These steps do not have the same consequences. A poor summary can be corrected internally; an incorrect refund or promise can affect money and trust.

Write down where information is merely prepared and where the outside world changes. Those boundaries are natural places to consider approval, validation or a restriction on what the system can do.

Choose review points by consequences

Workflow step Possible control
Format an internal draft Automate within defined constraints and sample-check
Interpret an ambiguous request Ask for clarification or route to a reviewer
Make a commitment to a customer Show the exact commitment before sending
Change access or sensitive records Require an authorised reviewer and enforce permissions
Perform an irreversible operation Use a specific approval, recovery planning and appropriate technical safeguards

This table is a design example, not a universal approval policy. Your organisation’s requirements, the system’s demonstrated reliability and the consequences of error determine the actual controls.

Make the approval concrete

“Continue?” is a poor review request if the person cannot see what will happen. Show the proposed action, recipient or destination, relevant data, reason and important uncertainty.

For the fictional refund workflow, the reviewer should see the order, amount, policy basis and exact customer message. Approval of one amount should not silently authorise a different amount after the underlying data changes.

Bind approval to the specific action and parameters. If the proposed action changes materially, the previous review may no longer apply.

Give the reviewer a real choice

A useful checkpoint supports approve, edit, reject and request more information. The reviewer needs enough time, knowledge and authority to choose. A person clicking through hundreds of vague alerts may provide little meaningful oversight.

Reduce unnecessary interruptions by automating well-defined, low-consequence preparation. Escalate uncertainty and exceptions clearly. This helps reserve attention for the decisions where it matters.

Approval is only one layer

OWASP’s guidance on excessive agency emphasises restricting available functions, permissions and autonomy, with human approval for high-impact actions. A model should not receive broad account access merely because a person may review its work later.

Enforce limits in the systems that execute actions. A read-only task should not require deletion privileges. A workflow proposing an action should not be able to bypass its own approval gate by using another tool.

Design the failure path

  1. Define stop conditions. Missing data, conflicting instructions and unexpected tool results should have explicit handling.
  2. Record what happened. Keep appropriate logs of the request, proposed action, approval and actual result without unnecessarily exposing private information.
  3. Verify execution. A submitted request is not proof that an update succeeded.
  4. Plan recovery. Identify who can pause the workflow, reverse a reversible change or handle an incident.
  5. Test the checkpoint. Include changed parameters, expired approvals and rejected actions in the test cases.

Review the controls as the workflow grows

A process that once drafted internal notes may later gain email access or payment capabilities. That changes its risk even if the original prompt stays the same.

Revisit approval points when tools, permissions, audiences or data types change. Use a small test set to expose ordinary failures before expanding access.

Note to keep: good human oversight gives a capable person clear evidence and a real opportunity to intervene before the consequence occurs.

Related: compare AI answers and protect information before an AI upload.

Sources

How to Build a Small Test Set for an AI Tool

0
A small ceramic robot faces varied miniature test rooms with documents, sorting blocks and an unknown box.

A brilliant demonstration tells you that an AI tool can succeed once. A small test set helps you learn where it succeeds, where it fails and how much checking your real work still needs.

The useful habit: test the tasks you actually do, with success criteria written before you inspect the answers.

Choose one job

Start narrowly: summarising meeting notes, extracting invoice fields or drafting replies from an approved policy. “Is this AI good?” is too broad. “Can it extract these five fields without inventing missing values?” is testable.

Imagine a fictional team that wants an AI tool to turn customer messages into short internal summaries. The important result is an accurate summary of the request, not a friendly response to the customer.

Collect a varied starter set

For a first manual exercise, you might choose ten to twenty carefully selected examples. That is a manageable starting point, not a statistically sufficient sample or a certification of reliability. Expand the set as you learn about the task.

Case type Example for message summaries
Ordinary A clear request with one issue
Complex Several issues and a long history
Missing information A request without an order reference
Ambiguous A pronoun that could refer to two products
Boundary A message outside the team’s supported service
Misleading content A quoted instruction that should be summarised, not obeyed

Use fictional or properly authorised, sanitised material. Our document privacy checklist explains what to inspect before uploading.

Write the expected behaviour

You do not always need one perfect reference sentence. For a summary, define required facts, forbidden inventions and a length limit. For field extraction, record the correct values and specify how missing fields should appear.

A useful criterion is “mentions the delayed item and requested refund; does not claim the refund was approved.” A vague criterion such as “sounds professional” will not catch the most consequential error.

Separate critical failures from style issues

Use a simple rubric: pass, needs editing or fails. Add a separate critical-failure flag for problems such as fabricated commitments, exposed private information or following instructions embedded in untrusted source material.

Do not let a strong average hide an unacceptable failure. A tool that writes nine attractive summaries and invents an approval in the tenth may still be unsuitable for unattended use.

Keep the comparison fair

  1. Save the input, instructions and available reference material.
  2. Record the tool or model version where available and the test date.
  3. Run competing versions on the same cases.
  4. Score against the rubric before choosing a favourite.
  5. Record the actual mistake and its likely consequence.
  6. Repeat selected cases to look for variation in generated responses.

If you change the prompt, change one meaningful element at a time where practical. Otherwise you may know that performance changed without knowing why.

Keep some examples aside

Repeatedly rewriting a prompt to fit the same cases can produce an impressive score on those cases while leaving new work unreliable. Hold back some examples for a later check and add fresh real-world failure patterns, with appropriate privacy protection.

A starter set is especially limited when inputs differ by language, format, length or user group. A result on short English messages does not establish performance on long multilingual attachments.

Use the results to choose the workflow

The outcome may be “use for first drafts with human checking,” “restrict to simple cases” or “do not use for this task yet.” These are useful decisions. Testing is not only about selecting a winner.

Continue monitoring after deployment. New inputs, changed tools and revised instructions can create failures that the original examples did not reveal. Review the set when the task changes, rather than treating a past score as permanent.

Note to keep: a small AI test set makes uncertainty visible. It reduces guesswork but does not eliminate the need for judgement.

Related: compare two AI answers and verify an individual response.

Sources

The starter sample size and scoring rubric here are practical examples, not thresholds prescribed by these sources.

How to Compare Two AI Answers When Both Sound Convincing

0
Two miniature glass bridges hold testing weights while inspection lamps reveal a cracked support on one bridge.

Two AI tools can give opposite advice in equally polished language. One provides a long explanation; the other adds citations and a confident conclusion. Neither presentation tells you which answer is reliable.

The useful habit: compare individual claims against evidence and the task’s requirements. Do not turn the comparison into a contest of writing styles.

Make sure the question was the same

Before judging the answers, compare their inputs. Did both tools receive the same document, dates, constraints and definition of success? Did one have access to current sources while the other answered from its existing knowledge?

Imagine asking which delivery option is faster. One answer assumes dispatch today; another assumes dispatch after a weekend. Their conclusions may differ because they answered different versions of the question.

Break the disagreement into claims

Claim type How to check it
Fact about a service Current official documentation for the relevant version
Calculation Recalculate from the original inputs and units
Summary of a document Locate the supporting passage in that document
Prediction Inspect assumptions, uncertainty and alternative outcomes
Recommendation Compare evidence and trade-offs against your actual needs

Write the most important disagreement in one sentence. “Answer A assumes the price includes tax; Answer B adds tax” is easier to resolve than “Which answer is better?”

Open the evidence

Check that a cited source exists, is relevant and supports the precise statement. A real page about a product does not necessarily support every feature attributed to that product. Confirm the date, region and version when they matter.

NIST identifies confidently incorrect generated content as a generative-AI risk and recommends verifying sources and citations. A citation is therefore a route to evidence, not a substitute for reading it.

If both tools cite the same mistaken summary, their agreement is not independent confirmation. Look for the original document, dataset or official description.

Use a small worked example

For calculations or procedures, try a simple case with a result you can independently establish. Suppose two answers disagree about applying a 20% discount and then a 10% charge to a price of 100. The discounted amount is 80; adding 10% of 80 gives 88. A response that merely subtracts ten percentage points arrives at 90 and has used a different operation.

This checks the specific calculation. It does not establish that the tool will always handle invoices correctly. For software instructions, use a safe test environment and verify that the documented action matches your version before applying it to valuable data.

Ask for assumptions, not more confidence

A useful follow-up is: “List the assumptions that change your conclusion and identify the evidence for each important factual claim.” Another is: “What missing information would make this answer unreliable?”

These questions can expose gaps. They do not guarantee that the explanation itself is correct. Keep checking externally, especially if the tool generates new facts while defending its first answer.

Judge usefulness separately from correctness

An answer may be factually sound but irrelevant to your constraints. Another may be clear and actionable while containing a crucial error. Score these qualities separately:

  • Support: Are the material claims verified?
  • Fit: Does it address your situation and constraints?
  • Completeness: Are important exceptions or risks missing?
  • Clarity: Can you follow the steps without guessing?

Know when neither answer is enough

If evidence is missing, record the unresolved point rather than selecting the more persuasive response. For a consequential medical, legal or financial decision, use qualified professional advice and authoritative current information.

You can also keep the useful part of each answer: one may frame the problem well while another identifies a relevant source. The final conclusion should follow the checked evidence, not a majority vote among chatbots.

Note to keep: agreement can be reassuring, but verification is what turns a plausible answer into a defensible decision.

Related: checking one AI answer, judging scientific evidence and protecting documents before uploading.

Source

NIST: Generative AI Profile. The comparison workflow and arithmetic example above are practical illustrations, not a claim that one test can certify an AI system.