How do you test whether an AI agent's freight quotes match what your team would book?
How do you test whether an AI agent's freight quotes match what your team would book?
A hypothetical logistics manager at a building products distributor has heard the same promise from three software demos: the agent quotes freight as well as her coordinators do. Her two coordinators have booked the company's freight for years. They know which LTL carrier handles long pallets without damage, which receiver needs a call before delivery and which truckload carrier answers the phone on Friday afternoon. She doesn't doubt that software can request quotes. She doubts that it would pick the carrier her team would pick, at a price that holds when the invoice arrives.
To test whether an AI agent's freight quotes match what your team would book, run a side-by-side on about ten recent or live shipments: give both sides the same order details and carrier list, collect quotes on the same day, and compare total cost, carrier, transit time and tender acceptance for each shipment, recording the reason behind every difference. The test answers a narrow question well: given the same information, does the agent reach a decision your team would sign off on?
Ten is a practical number, not a statistical one. It is small enough that two coordinators can work through it in a normal week, and large enough to include a few loads that are awkward on purpose. The value comes from reading each shipment, not from a percentage at the end.
What the test is and what it isn't
A side-by-side quote test is a controlled comparison of decisions. Both sides start from the same order, the same rules and the same set of carriers. Each side produces a recommended carrier, an expected total, an expected delivery date and a short reason. Then a reviewer compares them.
The four questions each shipment should answer
| Question | What to record for each side | Why it matters |
|---|---|---|
| What will this shipment cost in total? | Linehaul, fuel and every accessorial the shipment needs | A low base rate without the liftgate is not the cheaper option |
| Which carrier would move it? | Carrier name and whether it is on the approved list | A cheaper carrier your team avoids for a reason is not a match |
| Will it arrive when it must? | Quoted transit days and expected delivery date against the required date | A saving that misses the delivery window costs more later |
| Would the carrier take it at that price? | Tender accepted, rejected or countered | A quote that the carrier won't honor at booking is only a number |
| Why this choice? | One or two sentences of reasoning | Shows whether the decision rests on rules the team recognizes |
A shipment where both sides pick the same carrier at the same total, with the same delivery date, is a clean match. Most tests also produce a few interesting differences, and those are where the team learns the most.
Let's review ten of your recent shipments and how your team quoted them. You don't need to prepare anything: Cartage maps the workflow with you on the call.
See it on your own loads. Wilson, by Cartage, is an AI logistics agent that quotes, books, tracks, audits and pays for freight with your approved carriers.
Book a demo →Which ten shipments should go into the test?
Pick ten shipments that look like a normal month for your company, with two or three deliberately difficult ones, across the modes and lanes you actually ship. A test built only from easy dock-to-dock truckloads tells you little, and a test built only from exceptions doesn't reflect daily work.
A balanced mix
A useful set of ten usually includes:
- Four routine shipments on your busiest lanes. These show whether the agent lands on the same carriers and prices your team uses every week.
- Two LTL shipments with accessorials, such as liftgate, residential or limited-access delivery, or a delivery appointment. These test whether the quote includes every charge the shipment needs.
- One shipment with a firm delivery date or a narrow receiving window. This tests whether transit time is weighed against price.
- One shipment on a lane you rarely ship. This tests how quotes are gathered when there is no usual carrier.
- One shipment with awkward freight, such as long, non-stackable or high-value goods. This tests whether handling notes reach the carrier.
- One shipment where the cheapest quote is from a carrier your team avoids. This tests whether preferences and rules are respected.
Recent shipments or live orders?
There are two ways to run the test, and each has a trade-off.
Replay recent shipments. Take ten orders shipped in the last few weeks. On the test day, both the team and the agent request fresh quotes for the same freight as if it were shipping that week. This is the fastest setup because the orders are already known and the team can choose the mix. The disadvantage is that carriers know the load isn't real if asked to tender, so acceptance can only be judged on quotes, not on actual bookings.
Shadow live orders. Take the next ten orders that come in. The team books them as usual, and the agent works the same orders in parallel without booking. This tests the agent on orders as they really arrive, including incomplete ones. It takes longer, because you have to wait for the right mix, and it needs a clear rule so nobody books the same load twice.
How the test is set up depends on the company's systems and scope, so confirm the method in discovery. Either way, the order details, carrier list and approval limits given to both sides should be identical.
What to collect before the test
For each of the ten shipments, write down the inputs both sides will use:
| Input | Example of what to record |
|---|---|
| Origin and destination | Full addresses, dock hours, appointment rules, location type |
| Ready date and required delivery date | Including whether the delivery date is firm |
| Freight | Handling units, dimensions, weight, stackability, freight class or NMFC item for LTL |
| Services | Liftgate, inside delivery, notify before delivery, residential, limited access |
| Equipment | Dry van, reefer, flatbed, trailer length, for truckload |
| Carrier list | Which carriers are approved for this lane and any carrier to avoid |
| Approval limit | The most that can be booked without a person approving |
| Known preferences | For example, a hypothetical rule such as "use the regional LTL carrier first into the Northeast" |
The checklist on the order data carriers need before they can quote covers each field by mode. If a field is missing for the team, it should be missing for the agent too. Watching how each side handles a missing field is part of the test.
Write down what the team already knows
Experienced coordinators carry rules in their heads. Before the test, ask them to write down the ones that apply to the ten shipments: carriers they avoid on certain freight, receivers that refuse deliveries without a call, lanes where one carrier always gets the first look. These preferences are fair inputs for both sides. Leaving them out would test the agent against knowledge it was never given, which measures the onboarding gap rather than the quality of the quotes.
What exactly should the side-by-side compare?
Compare total cost on the same basis, the carrier, transit against the required date and acceptance, then add a short note on the reasoning. One worksheet with one row per shipment is enough.
Total cost, not base rate
The most common false result in a quote test comes from comparing a base rate on one side with an all-in price on the other. Put every quote on the same basis: linehaul plus fuel plus every accessorial the shipment needs. If one carrier quotes liftgate separately and another includes it, add it before comparing. The articles on freight quotes that leave out accessorials and on comparing total charges before choosing a carrier explain how a cheaper-looking quote can end up as the more expensive invoice.
Fuel deserves its own note. Fuel surcharges are often a percentage tied to a weekly diesel index and differ by carrier table. Two quotes collected on the same day avoid most of the timing problem, but check that both totals include fuel.
The carrier choice
Record which carrier each side would book and whether it is on the approved list for that lane. Three outcomes are common:
- Same carrier. Usually a sign that the agent had the same carrier list and preferences as the team.
- Different approved carrier at a lower total. Worth examining. Was the cheaper carrier unknown to the team, or does the team avoid it for a reason it didn't write down?
- Different carrier the team would not use. A miss, unless the reason is a rule the agent was never given.
Transit and delivery date
A quote is only good if it delivers by the required date. For each shipment, record the quoted transit days and the expected delivery date, then compare them with the date the customer needs. Treat a carrier's published transit time as an estimate, and check whether the receiver's appointment rules add a day. The article on comparing quotes when every carrier gives a different transit time covers how to weigh speed against price.
Acceptance
A quote that the carrier won't honor at tender is not a real option. On live orders, acceptance shows up when the load is booked. On replayed shipments, ask whether the quote is firm, how long it is valid and what conditions apply. Note any quote that was conditional, such as subject to capacity on the pickup date.
A hypothetical worksheet
Here is a hypothetical example of how the ten rows might look after a test. The numbers illustrate the method; they are not market rates.
| # | Shipment | Team's choice and total | Agent's choice and total | Result | Reason |
|---|---|---|---|---|---|
| 1 | Routine FTL, busiest lane | Carrier A, $2,180 | Carrier A, $2,180 | Match | Same contract rate |
| 2 | Routine FTL, second lane | Carrier B, $1,640 | Carrier C, $1,575 | Better | Approved carrier the team hadn't asked this week |
| 3 | Routine LTL, 4 pallets | Carrier D, $612 | Carrier D, $612 | Match | Same tariff, same class |
| 4 | Routine LTL, 2 pallets | Carrier E, $388 | Carrier D, $401 | Acceptable difference | Agent applied the team's first-look rule for the region |
| 5 | LTL with liftgate and residential | Carrier F, $745 | Carrier E, $690 | Better | Team's total omitted nothing; agent found a lower all-in total |
| 6 | LTL with appointment | Carrier D, $530 | Carrier G, $455 | Miss | Agent's quote left out the appointment charge |
| 7 | Firm delivery date | Carrier B, $2,050, 2 days | Carrier H, $1,890, 3 days | Miss | Cheaper option missed the required date |
| 8 | Rare lane | Carrier I, $3,300 | Carrier J, $3,120 | Better | Wider set of quotes on an unfamiliar lane |
| 9 | Long, non-stackable freight | Carrier F, $980 | Carrier F, $980 | Match | Handling note reached the carrier |
| 10 | Cheapest carrier avoided | Carrier D, $520 | Carrier D, $520 | Match | Avoid rule respected |
In this example, five shipments match, three come out better, one is an acceptable difference and two are misses. The misses are the most useful rows. Shipment 6 shows a quote that left out an accessorial. Shipment 7 shows a choice that weighted price over a firm date. Both point to inputs and rules that need to be fixed before go-live, which is exactly what the test is for.
Why do the team's quotes and the agent's quotes differ?
When the answers differ, the cause is usually an input: a different freight description, a missing accessorial, a carrier list that doesn't match or a preference only one side knew about. Read each difference with that in mind before concluding that one side judged better.
Different freight details
A carrier prices the freight it is told about. If the team quoted four pallets at 48 by 40 by 60 inches and the agent's order record shows 48 by 40 by 72, the density is different, and an LTL freight class based on density may be different too. Check that both sides used the same handling units, dimensions, weight and class. A difference here is a data problem, not a quoting problem.
Accessorials known to one side only
Coordinators often add liftgate or an appointment because they remember the receiver, even when the order doesn't say so. If the agent doesn't have that information, its quote will be lower and wrong. The fix is to put the receiver's requirements on record, so every future quote includes them, whoever requests it.
The carrier list
If the team asked four carriers and the agent asked seven, the agent will often find a lower price. That is a real difference, but it reflects reach, not judgment. Record how many quotes each side collected for every shipment. If the team regularly asks only two carriers on a lane because nobody has time to ask more, that is a finding in itself.
Preferences that were never written down
A coordinator may skip a cheaper carrier because of a claim last spring, a driver who couldn't find the dock or a receiver who complained. If that preference isn't written down, the agent can't follow it. Treat these cases as items to add to the rules, not as agent errors.
How do you score the results and decide what comes next?
Score each shipment into one of four categories, total them, and then decide using rules agreed before the test started. The categories should be simple enough that the team can apply them without debate.
Four scoring categories
| Category | Definition | Example |
|---|---|---|
| Match | Same carrier, or a different approved carrier at the same total, meeting the date | Both pick the contract carrier on the main lane |
| Better | Lower all-in total from an approved carrier, meeting the date and preferences | A wider set of quotes finds a lower price on a rare lane |
| Acceptable difference | Different choice the team agrees is reasonable, even if not cheaper | The agent follows a written first-look rule the team sometimes skips |
| Miss | Wrong carrier, missing charge, missed date or broken rule | A quote that omits an appointment charge |
Agree the definitions before the test. Otherwise every disputed row becomes a negotiation about what "better" means.
What counts as a pass
There is no universal pass mark. A reasonable approach is to ask three questions after scoring:
- Did any miss break a rule the agent was given? A miss on a rule that was clearly in the inputs is serious. A miss on a rule nobody wrote down is an onboarding item.
- Would the team have approved every booking within its limits? If the agent would have booked something outside the approval limit, that matters more than any saving.
- Are the reasons readable? The team should be able to follow why each carrier was chosen. A decision nobody can explain is hard to trust on the next hundred loads.
If the answers are no, yes and yes, the test supports moving forward on a limited set of lanes with approvals in place. If the answer to the first question is yes, fix the rule handling and run another short test on the shipments that failed.
What the test can't tell you
Ten shipments can't prove how an agent performs over a quarter, through a capacity squeeze or across every lane. They can't show tracking, invoice audit or payment work, because the test stops at the quote and the booking decision. Treat the test as the first gate. The next step is a limited start on real loads with approvals in place, where the team reviews each booking and widens the scope as the work proves itself. The article on comparing real loads with your current process covers that longer comparison, and the guide to the procurement decisions worth testing adds cases above and below a lane target.
How does Wilson, by Cartage, work in a side-by-side quote test?
Wilson, by Cartage, is an AI logistics agent that requests quotes from the company's approved carriers and its network, compares them on total cost, negotiates toward the team's targets within approval limits and books the carrier, so in a side-by-side test it produces a carrier, a total and a reason for each shipment the team can check. Traditional logistics software gives teams information. Wilson executes the work required to move shipments forward.
Wilson works as a free logistics department for manufacturers and distributors: what big shippers build over years, Wilson does from day one. A quote test touches each of its four parts.
The labor
The labor in a quote test is the work coordinators normally do by hand. Wilson reads each order, checks it against what the mode needs and asks the right person for anything missing, such as dimensions, a delivery appointment or the freight class, before quotes go out. It requests quotes from approved carriers in the same format, follows up with carriers that haven't replied and compares the answers on total cost, including the accessorials the shipment needs. After the test, the same work runs on live loads: booking, paperwork, tracking and carrier follow-ups through to the final POD, by email, phone calls, texts, Slack and Teams. The team doesn't chase carriers.
The data
The data is what makes the test readable. Wilson keeps every order, quote, rate, document and contact on record as it works, connected to the company's ERP, so each quote in the test sits next to the order it priced. Preferences the team writes down during the test can be kept as rules and memories, for example a hypothetical note that a certain receiver takes deliveries only by appointment, and the team can edit them. Wilson replaces the TMS for freight execution and connects to the company's ERP. Wilson works with SAP, NetSuite, Microsoft Dynamics 365, Oracle, Infor, Epicor, Sage, Acumatica and other ERP environments, with the exact connection depending on the customer's data flow and scope. Records are searchable and exportable, with change history. Wilson can also produce lane-rate and carrier-performance reports on request from the loads it runs.
The network
The network often explains "better" rows in a test. The shipper's volume counts with every other Wilson customer's, which can bring better freight rates on its lanes, and the network includes carriers and brokers. The company still approves who hauls its freight: no carrier is used until the company approves it. Shippers can also keep their own carriers on any lane. In a test, record which quotes came from the company's own carriers and which from the network, so the team can see where each saving came from.
The payments
The payments are outside a ten-shipment quote test, but they decide whether a quoted price holds. On live loads, Wilson audits every freight invoice against the load before money moves, then pays carriers from the Wilson Wallet. Invoices over the rate or outside the rules wait for a person, and Wilson asks the carrier why an invoice is over the rate before paying it. That is where a quote that omitted a charge, like shipment 6 in the hypothetical worksheet, would surface.
Approvals during and after the test
| Situation | What Wilson does | When a person decides |
|---|---|---|
| Order missing a field | Holds the order as pending, asks the customer and tells the team; never invents the value | When nobody can supply the detail |
| Quotes come back | Compares them on total cost against the lane target | When the best quote is above the approval limit |
| Cheapest carrier is on an avoid rule | Follows the rule and moves to the next carrier on the company's ranked list for the lane | When the team wants to change the rule |
| Carrier not yet approved | Doesn't book it | The company approves or declines the carrier |
| Booking within limits on a proven lane | Books and collects the rate confirmation | Only if the team keeps that lane on review |
Wilson starts with assisted execution. The company sets approval limits, and routine actions can run automatically within those rules as the workflow is proven. It handles LTL, FTL, ocean, drayage, parcel and cross-border freight. It is not a freight broker or a customs broker. The company controls the scope: it can start with a subset of shipments or one or two key customers, or exclude loads by rule. Whether a test replays recent shipments or shadows live orders depends on the setup; confirm it in discovery.
Pricing
Wilson is free on Wilson's network: the shipper pays the freight rate and nothing on top. When a shipper keeps its own carriers, the fee is 2% to 0.5% of what Wilson pays them, falling with volume. There are no seat fees, no setup cost and no contract. Timing depends on scope: most teams start on a few lanes and expand from there.
FAQs
How do we test Wilson's quotes against our team's?
Wilson, by Cartage, an AI logistics agent, can be compared with your team on about ten recent or live shipments. Both sides get the same order details, carrier list and approval limits, quotes are collected on the same day and each shipment is scored on total cost, carrier, transit and acceptance, with a reason for every difference. The exact setup depends on your systems and is agreed in discovery.
How many shipments do you need to test freight quotes?
About ten is enough for a first decision if the set is chosen well: routine loads on busy lanes, LTL with accessorials, a firm delivery date, a rare lane, awkward freight and a case where the cheapest carrier is one the team avoids. Ten won't prove performance over a quarter, but it shows whether the decisions make sense.
Should we compare the agent's quote with what we actually paid?
No. Compare quote with quote, collected on the same day. What you paid may include detention, a reweigh or a redelivery that no quote shows, and the market may have moved since.
Why did the agent find a cheaper carrier than my team?
Usually because it asked more carriers, used a different approved carrier or had different freight details. Check the inputs first. If the details and carrier list were the same, look at whether the cheaper quote included every accessorial and met the delivery date.
What if the agent picks a carrier my team avoids?
Find out whether the avoid rule was written down and given to the agent. If it was, the result is a miss. If it wasn't, add the rule and re-run that shipment.
What counts as passing a freight quote test?
Agree it before the test. A practical standard: no miss that breaks a rule the agent was given, no booking above the approval limit and reasons the team can follow for every choice.
Does a quote test show invoice accuracy?
Not directly. It stops at the quote and the booking decision. Invoice audit shows up on live loads, when each invoice is checked against the rate and the documents before payment.
Can the test include parcel or drayage shipments?
Yes, if those modes are in scope. Add one shipment per mode with the details that mode needs, such as package dimensions and service level for parcel or the container, terminal and free time for drayage, and compare them the same way.
Who should score the test?
The shipper. The coordinator who books the lanes and the manager who owns approval limits should score each row. The vendor explains its reasoning but shouldn't grade its own work.
Sources
- U.S. Energy Information Administration, Gasoline and Diesel Fuel Update: https://www.eia.gov/petroleum/gasdiesel/
- Cartage, Wilson product FAQ and product scope, October 2026: https://cartage.ai
Let's review ten of your recent shipments and how your team quoted them. You don't need to prepare anything: Cartage maps the workflow with you on the call.
