Finding the Best Model for Live Events Ticketing Analysis
Every model we tested got the ticketing questions right. So the choice stopped being about smarts and instead becomes about (1) differences in speed and cost, and (2) personality and vibes.
TL;DR;
If you’re building an agent to do data analysis for live events, there are quite a few models — closed and open weights — that will do a great job. We took the “workhorse tier” and ran each one through 6 quick question answers and 6 deeper analysis requests inside the exact harness we use every day.
Every model got the numerical answers right. Once correctness is a given, the choice stops being about which model is smartest and becomes about two things:
- The things you can measure: speed, cost, how much it parallelizes work, how much it writes
- The “vibes”: formatting, what it explains, whether it shows its work
On the measurable side, we were most impressed by the efficiency of the newest OpenAI models, driven by their parallel tool calling, and intrigued by Echo as a new entrant. On the vibes side, Claude still loves its tables and words while the new OpenAI models answer like my Gen Z nieces over text.
Balancing speed, cost, and brevity
How each model performed on average in terms of time, cost, and length of answer
Why we ran our own harness
As part of our work at Extra Point, I built a data analyst agent using Mastra to handle our day to day workload. This agent sits on top of our warehouse, and all the context about our business and clients that we’ve carefully organized. That means we don’t need the beefiest model to get the best results. So the real question was who would be our pareto BFF: the model that gives us the best answers for the least time and money.
The public benchmarks didn’t help me find it. I wanted to know what it would be like to live with this agent day in and day out — how it would perform in our harness, answering the kinds of ticketing questions we ask every day.
I used Mastra’s datasets and experiments to build a two tier benchmark around our existing agent, rotating models through the exact harness we use every day with the same tools and skills.
- The “regression tier” is a set of “quick questions” and golden answers. This serves both as a model baseline and our frontline eval/test suite when making changes to prompts. Think “what was ATP for this section” or “what was the total revenue for this event”.
- The “deep analysis tier” is questions we had actually asked our agent, cleaned up and paired with golden answers. I also built a rubric for each question which I’m still tuning.
Every model got it right
The math, it turns out, is mathing. Across both the regression and deep analysis tier, all the models we tested produced correct answers. I took some of that as a sign our warehouse is in great shape, and some of it as a sign we weren’t yet testing for the large, open-ended analysis that tends to produce real separation.
Getting results this uniform shows how much the model world has flattened in the past year. When Sonnet launched a year ago it was in a class of its own for efficient agentic use. There are now a wide variety of equally good competitors.
Once everyone’s right, what’s left? Mostly speed, cost, and “vibes.”
When every model clears the correctness bar, the decision moves to everything around the answer. That everything comes in two layers.
The first layer you can measure: how fast a model responds, what it costs, how much of its work it does in parallel, how much it writes. The second layer you can’t — it’s the feel of the output once the facts are settled. That second layer is what most of us mean by “vibes”: formatting, what it explains, whether it shows its work or just hands you the number.
Most of us are already fluent in this. I for one can’t read or hear “load bearing” without thinking of Claude. As raw ability converges, that feel is going to matter more, and both internal and external builders will have to reason about it deliberately as they choose models for their teams and customers.
The rest of this post takes those two layers in order: first the parts we could measure, then the parts we could feel.
What actually drives the clock
The measurable layer is mostly a story about time, so we dug into what actually drove each model’s time to respond and whether it varied across cases.
Model responses by speed and length
One dot per model per question — right is slower, up is wordier. Click a dot to isolate its model; click empty space to bring everyone back.
You’ll notice that answer length doesn’t correlate with time to answer. So time to answer isn’t purely being driven by how many tokens the model generates in its final answer, but also by how much work it does to generate that answer (and how it does that work).
Tool calls per round trip vs time
Right is more tool calls packed into each round-trip; up is more seconds of thinking per round-trip.
Which middle-school math-quiz kid are you?
When I was a grade schooler, my KPI on quizzes was whether I could be the first to hand mine in. My least favorite request was to show my work. My wife, on the other hand, always needed an extra blue book for her written tests.
Our models split the same way. Terra and Luna are me: blazing through parallel tool calls and short turns to hand the answer in first, with almost none of the work shown. Opus is my wife: it parallelizes less and takes half as many round trips, but thinks about 4x longer on each one and shows every step.
Then there’s the kid who finishes first by writing faster, not by skipping work. Gemini 3.6 Flash never parallelizes — a single tool call per round trip — but generates tokens so fast it keeps pace anyway. Same fast answer as Luna and Terra, opposite engineering: they win on batching and brevity, Gemini wins on raw speed. (Echo is the kid still figuring out the test: slow tokens, early in its journey, but I’d expect that to climb.)
| Model | Avg. time | $ / answer | Tokens generated | Tool calls / trip | Tokens / sec |
|---|---|---|---|---|---|
| Luna | 0:56 | $0.01 | 3.5K | 1.9 | ~64 |
| Terra | 0:57 | $0.12 | 3.4K | 1.9 | ~60 |
| Gemini 3.6F | 1:18 | $0.34 | 8.9K | 1.0 | ~115 |
| GLM 5.2 | 1:23 | $0.09 | 8.5K | 1.8 | ~104 |
| Sonnet 4.6 | 2:09 | $0.26 | 4.7K | 1.3 | ~43 |
| Echo | 2:40 | $0.01 | 6.4K | 1.8 | ~40 |
| Opus 5 | 3:21 | $0.77 | 10.6K | 1.4 | ~57 |
All figures are averages per deep question. The tradeoff is in the shading: Gemini 3.6 Flash and GLM generate tokens roughly twice as fast as anyone else, but never (or barely) batch their tool calls. Luna and Terra generate at mid-pack speed and win anyway — parallel tool calls plus ruthless brevity. Echo has neither going for it, which is the whole story of its clock.
The bigger gap is cost
Money is where the models separate most. At equal correctness, Luna and Echo came in around a penny a question while Opus ran about 77 cents — roughly 60x. When every answer is right, that spread is the whole argument for not defaulting to the biggest model.
Whether that 60x matters depends entirely on who’s asking. Volume comes from seats, and seats come in two flavors. The social media manager living in the agent asks dozens of questions a day; the C-level exec checks in twice a week. The exec alone is lunch money on any model. Once you staff a growing team with a realistic mix of the two, the big-model habit turns into a five-figure annual bill, while the penny tier handles the same work for a rounding error.
When the cost gap starts to matter
Estimated monthly spend as the team grows, assuming one in five teammates is a power user asking 7–8 questions a workday and the rest ask 1–2 a week. Click a model in the legend to isolate its line; click empty space to bring everyone back.
Now the vibes: what each model felt like
Speed and cost sort the models on paper. Reading the actual answers is where the personalities show up. These were our impressions.
Opus 5. Writes like a senior consultant who won’t let you leave the room with the wrong takeaway: headline, receipts, tables, drilldowns, caveats about what the numbers don’t say. Longest and slowest answers in the field, but almost none of it is filler. Great for getting in the weeds, risks being ignored by team members with less time or interest.
Sonnet 4.6. Reads like a great Slack message from an analyst who knows you’re skimming: bolded verdict up top, emojis, a sharp “point you might have missed”. The classic “Claude” voice and tone, with its occasional drift into burying the insight inside the formatting.
Terra. The disciplined one: crisp executive summaries, every number carefully labeled, scope guarded like a lawyer. Rarely dazzles, never overreaches, predictably direct.
Luna. The shortest answers and the fastest clock, with almost no narrative around the numbers. When you know what you’re asking it’s an efficient dream, but gives almost no cites, analysis, or illustration.
Gemini 3.6 Flash. Feels like the house style of a good BI team: polished report formatting — executive summary, bolded key metrics, artifact links — delivered at nearly Luna speed. Maybe not the deepest read in the room, but consistently presentable and consistently quick.
GLM 5.2. Methodical Part A / Part B structure with real analytical ambition. It will build the year-over-year diff you didn’t ask for. That ambition cuts both ways: it’s the model most likely to construct a clever answer where a shrug was the right call.
Echo. The employee who brings their whole self to work: Narrates its thinking. You watch it check itself (“I need to correct course”) and quote its sources verbatim before it lands. Endearingly transparent, occasionally long-winded, and when its tooling misbehaves it fails loudly rather than quietly, clearly a work in progress.
Where we landed
When every model is right, “which one is best” is the wrong question. The real question is which personality you want answering, and what you’re willing to pay for it. The answer changes with the question you’re asking.
The two models that surprised us make the point. Gemini gets to a fast answer on raw token speed while never parallelizing; Luna gets there as a middling generator that batches its tool calls and stays terse. You don’t pick between them on a benchmark. You pick on which behavior fits the job, and at a penny a question versus Opus’s 77 cents, the job usually decides.
So we’re not crowning a model. We’re running an experiment that uses Mastra’s request context to route simple questions to Terra and complex ones to Opus. We’ll report back on whether that felt like a smart optimization or whether it just made us nervous about which version of the analyst we’d get each time we asked.
We’re also going to try out Echo in our assistant harness. Given this analysis the promise of smart model selection at the provider level is super intriguing so we want to get a sense of how it works.