Go Build the Arena

In April of this year I experienced what I felt like was the end of my job. I saw a blog post and immediately tried out the idea as soon as I could.

For four years our ML system rested at about 65-70% accuracy (measurements vary). I then spent 2 weeks and built a little arena where my AI bot could 1) analyze our data, 2) identify gaps and propose ideas, 3) an agent could implement those ideas via code changes, 4) the AI could then run the code and 5) view the output.

My goal was to optimize a time series forecasting solution - measured by how close predictions matched expected outcomes. (The way you test a forecast is by going into the past, say December of 2025, and predicting ahead 6 weeks into an already-observed “future” and then you compare your predictions against actuals.)

My team and I told the agent to build 150 algorithms (ideas). It just churned. It generated algorithms none of us had ever heard of, and some of them were very good at forecasting.

The result? About a 10-15% increase in accuracy across time series forecasts in our portfolio.

One of the things that’s exciting about this time is all the opportunities that await those who learn some simple ideas and apply them in their domains.

For example, AI research is a big topic. If we teach AI the scientific process and give it the ability to test ideas, we can iterate through ideas humans never thought possible.

Earlier this year I discovered autoresearch and saw how Shopify did incredible work in solving many of their own internal problems using this framework.

Regarding this work, another commented: “This all added up to a 53% improvement on benchmarks - truly impressive for a codebase that’s been tweaked by hundreds of contributors over 20 years.”

20 years and it took a few weeks to optimize performance by 53%.

All of that 20 years of work, however, laid the groundwork for the AI to improve a system. You can’t optimize something unless it’s measured.

Not all problems can be optimized in a closed-loop ecosystem. Most code is like a race car: you can make a change and test it right away. Other things take months to see the effects (clinical trials, credit card underwriting). In this case, you can still conduct more efficient scenario testing and Monte Carlo entire outcomes.

In sum, if you can observe it, and test it empirically, so can AI. Go build the arena and see what the AI experiments for you. That’s the new job.