Skill or Luck in a Backtest: How to Read the MCPT p-value

Where your result lands after the returns are shuffled a few thousand times — and what that number does not answer.

The backtest finishes and the Sharpe looks decent. But the question in the back of your mind is still unanswered: did the strategy really catch something, or did you just hack a few cuts at random inside a rising market? One way to answer it is to shuffle the data a few thousand times and see where your result ranks in that pile of noise — which is exactly what the "MCPT p-value" row on the workspace Backtest tab does. This article covers what it tests, how to read it, and the question it still has not answered once it passes.

What is MCPT?

MCPT (Monte Carlo Permutation Test) is a statistical test: it reshuffles the return series of the backtest period many times over, recomputing Sharpe each time on the same set of entry and exit positions, to build a whole distribution of what pure luck would look like — then checks where your actual result falls on that distribution.

The smaller the p-value, the rarer it is that a random shuffle can beat you. With the default of 2000 permutations, p = 0.05 means 100 of those 2000 shuffles did at least as well as you; p = 0.005 means only 10 did.

It shuffles the returns, not your entries and exits

This is the most critical part of the design, and the easiest to misread: what gets shuffled is the forward return series, while your position series stays fixed throughout.

What each permutation does:

position（your positions）　← fixed, computed once outside the loop
× vol_scalar（position scaling）　← fixed, computed once outside the loop
− fee_cost（fees）　← fixed, computed once outside the loop
× shuffled return series　← the only term redrawn each time
= Sharpe of this permutation

Because fees and position scaling are both computed outside the loop, every permutation bears the same trading cost and the same leverage. The only variable left is the order in which the returns arrive — that is, were the moments you picked better than randomly picked moments.

Why not just shuffle the positions instead?

Because that would be a freebie. Shuffling a binary position array creates far more entry and exit switches than the strategy itself ever makes; at fee = 0.0005, every permutation would carry roughly 30 to 40 times the fee drag, and every shuffled Sharpe would be forced deep into negative territory. Whether or not your strategy has a real edge, the p-value would come out beautiful and meaningless.

How this p-value gets produced

It is automatic, not optional. It runs once on every Type A backtest, and the result is written straight into the backtest stats and shown on the workspace Backtest tab.

Two behaviours worth remembering: MCPT does not run during a parameter scan (one scan is dozens to hundreds of backtests, and adding it would make the scan unusably slow), so the p-value you see always comes from the single backtest run after the parameters were adopted; and once a strategy is live, every live or scheduled trigger carries the same MCPT fields along unchanged — same code, same parameters, so the p-value still holds, which is why a running strategy keeps showing it.

What p-value counts as a pass? The verdict is not made on your machine

The workspace Backtest tab does not judge pass or fail; it only shows the number, plus one line of explanation. When p < 0.05 it reads:

When it does not pass, it reads:

The real pass or fail shows up in the Strategy Library's three quality gates. Gate 2 is called "Statistically significant" and its description reads "MCPT permutation test p < 0.05", while the actual verdict requires two conditions to hold at once:

Only when all three gates pass or do not apply does a strategy carry the "Verified" badge. To put the division of labour plainly: your machine only computes the p-value and the permutation count; whether that counts as a pass is the listing side's call.

MCPT on a real strategy: the 2026-09-17 run

Run the tutorial example btc_sma_cross that ships with the workspace once (BTCUSDT 1h, from 2022-01-01, SMA_FAST = 45 / SMA_SLOW = 100, not a single parameter touched, 41,287 bars). Every number below comes from that one 2026-09-17 run.

The line the backtest prints to stdout itself is:

MCPT p-value: 0.0160  (n=2000, significant edge at 95%)

How to read it: p = 0.0160 means that out of 2000 random shuffles, 32 produced a Sharpe at least as good — in other words, over this stretch of history these entry and exit timings do not look randomly chosen. The permutation count of 2000 is also above the gate's requirement of 1000, so this item passes.

Same strategy: MCPT passed, out-of-sample did not

This is the most important section of the article. On the same day and the same code, that strategy was also put through a walk-forward out-of-sample validation:

One passed, one did not, and neither is miscalculated — because they are simply not answering the same question. MCPT tests whether this already-decided set of entries and exits looks like more than guessing over this stretch of history; walk-forward tests whether the act of picking parameters from history still holds up on the next stretch it has never seen. Passing the former does not imply the latter.

This is not saying the strategy is broken — it is the tutorial example that ships with the workspace, and the point here is the difference in meaning between the two tests. For the full walk-forward numbers and how to read them, see How Out-of-Sample Validation Works.

The "Actual OOS Sharpe" line on the chart is mislabelled

If you ask the agent to plot the MCPT histogram and send it into the chat, the legend on that chart will print a line reading:

Actual OOS Sharpe = ⟨your number⟩

OOS is short for out-of-sample. That line is a leftover label from an old comment; do not read it literally. MCPT runs on the whole backtest dataset from start to finish, with no training window, no test window and no split ratio — it cannot possibly be an out-of-sample figure, and the comparison table above is the proof.

The chart on the workspace Backtest tab does not have this problem: its red line is labelled "Actual Sharpe". Also, the chart in the chat comes from a manual re-run, and a manual re-run does not necessarily use the same parameters as the automatic one, so the numbers on it need not equal the row on the tab.

Trap 1: the permutation count is a bar in itself, and it gets cut automatically

How many permutations the automatic MCPT actually runs is not entirely up to you. There is a runtime budget formula:

actual permutations = min( MCPT_N, 20000, max( 200, 4e8 ÷ bar count ) )

The more bars, the smaller 4e8 ÷ bar count gets. The run above had 41,287 bars, far from the ceiling, so not one of the 2000 permutations was cut. But the comparison recorded in the official documentation reads: 40,000 bars → 2000; 200,000 bars (two years of 5-minute data) → 2000; 1,000,000 bars (two years of 1-minute data) → 400.

The problem is that the listing gate requires permutations ≥ 1000. Working the formula through, once the bar count exceeds roughly 400,000 the actual permutation count drops below 1000 — and the gate fails you even at p = 0.001. Nothing on screen hints at it: the cut is only recorded in the host's log, the workspace explanation text happily reports the cut-down count, and there is no warning styling at all.

Trap 2: the Sharpe on the red line is not the same number as the one on the card

Look back at those two tables: for the same backtest, the Sharpe Ratio card says 0.6751 while the histogram's "Actual Sharpe" says 0.8985, a gap of 0.22 that is visible at a glance. This is not a bug; they are two different calculations:

The key is the second row: the automatic MCPT does not pass the strategy's own target-volatility setting through, it always uses the function's defaults. btc_sma_cross does no volatility targeting, yet its MCPT Sharpe is still the number you get "after applying 30% target volatility and a 2× cap" (for what target volatility does, see How Volatility Targeting Works). The two numbers differing is normal; the interface does not explain this today.

No MCPT row on the Backtest tab?

It fails quietly: no error message, the backtest succeeds as usual, and the front end is simply missing that row. There are at least four causes, and the screen does not tell them apart:

The reason for writing nothing rather than writing something wrong is very practical: any NaN or inf makes the whole machine's strategy upload bounce, so the moment the numbers are not clean, these fields are dropped as a group.

Portfolio strategies have no MCPT, and no substitute either

Backtests of Type C strategies (portfolio strategies: N symbols with a weight vector) do not run MCPT. The reason is that the test's structure is "one price series against one position series", and a portfolio strategy does not have that structure. The honest part worth saying out loud: there is no substitute test.

Once you have a p-value, check these five things first

What's next

Once the p-value makes sense, the next question is the right half of that comparison table: does it still hold on a stretch of data it has never seen? That road is out-of-sample validation, and the workspace has its own tab for it. The official strategies in the Strategy Library that carry the "Verified" badge cleared exactly this test at gate 2 — every one of them ships with a real backtest whose numbers you can read directly: Strategy Library.