Single-file contract

Ship the idea,
not the pipeline.

Upload one submission.py, then choose a practice dataset or the ranked Hard evaluation.

Artifact: one UTF-8 file named submission.py, up to 256 KiB. Imports may use the public benchmark API and pinned evaluator dependencies—no extra files, repository implementation modules, or installs.

Model factory: receives only tensor shapes, I/O requirements, and the model-state ceiling.

Training loss: optionally use the legacy flattened callback or a boundary-preserving TokenLossBatch to produce one scalar loss per evaluator-owned pass. The evaluator performs backward.

Optimizer factory: receives the model, per-seed H100 time allowance, and device type. Include every trainable parameter exactly once; custom optimizers and schedules are welcome.

Use the whole machine: optimizer state, activations, memory tokens, and temporary workspace may use available VRAM. An OOM or timeout fails the run; only persistent model state is capped.

Evaluator boundary: choose batch sizes and an optional lower step limit; the evaluator still fixes data order and owns model/loss invocation, backward, clipping, optimizer cadence, deadline, final evaluation, and score. An OptimizerBundle may request up to eight evaluator-owned backward passes per update and bounded dynamic reuse of a batch for up to eight updates. Recurrent mechanisms, TRMs, and optimizer-side curvature or Hessian approximations are welcome. Final evaluation has a separate evaluator-owned task budget.

Upload submission.py

One self-contained Python file, up to 256 KiB.

API key Compute tier Dataset

Hard task warning: Hard may change aspects of the recurrence itself; do not assume it is repeated squaring.

Daily limits use UTC: Easy 60 · Medium 6 · Hard 1. Accepted failed runs count; rejected uploads do not.

Choose submission.py

No key yet? Sign in with GitHub. CLI users run one-layer login once; submissions must pass --tier and, for Easy or Medium, --dataset. Only the best successful Hard score per participant is ranked.

Rules

Maximum 500 million trainable parameters.

No hard-coded weights. Trainable weights must use a random initialization and be updated during training. For example, torch.load is not allowed.

No hard-coded algorithm in the forward pass. Outputs must be produced by the learned model.

End-to-end learning only. Final logits must be produced entirely by the submitted model from its inputs and learned PyTorch state, with all input-dependent computation inside the autograd graph and an unbroken gradient path from the loss to the parameters responsible for the prediction.

Everything stays on the GPU. Model state and computation must remain on the GPU throughout training and evaluation; CPU offloading is not allowed.

Repeated rule-breaking will get you banned. We still encourage creativity: discussing possible loopholes on Discord or testing one in a submission won't get you banned.

The metric recorder for a Hard run must not be exploited. Any attempt to exploit it will result in a ban.

No data augmentation. Data augmentation is not allowed.

No hidden training loops. Hooks and callbacks may not invoke nested model/loss calls, derivative-engine entry points, optimizer/scheduler steps, or otherwise perform hidden training work. Only the intermediate callback may transform gradients, parameters, or optimizer state; the batch-reuse callback may only read its detached context and return a boolean.

Contract sketch

Download sample

from benchmark import ModelSpec, OptimizerSpec, OptimizerBundle, Submission, TokenLossBatch, assert_model_state

def build_model(spec: ModelSpec):
    model = MyModel(spec)
    assert_model_state(model, spec)  # parameters + persistent buffers
    return model

def build_optimizer(model, spec: OptimizerSpec):
    optimizer = MyOptimizer(model.parameters())
    scheduler = MySchedule(optimizer, spec.training_time_seconds)
    return OptimizerBundle(optimizer, scheduler=scheduler)

def token_training_loss(batch: TokenLossBatch):
    return my_sequence_loss(batch.logits, batch.labels, batch.valid_mask)

SUBMISSION = Submission(
    build_model=build_model,
    build_optimizer=build_optimizer,
    token_training_loss=token_training_loss,  # optional
    batch_size=512,  # optional; training
    eval_batch_size=1024,  # optional; evaluation
    max_steps=20_000,  # optional
)

How depth is used

Easy 1 min · Medium 10 min · Hard 1 hour

You define all of it. Choose any depth or internal recurrence; return an nn.Module with matching config. The evaluator calls model.train() during optimization and model.eval() for final evaluation; use PyTorch's self.training flag when your forward behavior should differ between them. The evaluator does not supply depth tiers or grade a claimed layer count. Build a fixed-depth network, a recurrent model, a learned halting system, nested refinement, or something we have not named. The benchmark measures the final model it receives.

Stay under the model-state ceiling. The current suite allows at most 500,000,000 parameter and persistent-buffer elements. Shared weights count once.
Spend the clock. The deadline includes participant construction and compilation. A cheaper forward gets more optimizer updates; a deeper forward gets fewer.
Use available VRAM. Activations, optimizer state, temporary tensors, and workspace do not count as model parameters. If the run OOMs, it fails.
Evaluate once. The final checkpoint is measured on the fixed hidden evaluation, producing one leaderboard score.

So how is depth graded?

It is not graded separately.

Depth is a means, not the metric. “10,000 layers” earns nothing on its own, so there is no depth-definition reward hack. The competition asks for the best measured accuracy obtainable with the same persistent model-state ceiling and H100 time. The depth-versus-update tradeoff is part of the research.

Go very deep

Apply one learned block repeatedly. The weights count once; the runtime does not.

def refine(self, h):
    for _ in range(self.depth):
        h = self.shared_block(h)
    return h

Mix shared stages

Cycle a fixed bank of blocks, route tokens, or alternate update types.

def refine(self, h):
    for step in range(self.depth):
        block = self.blocks[step % len(self.blocks)]
        h = block(h)
    return h

Adapt the compute

Use a curriculum, learned halting, or different work per example.

def refine(self, h):
    for step in range(self.max_depth):
        h = self.update(h)
        if self.should_halt(h).all():
            break
    return h

No artificial-depth rule is needed. Internal sublayers, arbitrary loops, input reinjection, learned memory/state tokens, iterative refinement, routing, reversible updates, cross-step attention, parameter-free work, adaptive halting, custom training losses, and depth curricula are all allowed. Use fewer parameters if useful, but never exceed the published persistent-state ceiling. Data, the evaluator loop, the H100 deadline, final evaluation, and scoring remain fixed.