Single-file contract

Ship the idea,
not the pipeline.

Upload one submission.py, then choose a practice dataset or the ranked Hard evaluation.

Artifact: one UTF-8 file named submission.py, up to 256 KiB. Imports may use PyTorch, the public repository modules, and pinned evaluator dependencies—no extra files or installs.

Model factory: receives only tensor shapes, I/O requirements, and the model-state ceiling. Choose any depth or internal recurrence; return an nn.Module with matching config.

Training loss: optionally turn final logits, auxiliary outputs, and current labels into one scalar loss. The evaluator performs one backward pass.

Optimizer factory: receives the model, per-seed H100 time allowance, and device type. Include every trainable parameter exactly once; custom optimizers and schedules are welcome.

Use the whole machine: optimizer state, activations, memory tokens, and temporary workspace may use available VRAM. An OOM or timeout fails the run; only persistent model state is capped.

Evaluator boundary: data, sampling, batches, one forward, one backward, clipping, optimizer cadence, deadline, final evaluation, and score remain fixed.

Contract sketch

Download sample

from benchmark import ModelSpec, OptimizerSpec, OptimizerBundle, Submission, assert_model_state

def build_model(spec: ModelSpec):
    model = MyModel(spec)
    assert_model_state(model, spec)  # parameters + persistent buffers
    return model
def build_optimizer(model, spec: OptimizerSpec):
    optimizer = MyOptimizer(model.parameters())
    scheduler = MySchedule(optimizer, spec.training_time_seconds)
    return OptimizerBundle(optimizer, scheduler=scheduler)

def training_loss(logits, labels, auxiliary):
    return my_loss(logits, labels, auxiliary)

SUBMISSION = Submission(
    build_model=build_model,
    build_optimizer=build_optimizer,
    training_loss=training_loss,  # optional
)

How depth is used

Easy 1 min · Medium 10 min · Hard 1 hour

You define all of it. The evaluator does not supply depth tiers or grade a claimed layer count. Build a fixed-depth network, a recurrent model, a learned halting system, nested refinement, or something we have not named. The benchmark measures the final model it receives.

Stay under the model-state ceiling. The current suite allows at most 207,489 parameter and persistent-buffer elements. Shared weights count once.
Spend the clock. The deadline includes participant construction and compilation. A cheaper forward gets more optimizer updates; a deeper forward gets fewer.
Use available VRAM. Activations, optimizer state, temporary tensors, and workspace do not count as model parameters. If the run OOMs, it fails.
Evaluate once. The final checkpoint is measured on the fixed hidden evaluation, producing one leaderboard score.

So how is depth graded?

It is not graded separately.

Depth is a means, not the metric. “10,000 layers” earns nothing on its own, so there is no depth-definition reward hack. The competition asks for the best measured accuracy obtainable with the same persistent model-state ceiling and H100 time. The depth-versus-update tradeoff is part of the research.

Go very deep

Apply one learned block repeatedly. The weights count once; the runtime does not.

def refine(self, h):
    for _ in range(self.depth):
        h = self.shared_block(h)
    return h

Mix shared stages

Cycle a fixed bank of blocks, route tokens, or alternate update types.

def refine(self, h):
    for step in range(self.depth):
        block = self.blocks[step % len(self.blocks)]
        h = block(h)
    return h

Adapt the compute

Use a curriculum, learned halting, or different work per example.

def refine(self, h):
    for step in range(self.max_depth):
        h = self.update(h)
        if self.should_halt(h).all():
            break
    return h

No artificial-depth rule is needed. Internal sublayers, arbitrary loops, input reinjection, learned memory/state tokens, iterative refinement, routing, reversible updates, cross-step attention, parameter-free work, adaptive halting, custom training losses, and depth curricula are all allowed. Use fewer parameters if useful, but never exceed the published persistent-state ceiling. Data, the evaluator loop, the H100 deadline, final evaluation, and scoring remain fixed.

Ship the idea,not the pipeline.