Chapter 19

layers/residual

github.com/openfluke/welvet/layers/residual✅

Why it exists

Skip connections stabilize deep stacks: y = F(x) + x with correct skip grads.

What it is

F is Dense Dim→Dim, or mixed Ops via NewFromOps (Dense, SwiGLU, RMSNorm, LayerNorm). Parallel as F is parallel.ResidualGraft (y = F(x)+x) — Residual cannot import parallel.

NewFromOps builds F over mixed Ops. Parallel as F is parallel.ResidualGraft → y = F(x)+x. Skip grads still land on F and the identity path. Nested mixed Residual is on the v1.0 board.

package main

import (
	"fmt"

	"github.com/openfluke/welvet/core"
	"github.com/openfluke/welvet/layers/residual"
)

func main() {
	l, err := residual.New(residual.Config{Dim: 16, Depth: 2})
	if err != nil {
		panic(err)
	}
	x := core.NewTensor[float32](1, 16)
	_, y, err := residual.Forward(l, x)
	fmt.Println(len(y.Data), err)
}

16 <nil>