III · Layers

Chapter 13

layers/swiglu — gated FFN

github.com/openfluke/welvet/layers/swiglu✅

Why it exists

Modern decoder FFNs are SiLU(gate)⊙up → down. Projections must share Dense’s quant/backend matrix.

What it is

Gate/Up/Down Dense children; DefaultFFN(dModel); WebGPU SiLU⊙ fuse on forward.

Go example

examples/13-swiglu/main.go

Run:cd welvet/examples/13-swiglu && source ../env.sh && go run .
package main

import (
	"fmt"

	"github.com/openfluke/welvet/core"
	"github.com/openfluke/welvet/layers/swiglu"
)

func main() {
	l, err := swiglu.New(swiglu.DefaultFFN(64))
	if err != nil {
		panic(err)
	}
	x := core.NewTensor[float32](1, 64)
	_, post, err := swiglu.Forward(l, x)
	fmt.Println(len(post.Data), err)
}

Output

exit 0 · last run via go run .

64 <nil>