Compositions
AIChatLayout
Full-page AI chat layout with conversation sidebar, chat thread, and prompt composer bar.
Preview:1280px
Can you outline the key design decisions for a mixture-of-experts transformer?
Sure. The three core decisions are: expert routing strategy (top-k vs. soft), load balancing via auxiliary loss, and expert capacity buffers to prevent token dropping. For Phase 21 I would recommend top-2 routing with a z-loss regulariser.
What auxiliary loss function works best at the 70B scale?
At 70B, the Switch Transformer auxiliary loss (alpha * sum_i(f_i * p_i)) with alpha around 1e-2 keeps experts balanced without hurting perplexity. Combine it with z-loss (1e-3) to stabilise router logits.
Props
| Prop | Type | Default | Description |
|---|---|---|---|
| conversations | ConversationItem[] | sample data | List of past conversations for sidebar. |
| messages | ChatMessage[] | sample data | Messages for the main chat thread. |
| onSend | (msg: string) => void | - | Fired when user sends a message. |