LLMs on the Jetson Orin Nano (original)

Good for

• Small 1–3B (chat, simple agents): fast, feels like chat, median 24.7 t/s (≈19 words/s), best 27.7 t/s, 3 tests
• Medium 7–8B: usable, median 14.0 t/s (≈10 words/s), best 14.2 t/s, 3 tests
• Not tested yet: Big 14B+. Help fill the gap

India price: now sold as the Jetson Orin Nano Super. Labels: ≥15 t/s fast · 5–15 usable · 2–5 slow · <2 too slow to chat. 1 token ≈ 0.75 words.

Real-world uses

Our data for this board: small (1–3B) models: median 24.7 t/s across 3 tests (fast, feels like chat); medium (7–8B) models: median 14.0 t/s across 3 tests (usable). Use cases that need chat-speed replies want ≥5 t/s; background jobs can live with 2 t/s.

• Camera + vision-language assistant: CSI camera input with a vision model, then an LLM describes or answers questions about what it sees.
• ROS 2 robot brain: Runs ROS 2 and an LLM side by side to turn spoken/typed goals into robot tasks.
• Offline voice assistant: Speech-to-text → local LLM → text-to-speech with no cloud dependency.
• Factory / farm inspection: Detect defects or crop issues on camera, with an LLM writing the summary report.
• Private document Q&A: RAG over manuals or company docs that never leave the device.
• Agent controlling hardware: 40-pin header GPIO lets an agent switch relays, read sensors, drive motors.

India prices

This board is now sold as the Jetson Orin Nano Super. See that page for current India prices.