Business
Inference vs training cost
Training is a giant one-time bill; inference is the ongoing running cost.
Building a model, called training, is a massive upfront expense that can run into the tens or hundreds of millions of dollars, but it happens relatively few times. Inference, the cost of actually answering each request, is smaller per use but never stops and adds up as millions of people use the model. So a company pays a huge sum to create a model, then keeps paying, bit by bit, every time someone uses it.
For example
Training a flagship model might cost a fortune once, while inference is the steady electricity bill of serving it to users every day.
Related
APIA doorway that lets other software talk to an AI model.Tokens as costYou usually pay for AI by the token, for both what goes in and comes out.Rate limitsA cap on how many requests you can send an AI in a given time.On-device AIAI that runs right on your phone or laptop instead of a distant server.