David Gerard@davidgerard@circumstances.run
@michael@westergaard.social wot's the running costs like?
do they tend to use a cloud or on-prem?
(i am very interested in practicalities of self-hosting this stuff, not sure how systemically important it is yet but it's the sort of thing that may well be)
Michael Westergaard@michael@westergaard.social
Depends. Mostly cloud. More cloud providers allow hosting models and only pay per use. Of course, your data gets mixed in with others to facilitate that, and you pay per token like at OpenAI.
Other rent you a GPU, and you par per month depending on the type of GPU (which is dictated by model needs).
These are the prices from DigitalOcean (white background, US but purely in the hosting business) and Scaleway (black background, French). Typically around a dollar/euro/pound per 1M tokens for a mid-tier model or 2k/month.