Side Project Incident

I had my first side project incident this month! Do you want to guess what my Google Cloud Platform bill was?

I've been working on a text to podcast app because there are lots of online long form essays and blogs I'd prefer to listen to than read.

Text to speech (TTS) APIs are frankly incredible, but they are not free.

I have been using TTS to create my own podcast of articles and for nearly no cost. But, that is only because my usage never exceeded the free tier allowance.

Well, that is until I shared the app with a few friends - one of whom immediately decided to convert a 300 page book to audio. And the 300 pages would have been fine were it not for two small bugs.

First, the book contained an em dash, which is not a character the TTS API knows how to handle. This resulted in the API throwing an error.

This is where the second bug came into play. I had set up some retry logic to handle TTS failures, but it was misconfigured. Instead of retrying three times, it was set to retry with exponential back off for one hour. In practice this meant 14 attempts instead of three.

Fourteen attempts to process three hundred pages added up. But my friend also submitted the book twice so it ended up being 28 x 300 pages. Not good.

Google was kind enough to heavily discount the cost, but at the end of the hour my bill had quickly eclipsed $450.

In addition to fixing the two bugs, I added some safeguards like character and episode limits to help prevent this from happening again.

Feel free to give it a try. If you'd like less strict limits send me your credit card info and I'll be happy to generously increase your monthly quota.