330: 50% New Users Overnight… and a Burning Server

Download MP3
Arvid:

This week was a duality of extremes for me. On the one hand, I experienced unprecedented growth, a strong growth spurt in my user numbers. While on the other hand, I stumbled into a frustrating scarcity and infrastructure problems. That's entrepreneurship for you, big wins while struggling to not lose control of the whole thing. And this juxtaposition of these two experiences in one single week was both very challenging and extremely enlightening.

Arvid:

So today, I wanna share with you the story of navigating the good and the bad simultaneously. My name is Albert Keil and this is the Bootstrap Founder. This episode is sponsored by acquire.com. Now, let me get to the good news. On Tuesday night, Peter Levels decided to give PodScan.fm a try, and I had tweeted about how PodScan found a mention of itself in a podcast.

Arvid:

And Peter saw it, set up an account, and immediately got his first mention. That was really lucky. I mean, people talk about him a lot, so it's not very surprising that eventually somebody would talk about Peter Davos, but his timing was just perfect. Right? He signed up, he put in the alert, and a couple minutes later, that was his first mention, got his first email into a podcast that he didn't know he was gonna be mentioned on.

Arvid:

So he was impressed enough at this time to then tweet about this to his over 300,000 followers. I got the levels bump. And this shout out led to a 50% increase in my user base within like 24 hours. It was crazy from a 1000 to 1,500 users overnight. It was incredible to see so many people, including several of my tech founders and heroes start using PodScan just from this one tweet.

Arvid:

Now Peter Levels is known for being very selective with his endorsements. So when he praised PodScan, I think it carried significant weight with the community. His tweet read somewhere along the lines of I just tried this and within minutes, I got alerted about a podcast match, super impressive, Arvid's project is really cool. I'm not affiliated, but I just think it's really cool and I love this. This kind of organic endorsement from someone so influential was a game changer for me and the product and everything that has to do with it.

Arvid:

Because immediately after this tweet, my user registration skyrocketed, and it wasn't just the sheer number of new years that was astonishing for me, but also the caliber of people signing up, that was incredible. But a lot of my indie hacker heroes were there. Several tech millionaires and founders of venture backed startups, like CTOs of big venture backed startups. I'm not gonna mention names or companies, but you would not believe it. Like, the kind of people that all of a sudden were signing up to PodScan that I many of them were on my personal dream customer list that I talked about a couple of weeks ago.

Arvid:

Candidates for that list that I would reach out to in the next year or so, they are now already using the product, which causes a lot of headaches because I'm like, oh, is it ready yet? But in the end, it really is. If Peter can use it, so can they. And people just started using it. It felt surreal to see names of the people that I admire, that I had read books from just appear in my user list, like, on my Slack channel for new sign ups.

Arvid:

I get an email for every new sign up, and when I woke up in the morning, my email the voice folder was just full of hundreds of new sign up emails. So it's probably one of the best feelings you can have. Maybe only better feeling probably is hundreds of subscription emails, but still, it's it was awesome.

Arvid:

It was a lot of fun.

Arvid:

And this influx of new users brought super valuable feedback and plenty of feature requests. And all of that, I used for validating my own assumptions about the jobs to be done that people had. Right? I'm now starting to get into conversations with lots of these people just to figure out, okay, so did you just sign up because Peter told you to or did you sign up because you wanted to and Peter kind of gave you that last push. I'm just trying to figure out where people are, if they are part of my perfect audience or not.

Arvid:

There will obviously be some churn from this, just in terms of people checking it out and not using it, but it's perfectly fine. I know that there's an opportunity for me to drill even further into my audience right now. And each of these users brought a fresh new perspective and these insights are just super invaluable, right? Like you have this moment of abundance and immense motivation when people like this join your product and I felt it so strong. So I started engaging with people through emails and customer support, so I could start to learn about their needs and how PodScan can better serve them in the future.

Arvid:

And it's pretty clear to me that this growth spurt was not just a temporary spike in my sign ups, but it is still an ongoing massive opportunity to refine and enhance my product based on real user feedback. So I'm digging my claws into these people now because I have them now using this or at least having signed up, having given me the email for a little bit of communication. Right? I'm not gonna spam them. I don't like that.

Arvid:

I'm not a fan of this, but I will try to reach out and see if there is some kind of alignment. And what I saw already and what made this moment even more special to me was seeing how quickly people recognize the innate value of PodScan. Peter is the best example because that's probably what made his tweet so interesting. He has a name that is often misspelled, like it's Peter with an I and not just Peter with an e, right, it's I e. And he immediately included these common misspellings in his alert settings because he's a smart dude.

Arvid:

And he got the alert for Peter with an e and not with I, his actual name. And he shared that as a screenshot. And I think this immediate level of utility made a significant impression on new users and that solidified their trust in the product, right? They saw, okay, here's a smart person using a tool smartly getting results. This is for me, I'm also smart.

Arvid:

There's just something in there. And so far, so amazing. That was a wonderful positive thing this week, but that was only one side of my week. On the flip side, I was grappling with the challenges of building my full text search feature I've been trying to work on for the last couple weeks to replace or at least augment my current search engine, and it just wouldn't work. So here I am getting this influx of new users, but all I can think about and all I can do is work on this weird full text search problem.

Arvid:

Because my database right now, that all these users are getting a benefit from and filling with new alerts and the new podcasts that are coming in every day is 100 of gigabytes and millions of rows, and it's too large to handle efficiently just by the database itself for search. I recently attempted to add a full text index to my podcast episode transcripts and all of that, so I could run searches more independently from external tools. And I'll talk about the specifics of what I use and what I now ended up using, but trying to do all of this while also running a production system led to a lot of performance issues. Surprisingly, of course, it will. Right?

Arvid:

I was very optimistic when I just tried adding a full text index, something that is incredibly complicated to a database of, like, 500 gigabytes of text data. The database would lock up and the speed of the queries that would normally be sent drastically decreased. It was frustrating and a time consuming problem. And I even learned about more things like blue green deployments on AWS and Percona SQL toolset, made schema changes and hot swept the databases, and there was all of that stuff and ran it in a bigger box, ran it in a bigger instance on the the AWS Cloud, and nothing seems to work. To give you a sense of the scale here, imagine this database with tens of millions of rows and each of these rows is filled with a giant chunk of text data, a full transcript of a podcast episode.

Arvid:

Thousands of word being spoken with time stamp data. So searching through this data obviously requires a lot of processing power and a lot of memory. And the machine that I run this on does not have enough memory to keep the whole database in RAM, so it has to use disk and disk is slow. So when I initially tried creating regular indexes on the data, not even full text, just regular, so you could do like a partial search. I hoped it would speed up the search process, but the sheer volume of data meant that even with indexing, performance was not usable.

Arvid:

You would create and it would take like 2 minutes to come back. So then I attempted to add the full text index, which is designed to handle large amounts of text data more efficiently. It does this kind of scoring like I'm not sure how to exactly call it, but the idea is you kinda fragment every single piece of text, every single text into the individual words, then you keep an index of how often these words appear per word in each of the texts that are in the database. And then you look for a search query, you take the individual words, and you look for frequencies of use, and then you kinda bring those things to the front. That's kinda how this works.

Arvid:

That's a very lame description of it, but that's how a full text index works on a very abstract level. And it just takes a lot of time. It takes a lot of time to build the index, it takes a lot of time to query the index, And the indexing process itself when I ran it would look up the database for hours and that is really bad because it makes it inaccessible for anything else. That includes users logging in or me adding new podcasts, it just doesn't work. And I used this blue green deployment that AWS has, which is kind of you spin up a complete clone of your database system, which is kept in sync with the original, and then you can do stuff on the clone that doesn't affect the original and then you kinda hot swap them once your changes are done.

Arvid:

And even then, I created the index on there. It took me like 8 hours just to create the index hot swapped, but performance took minutes. Sometimes it would even lock up that database just to try and run a full text query. It was pretty horrible. And so MySQL full text search on 500 gigabytes of text data, yeah, that's not happening.

Arvid:

And the breaking point for me personally came when the tool that I was currently using for search outside of MySQL, it's called MyVsearch, started locking up due to well, I don't really know due to what. I think it was excessive indexing or just a lot of access of of the files, the underlying files. It just started, like, not responding to requests and that was a problem because that is search and search is pretty central for PodScan, right? It's a it may not be the feature right now because alerting is important, but the API has searched, the UI has searched, and people wanna search, people wanna see stuff. If a new user comes onto the platform and can't search for the name, well, how can they trust that I can find it, right?

Arvid:

It's very important. So I've been using MileySearch since the start and it's pretty powerful. It can handle large datasets, it's meant to and I think it's just my setup and the way I use it was pushing it to its limits and it had a little bug and that was frustrating. So I voiced these frustrations on Twitter as I usually do because I know maybe somebody out there can help me or maybe somebody out there has the same problem and we can commiserate. And to my surprise, the leadership of the Miley Search team and the CEO and the CTO, they started reaching out to help.

Arvid:

And not only did they eventually provide a solution, but they also guided me in how to deploy this fix without needing to do a full migration. And that was unexpected. Like it's an open source project, right? Maillesearch is open source, they have a paid solution that's kind of how they monetize where they host this for you and then charge you money on a monthly basis. So an interesting approach to monetize open source software to begin with.

Arvid:

But the Miley Search binary and everything around it is open source. So seeing these people so active on Twitter trying to help me out because I had an interesting project and I had an interesting problem and then actually fixing the bug that was causing this lockup and telling me how to install the binary, like literally giving me the steps. That was really cool and it did make me rethink my whole approach. Right? Their support and their responsiveness were phenomenal, and they transformed my scarcity mindset in that moment into a problem solving opportunity.

Arvid:

Instead of running away from the issue, why not just tackle it and fix it for good? In parallel, I faced another and related infrastructure issue with Mighty Search, the current instance that I'm running. Well, like my database, it was growing rapidly and it uses a lot of disk because it has to be able to keep everything in RAM for for quick indexing, but it needs to live somewhere. That data needs to be on a disk And that disk was nearing its capacity. So Hetzner, my hosting provider, offers several cloud servers and I took the biggest one to start out with.

Arvid:

I think it has a 960 gigabyte disk and I think I'm at 49% of that. So I know that in another month or so, when I grow into this, the disk will just be full. And I needed to think about, well, how am I gonna survive the next year? I need a bigger disk. I need a bigger SSD, and it needs to be SSD.

Arvid:

It can't just be a kind of an attached SSD, a network drive, or even a hard drive, like a regular spinning disk hard drive. It needs to be an SSD that is, like, immediately connected to the server because of the the hardware software combination that Mylesearch uses to kind of, database system they use internally to store that data. It needs to be on an SSD, it doesn't work on a regular HDD. So I had no options because Headstone did not provide an upgrade version to this on their cloud instances. And AWS was also proving to be inadequate for my needs because the thing that I can afford there with EBS, their kind of block storage that they have also doesn't work with Meily.

Arvid:

MIME is, from what I understand, it has very specific things that it needs, so I thought I was out of luck. But look at the rest of the week, growth mindset. That's what I told myself. Let's not allow scarcity to drive my decisions. Let's change stuff because I noticed that my view on all of this was quite narrow and limited.

Arvid:

And I asked myself, how can I zoom out and look for the non obvious solution? And when I let go of what I thought my options were, the limited options that I thought I had, a thought struck me. Well, cloud isn't the only place that I should look at. And Hetzner, fortunately, is a place where cloud stuff is just one of its many offers. And it's really funny because the solution to this was staring me right in the face.

Arvid:

The whole week, I just didn't notice it. Because over the last 10 days or so, I migrated my whole transcription server fleet into dedicated GPU based servers on Hetna. Quick story about that. 2 weeks ago, I was looking at roughly $16,000 in monthly expenses for GPU compute. That's a lot.

Arvid:

And transcribing all podcasts everywhere isn't the cheapest thing to do, and I I know that, but I knew also there was savings potential. But I would think I was running a 10 graphics cards for 75¢ US per hour. And I thought, well, that's quite expensive. Let's see if there's something more. So as I always do, I tweet about this and I ask people, well, what would you think?

Arvid:

What would you use? Is there something better, something cheaper, something comparable but cheaper? And people suggested dozens of platforms and different approaches to me. And one of them was Hetzner's GEX 44 dedicated server, which is a 14 core 64 gigabyte RAM machine with an NVIDIA RTX 4 1000 SFF Ada Generation GPU. It's a mouthful, but it's really good.

Arvid:

It's a really good machine learning and AI GPU for under $200 a month. If I calculate this into dollars per hour, it's 27¢ per hour. Now compare this with my $0.75, right? That was a third of what I was paying for GPU for a slightly more efficient card before. If you compare the the RTX 4000 SFF auto generation to the a 10, you'll see that the a 10 is like maybe 15% better, but 15% better for 3 times the price.

Arvid:

So I ordered one of these servers immediately. I set up a few tests and did some quick math. If I were to migrate my server fleet over, I could push the $16,000 a month down to $9,000 a month within a day within the migration. And I could save $7,000 per month just by switching over to these dedicated servers. So I did.

Arvid:

I got 14 servers, then they stopped giving me them for just, like, you know, being able to automatically create new ones and I needed to put in an order for more. And now I'd run 30 of these servers in parallel on Hetzner all dedicated. And they're super they're excellent. They are very cost effective. And isn't that something I was also looking for as infrastructure for my search server?

Arvid:

Well, yes, right? And when that clicked in my mind, I explored that dedicated server options further. I looked outside of the GPU server because I don't need that for my my search, and I found a Hetson server with a 4 terabyte rate 1 mirror and a massive amount of RAM. Like, I think 300 gigabytes of RAM or something, perfectly suited for my needs at a very reasonable cost, just $300 a month. It's crazy.

Arvid:

Like on AWS, this would probably have been at least 5 times that, if not 10 times that. 300 gigs of RAM and 4 terabytes of SSD storage, like enterprise grades, data center grade storage, $300 a month? Yeah. Sure. I I'll take it.

Arvid:

So I booked one of these. I ordered 1 and it just came in. As I'm recording this, just an hour ago that the order went through and now it's provisioned, it's being provisioned. So this server will future proof my infrastructure for search for at least a good year or so because, 4 terabyte is 10 times the amount of data that I have right now. So if I will 10 x my data within the next 10 or 12 months, which is not unlikely, I will still be fine with the server and then I can start charting and maybe get a bigger one or whatever.

Arvid:

I'll figure it out then. Right now, this is a year of me not having to deal with this and it will allow me to provide reliable service to my customers because the mighty search people are now in my Twitter accounts, right? The people the wonderful people who were extremely helpful, who have looked into the issue that I had and then a couple days later actually pushed a new version that solves that bug. I can't help but see that there was some kind of connection between these two things and it is really cool. The decision to move my mighty search instance to this new server is a turning point for me because I'm leaning into mighty more for search and I will walk away from trying to force my main database to also be a search engine.

Arvid:

It's just gonna be a database and maybe I'll set another search engine like Sphinx or Manticore on top of this at some point. But right now, Miley Search will do the trick and it's gonna be enough. And amidst these technical challenges, just another story, it just came to mind, another incident underscored this whole theme of ups and downs for the week. Remember Peter's tweet? This wonderful thing that got me all these hundreds of new users.

Arvid:

Well, just minutes after Peter's tweet about PodScan, the service that I use for emails to send all my transactional emails, which includes verification emails, postmark experienced an outage. Perfect timing, right? If there is one email you wanna be sent quickly and wanna be received quickly for people to click on, it's the account verification email directly after somebody signs up. Well, not that day. The delay in sending registration emails was so services And it really highlighted my dependency on external services as a solopreneur.

Arvid:

This is just so bad. You can't do anything about this. You can't, like, invent a new email service or send them manually. It's just not happening. The timing could not have been any worse.

Arvid:

And imagine the frustration of new users just eagerly waiting to verify their accounts for something that somebody else recommended to them only to experience, like, 5, 7 minutes of a delay. It was a lesson in how external factors will impact user experience and how little I can do about it. It was okay. It was fine. The people from Postmark, they found Peter Street, they found my little complaint about it, they apologized, and it's all good.

Arvid:

Like, some things just happen and they fixed it quickly. They were communicative about it. It was great, but it all worked out in the end. I added the little reply to Peter Street, because that was the source that funneled people into PodScan, and I told them, hey. There's a little delay.

Arvid:

Don't worry. It'll come. I got, like, 3 complaints about it overall out of 100 of people signing up. It was fine. And despite these setbacks and maybe because of them, this week was filled just with great opportunities for learning for me.

Arvid:

From Peter Street, I gained these valuable feedbacks from new users and that allows me to refine PodScan. My struggles with this full text indexing stuff taught me to just value the intricacies of SQL a bit more and understand indexing and various search engine technologies more. It was really cool. I discovered tools like Sphinx and Manticore and expanded my knowledge and the options for future projects inside of PodScan and maybe beyond. It's just nice to know stuff and know kinda how it works.

Arvid:

I mean, remember my explanation of how full text search indexes work? Well, I kinda know how it works, so that's all I need to do. It's the 80 20 rule of entrepreneurship. Understand it well enough to use it and then dive deep if you need to. And I revisited my existing setup and realized that I had misconfigured some of the indexing parameters in my research as well.

Arvid:

And then I tweak these a little and optimize the indexing process there, and I managed to achieve significant performance improvements just in the quality of search, which was also one of the reasons that I initially wanted to move towards full text search because it's more controllable. But mighty can do this. I just need to configure it right. That's really all it is. I just needed to understand ranking better.

Arvid:

That was what it was. I needed to understand how the results are ranked, by what order of potential ranking candidates they are ranked, and then I needed to adjust that and it was fine. So, yeah, you know. The real lesson from this week was that sometimes the solution lies in making better use of the tools that I already have. Right?

Arvid:

It's this importance of embracing both success and failure as opportunities for growth with what we're already given. The influx of users from Peter Street just opened doors to insights. My technical challenges led to a deeper understanding and better infrastructure solutions with a new server. And every setback was a chance to learn and improve, and that made me feel fortunate for the experiences even though they were sometimes a bit painful. And what seemed like a week of contrast turned into a week of great joy and learning.

Arvid:

I cut my expenses, like, almost by half. I got we increased my user base by 50%, like, that happened at the same time too. I deepened my understanding of my stack, my tech stack, and I discovered the value of just being persistence and finding support in my community, both from peers for motivation and from the experts that are building the tools that I use for actual technical support. And this duality of abundance and scarcity of success and failure ultimately enriched my journey as a founder. It's the spice that makes entrepreneurship interesting and it reminds me that the path of entrepreneurship is never straightforward.

Arvid:

It's always filled with these highs and lows and each of them offers its own set of lessons, right? Good lessons, bad lessons, just always lessons. And my week of contrast was a testament to this reality. So the support from the tech community, this resilience to tackle infrastructure challenges, and this joy of seeing my products in the hands of influential users like Peter sharing with more influential users that I know getting to use it is all part of the journey. And reflecting on the week, I realized that the key to navigating this, these contrast, is to maintain a mindset of continuous learning and adaptability, a growth mindset.

Arvid:

It's not about the hustle or the grindset, that's not what I'm talking about. It's more about not cutting off learning opportunities before they come to fruition. Not be so dialed in not to look only so narrowly focused onto your issues, but to broaden your view and see things outside of it. Because every problem is a chance to zoom out and improve the whole thing and then zoom back in and improve the initial thing that you're looking at. And every success is a little stepping stone to further improvement.

Arvid:

This mindset not only helps me in overcoming these challenges, but also in building a product that truly meets the needs of its users because I'm motivated by them to work with them and for them, right, involving people. Now that is just the most important part. I'm grateful for the experiences of this week, and they've reinforced my belief in the importance of community, open source and beyond, persistence is just not letting go and trying to find a way to make things work, and the willingness to learn from every situation. So as I move forward, I'm more committed than ever to refine PodScan and making it an indispensable tool for my users because they tell me they want it. They do like it.

Arvid:

And this is the journey of a bootstrap, alright. I'm excited to continue this journey with all its ups and downs and share it here with you. I wanna briefly thank my sponsor, acquire.com. Imagine you've been building this amazing solid SaaS product, you have had a lot of technical challenges and a lot of ups and downs, but you ultimately acquired customers and it's generating good MRR, so SaaS stream, but you don't really care about it anymore. You just hit a ceiling, skill ceiling, a focus ceiling, some kind of ceiling, you don't know what to do.

Arvid:

Well, what you shouldn't do is give up and just let it go to waste. What you should instead do is to think about how you can turn what you have in value into compensation, into money effectively. How you can take the thing that is valuable and give it to somebody who will give you money for that value. And instead of doing that, a lot of founders often just do something else. They go on to the next project and the old project loses its value, it stagnates, it's there's inaction, and the business becomes less valuable over time or completely worthless in the end.

Arvid:

So if you find yourself here and you think your story is likely headed down a similar road, I would consider the 3rd option here and that's selling your business on acquire.com. There's somebody out there who really wants to buy your thing and wants to keep building it that has motivation that you don't have right now, and they also have money that you don't have right now. So capitalizing on the value of your time today is a pretty smart move. Somebody will want to buy the business that you are kinda meh about at this point. Acquire.com is free to list.

Arvid:

They've helped hundreds of founders already. So go to try.aquire.com/arbit and see for yourself if this is the right option for you. Thank you so much for listening to the Bootstrap Founder today. You can find me on Twitter at Avid Kal, avid k a h l. You find my books on my Twitter course there too.

Arvid:

Please check out podscan dot f m if you're interested in the world of podcasts. And if you wanna support me and the show, please subscribe to my YouTube channel, get the podcast in your player of choice, and leave a rating and a player of choice and leave a rating and a review by going to rate this podcast.com/founder. Makes a massive difference if you show up there because then the podcast will show up in other people's feeds. And any of this will truly help the show. Thank you so much for listening.

Arvid:

Have a wonderful day, and bye bye.

Creators and Guests

Arvid Kahl
Host
Arvid Kahl
Empowering founders with kindness. Building in Public. Sold my SaaS FeedbackPanda for life-changing $ in 2019, now sharing my journey & what I learned.
330: 50% New Users Overnight… and a Burning Server
Broadcast by