As someone who worked with serverless for multiple years (mostly amazon lambda but others too) i can absolutly apporove the authors points.
While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems".
Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data.
Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks.
While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy".
For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.
The way to work around this issue is to provide a presigned S3 url
Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something)
I agree this is annoying and maybe I’ve been in AWS ecosystem for too long.
However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit
Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also.
In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more.
And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs.
But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place.
Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.
Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO.
Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards.
I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.
That’s why Knative (Serverless on Kubernetes) accepts containers. It’s the standard packaging format that lets you lift and shift apps to many different platforms.
One thing I could not find in the write-up is the change in the expense. Did serverless save any money, compared to always-up VMs? Did much of their load run under the free tier limits?
Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)
I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster.
Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.
I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc.
Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be cached in the server? What can be cached in the client? etc etc.
And then of course you can split and do some things on the server and some in the edge…
I think the "local maximum" we've gotten stuck at for application hosting is having a docker container as the canonical environment/deliverable, and injecting secrets when needed. That makes it easy to run and test locally, but still provides most of the benefits I think (infrastructure-as-code setups, reproducibility, etc). Serverless goes a little too far for most applications (in my opinion), but I have to admit some apps work really well under that model. There's a nearly endless number of simple/trivial utilities which wouldn't really gain anything from having their own infrastructure and would work just fine in a shared or on-demand hosting environment, and a massively scaled stateless service would thrive under a serverless environment much more than it would on a traditional server.
That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.
You are ready for misterio: https://github.com/daitangio/misterio
A tiny layer around stareless docker cluster.
I created it for my homelab and it gone wild
Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be.
There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces?
Worse, the big reason it was useful, managing dependency hell, has largely been solved by making developers default to not installing dependencies globally.
We don't really need Docker anywhere near like we used to, and yet it persists as the default, unassailable.
Of course hosting companies must LOVE it, docker containers must increase their margins by 10% at least!
Someone else down thread has mentioned a tooling fetish, I feel Docker is part of that fetish.
I think developers are drowning in tools to make things "easy", when in truth many problems are already easy with the most basic stuff in our tool belt (a compiler, some bash scripts, and some libraries). You can always build up from there.
This tooling fetish hurts both companies and developers.
Yeah; IMO Docker was our last universal improvement to productivity, in 2013, and very little we've invented since then can be said to have had such a wide-ranging positive impact, with such few drawbacks. Some systems are helpful for some companies, but then try to get applied to other companies where they don't make sense and things fall apart or productivity suffers. Cloudflare and others are trying to make v8 isolates a thing, and while they are awesome for some workloads, people want them to be the "next docker", and they aren't.
The model "give me docker image, we put it on internet" is staggeringly powerful. It'll probably still be the most OP way to host applications in 2040.
Docker + IaC* for me; git ops, immutable servers, immutable code, immutable config, (nearly) immutable infrastructure means I haven't had to drop to the command line on a server since 2015. If something is wrong you restart the container, if that doesn't work you restart the host it's running on. The "downside" to this is my "admin" shell skills outside of personal dev laptop commands have gotten rusty.
> If something is wrong you restart the container, if that doesn't work you restart the host it's running on
Haha, lucky you. If only world was this beautiful :) I regularly shell into Kubernetes nodes to debug memory leaks from non-limited pods, or to check some strange network issues.
It’s that, and the fact that precious few people seem to understand fundamentals anymore, which is itself fed by the desire to outsource everything to 3rd parties. You can build an entire stack where the only thing you’ve actually made is the core application, and even that is likely to be influenced if not built by AI.
And that is actually the advantage of serverless, in my mind. For some low-traffic workloads, you can host for next to nothing. Per invocation, it is expensive, but if you only have a few invocations of a workload that isn't very latency sensitive, you can run an entirely serverless architecture for pennies per month.
Where people get burned is moving high traffic volumes to serverless... then they look at their bill and go, "Oh my god, what have I done!?" Or they try to throw all sorts of duct tape at serverless to make it highly performant, which is a fool's errand.
Exactly. I've always found that how people want to use lambda is the exact opposite of how to use it cost effectively.
I've seen a lot of people want to use lambdas as rest endpoints and effectively replace their entire API with a cluster of lambdas.
But that's about the most expensive way to use a lambda! 1 request, one lambda.
Where these things are useful is when you say "I have this daily data pull and ETL that I need to do." Then all the sudden the cost is pretty dang competitive.
The amount of 0s in the price per second is mesmerizing, but just multiply this by 24h and 30 days, and you are well within the price range of a better EC2 with much better performance, plus you can process 1000 req/s instead of 1 req/s for the same price.
"Cheap" is relevant if you are talking about work load that is one off and doesn't run continuously. A lot of people use serverless to run a 24-7 service which sort of defeats the purpose. It doesn't get that cheap anymore.
Serverless is good if you have one off tasks that are used intermittently and are not consistent.
For a best price-to-performance ratio create your instances and do whatever is needed on them. Software stacks are not that complicated to delegate everything to the Wizards of Cloud Overcharging.
The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.
Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem.
However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing.
I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.
> the documentation is closer to marketing than truthful technical documentation
I participated in AWS training and certification given by AWS for a company to obtain a government contract and I can 100% say that the PAID TRAINING itself is also 100% marketing and developer evangelism.
100% agree with you. I took a corporate training, and at one point crammed for the developer cert. It it just marketing. There is never a question where the answer is "Just run this service on EC2 yourself". It is about maximizing your usage of AWS services.
Infra will always be full of so much nonsense because it’s really hard to tell successful developers their code and system design is unusable. People use it because they are paid to do so usually, but it’s literally some of the worst product development I’ve ever seen.
AWS will hopefully be reduced to natural language soon enough with AI, and their product team can move on (most likely they moved on a long time ago, and the revolving door at the company meant it was going remain a shittily thought out platform in long term maintenance).
Some things never change. I remember ~20 years ago a bunch of expensive F5s suddenly showing up to our offices because the CTO and enterprise architects were convinced that irules could solve all their performance problems for something that wasn't even cacheable (gaming results) and would have shoved too much of our logic into the underpowered CPUs on them.
They were a much nicer, if overpriced, load balancing alternative to the Cisco Content Switch we were using, though.
This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.
I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me.
I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly.
One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL.
To get anycast working, you need BGP, and to get it working well, I think you need a good understanding of BGP and a lot of points of presence and well connected at each. BGP's default metric of distance is number of networks traversed, which does funny things.
Say you're in city A where you use transit provider 1 and city B where you use transit provider 2. If a user is in city B and their ISP is only connected to transit provider 1, BGP says deliver your traffic to city A, because then traffic doesn't leave transit provider 1 until it hits your network. So for every transit network you use, you really want to connect to it at all your PoPs, and you probably want to connect to as many transit networks as feasible. If you're already doing multihoming at many sites, it's something to consider; if not, it's probably a whole lot of headache.
GeoDNS as others suggested is a good option. Plenty of providers out there, it's not perfect, but it's alright.
Less so for web browsers, but you can also direct users to specific servers. Sample performance for each /24 and /48 and send users to the best server based on the statistics, use IP location as a fallback source of info. Etc. Not great for simple websites, more useful for things with interaction and to reduce the time it takes for tcp slow start (and similar) to reach the available bandwidth.
You could start using DNS Traffic Shaping where DNS server looks at IP making the request and returns the IP of closest server.
Azure/AWS/GCP all have solutions for this and does not require you to use their services. There are probably other DNS providers that can do it as well.
Cloudflare can also do this as well but it's probably more expensive than DNS.
You took the words right out of my mouth. Between aggressive salespeople marketing any given product as a panacea for everything and mandates from above to arbitrarily use X thing to do Y, there’s a lot of just plain bad architecture out there.
I think they are shooting themselves in the foot with this approach. If you have to run a monte carlo simulation on every one of their services at your own time and expense just to understand performance and costs, people will naturally shy away from such black boxes.
> people will naturally shy away from such black boxes.
I don't this isn't true. In fact, it seems that in the industry, many developers don't proceed with caution and go straight into usage, only to find the problems later down the road. This is a result of intense marketing on the part of cloud providers.
The fact is most developers in most companies have very little choice. Many medium to large companies (1k-50k employees) the CTO gets wined and dined by AWS/Azure/Oracle and they decide to move to that cloud. They bring in their solutions architects and do the training. The corporate architects for the divisions set the goals. So the rank and file developers get told that they have to make this work in AWS using RDS and they have almost zero power over this choice.
I feel like every cloud build meeting should have a moment where everyone has to defend the question "Wait! could this be a regular database with a regular app on a server with a regular cache?"
Bo Burmham said, "self awareness does not absolve anyone of anything"
But here I dont think they (or their defenders) are still aware of the real lesson here.
Theres literally zero information thats valuable here. Its like saying "we used an 18 wheeler as our family car and then we switched over to a regular camry and solved all our problems." What is the lesson to be learned in that statement?
The real interesting post mortem would be if they go, "god in retrospect what a stupid decision we took; what were we thinking? Why did we not take a step back earlier and think, why are we doing it this way?" If they wrote a blog post that way, that would likely have amazing takeaways.
What did your internal discussion conclude for the question "Why did we not take a step back earlier and think, why are we doing it this way?"
Im genuinely curious because this is not singling out your team or org, this is a very common occurrence among modern engineering teams, and I've often found myself on the losing end of such arguments. So I am all ears to hear at least one such team telling what goes on in their mind when they make terrible architecture decisions and if they learned anything philosophical that would prevent a repeat.
Oh we had it coming for quite some time and knew we would need to rebuild it, we just didn’t have the capacity to do it unfortunately.
I was working on it on and off moving one endpoint at a time but it was very slow until we hired someone who was able to focus on it.
It didn’t feel good at all. We knew the product had massive flaws due to the latency but couldn’t address it quickly. Especially cause we he to build more workarounds as time went on. Workarounds we knew would be made redundant by the reimplementation.
I think we had that discussion if “wtf are we doing here” pretty early, but we didn’t act on it in the beginning, instead we tried different approaches to make it work within the serverless constraints cause that’s what we knew well.
I have had CTOs (two in my career) tell me we had to use our AWS credits since they were going to expire worthless. Both experiences were at vc-backed startups.
What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.”
I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics.
And yeah you’re right in hindsight it was a terrible idea to begin with
I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it
You don't need experience and there is not really a lot to know about "distributed systems" in this case, that's basic CS knowledge about networks, latency and what "serverless" actually is, you can read about it.
To be honest, to me it reads like people who don't understand the problem they're solving, haven't acquired the necessary knowledge to solve it (either by learning themselves or by asking/hiring people who have it), and seeing such an amateurish mistake doesn't inspire confidence for the future.
You should either hire people that know what they are doing or upgrade your knowledge about systems you are using before making decisions to use them.
Sometimes I see a post about sorting algorithms online. Some people seem to benefit from reading about these things, but often, I find there isn't much new information for me. That's OK, because I know somebody somewhere benefits from knowing this.
It is your decision to make this a circlejerk of musings about how the company must be run by amateurs. Whatever crusade you're fighting in vividly criticising them is not valuable at all. People need to learn and share so we can all improve, stop distracting from that point.
I would not assume this was a "rookie mistake". I've been here once or twice, and a common story is that engineers don't want to do it a certain way, but management overrules them for some vague hand-wavy reason like, "This way is more modern." Another common story is that you know you're not choosing the most [scalable|robust|performant|whatever] design, but ancillary constraints like time and money push you into a "worse is better" decision.
Or maybe the original implementation team really didn't know what they were doing. But I'd rather give them the benefit of the doubt. Either way, I appreciate them sharing these observations because sharing these kinds of stories is how we collectively get better as a professional community.
My personal experience is that if you want guaranteed anything (quick scaling, latency, CPU, disk or network throughput), your best bet is to manually provision EC2 instances (or use some API that does). Once you give up control hoping to gain performance for free, you usually end up with an unfixable bottleneck.
If you're looking for a middle ground between VMs and serverless, ECS Fargate is a good option. Because a container is always running, you won't experience any cold start times.
Yes, though unless you’re provisioning your own EC2s for them to run on, you have no guarantee about the server generation, and IME AWS tends to provision older stuff for Fargate.
This may or may not matter to you depending on your application’s needs, but there is a significant performance difference between, say, an m4 family (Haswell / Broadwell) and an m7i family (Sapphire Rapids) - literally a decade of hardware improvements. Memory performance in particular can be a huge hit for latency-sensitive applications.
ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.
You're confusing network proximity with application architecture. Edge deployment helps connection latency. Stateless runtime destroys it by forcing every cache access through the network.
The whole point of edge is NOT to make latency-critical APIs with heavy state requirements faster. It's to make stateless operations faster. Using it for the former is exactly the mismatch I'm describing.
Their 30ms+ cache reads vs sub-10ms target latency proves this. Edge proximity can't save you when your architecture adds 3x your latency budget per cache hit.
Realistically, they should be able to do sub ms cache hits which land in the same datacenter. I know cloudflare doesn't have "named" datacenters like other providers but at the end of the day, there are servers somewhere and if your lambda runs twice in the same one there is no reason why a pull-through cache can't experience a standard intra data-center latency hit.
I wonder if there is anything other than good engineering getting in the way of this and even sub us intra-process pull through caches for busy lambda functions. After all, if my lambda is getting called 1000X per second from the same point of presence, why wouldn't they keep the process in memory?
On serverless, whenever you call your code, it has to be executed but first the infrastructure has to find a place to run it and sometimes if there's no running instance available, it must fire up a new instance to run your code.
- Eliminated complex caching workarounds and data pipeline overhead
- Simplified architecture from distributed system to straightforward application
We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions.
Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever.
Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.
Last I heard (~5 years ago), lambda@edge doesn't actually run on edge POPs anyway; they're just hooks that you can put in your edge configs that execute logic in the nearest region before/after running your edge config. But it's definitely a datacenter round-trip to invoke them.
Adding that much compute to an edge POP is a big lift; even firecracker gets heavy at scale. And security risk for executing arbitrary code since these POPs don't have near the physical security of a datacenter, small scale makes more vulnerable to timing attacks, etc.
Their problem isn't serverless, rather Cloudflare Workers and WebAssembly.
All major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.
It can be good for connecting AWS stuff to AWS stuff. "On s3 update, sync change to dynamo" or something. But even then, now you've got a separate coding, testing, deployment, monitoring, alerting, debugging pipeline from your main codebase, so is it actually worth it?
But no, I'd not put any API services/entrypoints on a lambda, ever. Maybe you could manufacture a scenario where like the API gets hit by one huge spike at a random time once per year, and you need to handle the scale immediately, and so it's much cheaper to do lambda than make EC2 available year-round for the one random event. But even then, you'd have to ensure all the API's dependencies can also scale, in which case if one of those is a different API server, then you may as well just put this API onto that server, and if one of them is a database, then the EC2 instance probably isn't going to be a large percentage of the cost anyway.
Serverless only makes sense if the lifetime doesn't matter to your application, so if you find that you need to think about your lifetime then serverless is simply not the right technology for your use case.
I would doubt that this is categorically true. Serverless inherently makes the whole architecture more complex with more moving parts in most cases compared to classical web applications.
> Serverless inherently makes the whole architecture more complex with more moving parts
Why's that? Serverless is just the generic name for CGI-like technologies, and CGI is exactly how classical web application were typically deployed historically, until Rails became such a large beast that it was too slow to continue using CGI, and thus running your application as a server to work around that problem in Rails pushed it to become the norm across the industry — at least until serverless became cool again.
Making your application the server is what is more complex with more moving parts. CGI was so much simpler, albeit with the performance tradeoff.
Perhaps certain implementations make things needlessly complex, but it is not clear why you think serverless must fundamentally be that way.
Depends pretty much where those classical web applications are hosted, how big is the infrasture taking care of security, backups, scalability, failovers, and the amount of salaries being paid, including on-call bonus.
Serverless is not a panacea. And the alternative isn't always "multiple devops salaries" - unless the only two options you see are server serverless vs outrageously stupid complicated kubernetes cluster to host a website.
There's a huge gap between serverless and full infra management. Also, IMO, serverless still requires engineers just to manage that. Your concerns shift, but then you need platform experts.
For a thing, which permanently has load it makes little sense.
It can make sense if you have very differing load, with few notable spikes or on an all in on managed services, where serverless things are event collectors from other services ("new file in object store" - trigger function to update some index)
Agree, it seems like they decided to use Cloudflare Workers and then fought them every step of the way instead of going back and evaluating if it actually fit the use case properly.
It reminds me of the companies that start building their application using a NoSQL database and then start building their own implementation of SQL on top of it.
Ironically, I really like cloudflare but actively dislike workers and avoid them when possible. R2/KV/D1 are all fantastic and being able to shard customer data via DOs is huge, but I find myself fighting workers when I use them for non-trivial cases. Now that Cloudflare has containers I'm pushing people that way.
In that scenario, how do you keep cold startup as fast as possible?
The nice thing about JS workers is that they can start really fast from cold. If you have low or irregular load, but latency is important, Cloudflare Workers or equivalent is a great solution (as the article says towards the end).
If you really need a full-featured container with AOT compiled code, won't that almost certainly have a longer cold startup time? In that scenario, surely you're better off with a dedicated server to minimise latency (assuming you care about latency). But then you lose the ability to scale down to zero, which is the key advantage of serverless.
Cloudflare has containers now too, and having used AppRunner and Cloud Run, it's much easier to work with. Once they get rid of the container caps and add more flexibility in terms of container resources, I would never go back to the big cloud containers, the price and ease of use of Cloudflare's containers just destroy them.
"Self-Hosting : Being tied to Cloudflare's runtime meant our customers couldn't self-host Unkey. While the Workers runtime is technically open source, getting it running locally (even in dev mode) is incredibly difficult.
With standard Go servers, self-hosting becomes trivial:"
A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.
After building my first Serverless/Cloudflare worker app, this is why I migrated to Deno.
Deno enables you to run the same codebase in deno (self-hosted/local) and in deno deploy (serverless platform from deno).
I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo).
I ported my worker project into Django since cloudflare workers wouldn’t allow selection of region for hosting workers which is generally required due to data compliances. This is something all cloud providers provide from day one yet cloudflare made it an enterprise feature.
Also the vendor lock-in doesn’t help with durable objects and D2 instead of simply doing what supabase and others are doing by providing Postgres or standard SQLite as a service.
That said, as an example, an m8g.8xlarge gives you 32 vCPU / 128 GiB RAM for about $1000/month in us-east-1 for current on-demand pricing, and that drops to just under $700 if you can do a 1-year RI. I’m guessing this application isn’t super memory-heavy, so you could save even more by switching to the c-family: same vCPU, half the RAM.
Stick two of those behind a load balancer, and you have more compute than a lot of places actually need.
Or, if you have anything resembling PMF, spend $10K or so on a few used servers and put them into some good colo providers. They’ll do hardware replacement for you (for a fee).
They just use two servers and configure a loadbalancer within Cloudflare. Come on. Self-Hosting is no rocket science. You don‘t have to make it seem complicated. People have been doing this decades before AWS invented serverless.
Incredible that these kinds of services were hosted like this.
I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering.
Why didnt they go with a self hosted backend right away?
Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.
I doubt they literally said “perfect for low latency APIs” but their messaging is definitely trying to convince you that they’re fast globally, just look at the workers.ckoudflare.com page
Have you done new benchmarks since Cloudflare announced their latest round of performance improvements for Workers?
Just curious if this workload also saw some of the same improvements (on a quick read it seems like you could have been hitting the routing problem CF mentions)
Really great writeup. The charts tell the story beautifully, and the latency gains are surely a win for your company and customers. I always wonder about the tradeoffs. Is there a measurable latency difference for your non-colocated customers? What does maintenance look like for your Go servers? I assume that your Cloudflare costs dropped?
It’s faster for non-colocated customers too weirdly
I think cause connections can be reused more often. Cloud flare workers are really prone to doing a lot of TLS handshakes cause they spin up new ones constantly
Right now were just hang aws far hate for the go servers, so there really isn’t much maintenance at all. We’ll be moving that into eks soon though cause we are starting to add more stuff and need k8s anyways
Unlikely? They could've just as well deployed their single go binary to a vm from day 1 and it would've been smooth sailing for their use case, while they acquire customers.
The cloudflare workers they chose aren't really suited for latency critical, high throughput APIs they were designing.
I often don't know what to make of DHH. He's a living contradiction. On one hand he will continually rant about how bad the overhead and waste of cloud services is, and on the other hand he will staunchly defend the most inefficient programming language that is regularly used for backend development, as well as defend the enourmous overfetching that active record leads to.
Really I think DHH just likes to tell others what he likes.
As someone who worked with serverless for multiple years (mostly amazon lambda but others too) i can absolutly apporove the authors points.
While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems".
Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data.
Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks.
While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy".
For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.
The way to work around this issue is to provide a presigned S3 url
Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something)
I agree this is annoying and maybe I’ve been in AWS ecosystem for too long.
However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit
Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also.
In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more.
And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs.
But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place.
Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.
Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO.
Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards.
I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.
Don’t all “serverless” platforms accept docker images? I know Cloud Run does.
If I understand correctly your concern is mostly with “serverless functions” which abstracts away even more.
That’s why Knative (Serverless on Kubernetes) accepts containers. It’s the standard packaging format that lets you lift and shift apps to many different platforms.
Let me tell you about all the fun I'm having trying to execute my amazon lambda app locally so I can test before deploying...
One thing I could not find in the write-up is the change in the expense. Did serverless save any money, compared to always-up VMs? Did much of their load run under the free tier limits?
Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)
I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster.
Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.
I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc.
Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be cached in the server? What can be cached in the client? etc etc.
And then of course you can split and do some things on the server and some in the edge…
I think the "local maximum" we've gotten stuck at for application hosting is having a docker container as the canonical environment/deliverable, and injecting secrets when needed. That makes it easy to run and test locally, but still provides most of the benefits I think (infrastructure-as-code setups, reproducibility, etc). Serverless goes a little too far for most applications (in my opinion), but I have to admit some apps work really well under that model. There's a nearly endless number of simple/trivial utilities which wouldn't really gain anything from having their own infrastructure and would work just fine in a shared or on-demand hosting environment, and a massively scaled stateless service would thrive under a serverless environment much more than it would on a traditional server.
That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.
You are ready for misterio: https://github.com/daitangio/misterio A tiny layer around stareless docker cluster. I created it for my homelab and it gone wild
Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be.
There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces?
Worse, the big reason it was useful, managing dependency hell, has largely been solved by making developers default to not installing dependencies globally.
We don't really need Docker anywhere near like we used to, and yet it persists as the default, unassailable.
Of course hosting companies must LOVE it, docker containers must increase their margins by 10% at least!
Someone else down thread has mentioned a tooling fetish, I feel Docker is part of that fetish.
I think developers are drowning in tools to make things "easy", when in truth many problems are already easy with the most basic stuff in our tool belt (a compiler, some bash scripts, and some libraries). You can always build up from there.
This tooling fetish hurts both companies and developers.
Yeah; IMO Docker was our last universal improvement to productivity, in 2013, and very little we've invented since then can be said to have had such a wide-ranging positive impact, with such few drawbacks. Some systems are helpful for some companies, but then try to get applied to other companies where they don't make sense and things fall apart or productivity suffers. Cloudflare and others are trying to make v8 isolates a thing, and while they are awesome for some workloads, people want them to be the "next docker", and they aren't.
The model "give me docker image, we put it on internet" is staggeringly powerful. It'll probably still be the most OP way to host applications in 2040.
Docker + IaC* for me; git ops, immutable servers, immutable code, immutable config, (nearly) immutable infrastructure means I haven't had to drop to the command line on a server since 2015. If something is wrong you restart the container, if that doesn't work you restart the host it's running on. The "downside" to this is my "admin" shell skills outside of personal dev laptop commands have gotten rusty.
*Terraform, imo, released in ~2014
> If something is wrong you restart the container, if that doesn't work you restart the host it's running on
Haha, lucky you. If only world was this beautiful :) I regularly shell into Kubernetes nodes to debug memory leaks from non-limited pods, or to check some strange network issues.
It’s that, and the fact that precious few people seem to understand fundamentals anymore, which is itself fed by the desire to outsource everything to 3rd parties. You can build an entire stack where the only thing you’ve actually made is the core application, and even that is likely to be influenced if not built by AI.
The industry is creating learned helplessness.
Helps resumes! No one got to Staff suggesting bash scripts
A lot of people don’t know about compilers, bash scripts and libraries.
Excerpt AWS lambda is stupidly cheap!
For certain workloads :)
And that is actually the advantage of serverless, in my mind. For some low-traffic workloads, you can host for next to nothing. Per invocation, it is expensive, but if you only have a few invocations of a workload that isn't very latency sensitive, you can run an entirely serverless architecture for pennies per month.
Where people get burned is moving high traffic volumes to serverless... then they look at their bill and go, "Oh my god, what have I done!?" Or they try to throw all sorts of duct tape at serverless to make it highly performant, which is a fool's errand.
Exactly. I've always found that how people want to use lambda is the exact opposite of how to use it cost effectively.
I've seen a lot of people want to use lambdas as rest endpoints and effectively replace their entire API with a cluster of lambdas.
But that's about the most expensive way to use a lambda! 1 request, one lambda.
Where these things are useful is when you say "I have this daily data pull and ETL that I need to do." Then all the sudden the cost is pretty dang competitive.
The amount of 0s in the price per second is mesmerizing, but just multiply this by 24h and 30 days, and you are well within the price range of a better EC2 with much better performance, plus you can process 1000 req/s instead of 1 req/s for the same price.
How long is a piece of string ?
"Cheap" is relevant if you are talking about work load that is one off and doesn't run continuously. A lot of people use serverless to run a 24-7 service which sort of defeats the purpose. It doesn't get that cheap anymore.
Serverless is good if you have one off tasks that are used intermittently and are not consistent.
For a best price-to-performance ratio create your instances and do whatever is needed on them. Software stacks are not that complicated to delegate everything to the Wizards of Cloud Overcharging.
The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.
I’ve found this to be true, with one caveat.
Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem.
However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing.
I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.
> the documentation is closer to marketing than truthful technical documentation
I participated in AWS training and certification given by AWS for a company to obtain a government contract and I can 100% say that the PAID TRAINING itself is also 100% marketing and developer evangelism.
100% agree with you. I took a corporate training, and at one point crammed for the developer cert. It it just marketing. There is never a question where the answer is "Just run this service on EC2 yourself". It is about maximizing your usage of AWS services.
Infra will always be full of so much nonsense because it’s really hard to tell successful developers their code and system design is unusable. People use it because they are paid to do so usually, but it’s literally some of the worst product development I’ve ever seen.
AWS will hopefully be reduced to natural language soon enough with AI, and their product team can move on (most likely they moved on a long time ago, and the revolving door at the company meant it was going remain a shittily thought out platform in long term maintenance).
Some things never change. I remember ~20 years ago a bunch of expensive F5s suddenly showing up to our offices because the CTO and enterprise architects were convinced that irules could solve all their performance problems for something that wasn't even cacheable (gaming results) and would have shoved too much of our logic into the underpowered CPUs on them.
They were a much nicer, if overpriced, load balancing alternative to the Cisco Content Switch we were using, though.
This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.
I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me.
I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
Someone correct me if I’m wrong but:
DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly.
One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL.
EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)
To get anycast working, you need BGP, and to get it working well, I think you need a good understanding of BGP and a lot of points of presence and well connected at each. BGP's default metric of distance is number of networks traversed, which does funny things.
Say you're in city A where you use transit provider 1 and city B where you use transit provider 2. If a user is in city B and their ISP is only connected to transit provider 1, BGP says deliver your traffic to city A, because then traffic doesn't leave transit provider 1 until it hits your network. So for every transit network you use, you really want to connect to it at all your PoPs, and you probably want to connect to as many transit networks as feasible. If you're already doing multihoming at many sites, it's something to consider; if not, it's probably a whole lot of headache.
GeoDNS as others suggested is a good option. Plenty of providers out there, it's not perfect, but it's alright.
Less so for web browsers, but you can also direct users to specific servers. Sample performance for each /24 and /48 and send users to the best server based on the statistics, use IP location as a fallback source of info. Etc. Not great for simple websites, more useful for things with interaction and to reduce the time it takes for tcp slow start (and similar) to reach the available bandwidth.
You could start using DNS Traffic Shaping where DNS server looks at IP making the request and returns the IP of closest server.
Azure/AWS/GCP all have solutions for this and does not require you to use their services. There are probably other DNS providers that can do it as well.
Cloudflare can also do this as well but it's probably more expensive than DNS.
You took the words right out of my mouth. Between aggressive salespeople marketing any given product as a panacea for everything and mandates from above to arbitrarily use X thing to do Y, there’s a lot of just plain bad architecture out there.
>> is to run your own performance testing
I think they are shooting themselves in the foot with this approach. If you have to run a monte carlo simulation on every one of their services at your own time and expense just to understand performance and costs, people will naturally shy away from such black boxes.
> people will naturally shy away from such black boxes.
I don't this isn't true. In fact, it seems that in the industry, many developers don't proceed with caution and go straight into usage, only to find the problems later down the road. This is a result of intense marketing on the part of cloud providers.
The fact is most developers in most companies have very little choice. Many medium to large companies (1k-50k employees) the CTO gets wined and dined by AWS/Azure/Oracle and they decide to move to that cloud. They bring in their solutions architects and do the training. The corporate architects for the divisions set the goals. So the rank and file developers get told that they have to make this work in AWS using RDS and they have almost zero power over this choice.
I feel like every cloud build meeting should have a moment where everyone has to defend the question "Wait! could this be a regular database with a regular app on a server with a regular cache?"
The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable
Bo Burmham said, "self awareness does not absolve anyone of anything"
But here I dont think they (or their defenders) are still aware of the real lesson here.
Theres literally zero information thats valuable here. Its like saying "we used an 18 wheeler as our family car and then we switched over to a regular camry and solved all our problems." What is the lesson to be learned in that statement?
The real interesting post mortem would be if they go, "god in retrospect what a stupid decision we took; what were we thinking? Why did we not take a step back earlier and think, why are we doing it this way?" If they wrote a blog post that way, that would likely have amazing takeaways.
I can assure you that was pretty close to the internal conversation lol
Not sure what the different takeaways would be though?
What did your internal discussion conclude for the question "Why did we not take a step back earlier and think, why are we doing it this way?"
Im genuinely curious because this is not singling out your team or org, this is a very common occurrence among modern engineering teams, and I've often found myself on the losing end of such arguments. So I am all ears to hear at least one such team telling what goes on in their mind when they make terrible architecture decisions and if they learned anything philosophical that would prevent a repeat.
Oh we had it coming for quite some time and knew we would need to rebuild it, we just didn’t have the capacity to do it unfortunately.
I was working on it on and off moving one endpoint at a time but it was very slow until we hired someone who was able to focus on it.
It didn’t feel good at all. We knew the product had massive flaws due to the latency but couldn’t address it quickly. Especially cause we he to build more workarounds as time went on. Workarounds we knew would be made redundant by the reimplementation.
I think we had that discussion if “wtf are we doing here” pretty early, but we didn’t act on it in the beginning, instead we tried different approaches to make it work within the serverless constraints cause that’s what we knew well.
I have had CTOs (two in my career) tell me we had to use our AWS credits since they were going to expire worthless. Both experiences were at vc-backed startups.
What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
Not everyone is born with experience in distributed systems
Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.”
I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics.
0: https://gist.github.com/hellerbarde/2843375
I cofounded it yeah
And yeah you’re right in hindsight it was a terrible idea to begin with
I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it
That's fair, but then the framing matters. The article criticizes serverless architecture rather than acknowledging an evaluation failure.
"Serverless was fighting us" vs "We didn't understand serverless tradeoffs" - one is a learning experience, the other is misdirected criticism.
Yeah that’s fair
You don't need experience and there is not really a lot to know about "distributed systems" in this case, that's basic CS knowledge about networks, latency and what "serverless" actually is, you can read about it. To be honest, to me it reads like people who don't understand the problem they're solving, haven't acquired the necessary knowledge to solve it (either by learning themselves or by asking/hiring people who have it), and seeing such an amateurish mistake doesn't inspire confidence for the future. You should either hire people that know what they are doing or upgrade your knowledge about systems you are using before making decisions to use them.
Sometimes I see a post about sorting algorithms online. Some people seem to benefit from reading about these things, but often, I find there isn't much new information for me. That's OK, because I know somebody somewhere benefits from knowing this.
It is your decision to make this a circlejerk of musings about how the company must be run by amateurs. Whatever crusade you're fighting in vividly criticising them is not valuable at all. People need to learn and share so we can all improve, stop distracting from that point.
[dead]
I would not assume this was a "rookie mistake". I've been here once or twice, and a common story is that engineers don't want to do it a certain way, but management overrules them for some vague hand-wavy reason like, "This way is more modern." Another common story is that you know you're not choosing the most [scalable|robust|performant|whatever] design, but ancillary constraints like time and money push you into a "worse is better" decision.
Or maybe the original implementation team really didn't know what they were doing. But I'd rather give them the benefit of the doubt. Either way, I appreciate them sharing these observations because sharing these kinds of stories is how we collectively get better as a professional community.
My personal experience is that if you want guaranteed anything (quick scaling, latency, CPU, disk or network throughput), your best bet is to manually provision EC2 instances (or use some API that does). Once you give up control hoping to gain performance for free, you usually end up with an unfixable bottleneck.
If you're looking for a middle ground between VMs and serverless, ECS Fargate is a good option. Because a container is always running, you won't experience any cold start times.
Yes, though unless you’re provisioning your own EC2s for them to run on, you have no guarantee about the server generation, and IME AWS tends to provision older stuff for Fargate.
This may or may not matter to you depending on your application’s needs, but there is a significant performance difference between, say, an m4 family (Haswell / Broadwell) and an m7i family (Sapphire Rapids) - literally a decade of hardware improvements. Memory performance in particular can be a huge hit for latency-sensitive applications.
ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.
Sounds useful! I hear mixed things about Swarm. You like it?
Edit: found it. Cool! https://rove.dev/
There isn't much for them to mess with in EKS either. It is very close to the metal and easy to reason about.
This is basically criticizing them for admitting to being one of today's 10,000.
https://xkcd.com/1053/
Personally, I appreciate the info and the admission.
> Putting a latency-critical API on a stateless edge runtime
Isn’t this the whole point of serverless edge?
It’s understood to be more complex, with more vendor lockin, and more expensive.
Trade off is that it’s better supported and faster by being on the edge.
Why would anyone bother to learn a proprietary platform for non critical, latency agnostic service?
You're confusing network proximity with application architecture. Edge deployment helps connection latency. Stateless runtime destroys it by forcing every cache access through the network.
The whole point of edge is NOT to make latency-critical APIs with heavy state requirements faster. It's to make stateless operations faster. Using it for the former is exactly the mismatch I'm describing.
Their 30ms+ cache reads vs sub-10ms target latency proves this. Edge proximity can't save you when your architecture adds 3x your latency budget per cache hit.
Realistically, they should be able to do sub ms cache hits which land in the same datacenter. I know cloudflare doesn't have "named" datacenters like other providers but at the end of the day, there are servers somewhere and if your lambda runs twice in the same one there is no reason why a pull-through cache can't experience a standard intra data-center latency hit.
I wonder if there is anything other than good engineering getting in the way of this and even sub us intra-process pull through caches for busy lambda functions. After all, if my lambda is getting called 1000X per second from the same point of presence, why wouldn't they keep the process in memory?
On serverless, whenever you call your code, it has to be executed but first the infrastructure has to find a place to run it and sometimes if there's no running instance available, it must fire up a new instance to run your code.
That's hot start VS cold start.
Agreed. Wondering what sort of discovery or design phase their legacy arch went thru.
But but it's webscale!
These two have resonated with me deeply.
- Eliminated complex caching workarounds and data pipeline overhead
- Simplified architecture from distributed system to straightforward application
We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions. Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever.
Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.
But when were serverless systems like lambda and cloud workers "best practices" for low latency apis?
According to their marketing material, when they started supporting running in edge pop's, they became the best option for low-latency APIs.
Last I heard (~5 years ago), lambda@edge doesn't actually run on edge POPs anyway; they're just hooks that you can put in your edge configs that execute logic in the nearest region before/after running your edge config. But it's definitely a datacenter round-trip to invoke them.
Adding that much compute to an edge POP is a big lift; even firecracker gets heavy at scale. And security risk for executing arbitrary code since these POPs don't have near the physical security of a datacenter, small scale makes more vulnerable to timing attacks, etc.
Their problem isn't serverless, rather Cloudflare Workers and WebAssembly.
All major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.
At that point, why should I use serverless at all? If I have to think about the lifetime of the servers running my serverless functions?
It can be good for connecting AWS stuff to AWS stuff. "On s3 update, sync change to dynamo" or something. But even then, now you've got a separate coding, testing, deployment, monitoring, alerting, debugging pipeline from your main codebase, so is it actually worth it?
But no, I'd not put any API services/entrypoints on a lambda, ever. Maybe you could manufacture a scenario where like the API gets hit by one huge spike at a random time once per year, and you need to handle the scale immediately, and so it's much cheaper to do lambda than make EC2 available year-round for the one random event. But even then, you'd have to ensure all the API's dependencies can also scale, in which case if one of those is a different API server, then you may as well just put this API onto that server, and if one of them is a database, then the EC2 instance probably isn't going to be a large percentage of the cost anyway.
Serverless only makes sense if the lifetime doesn't matter to your application, so if you find that you need to think about your lifetime then serverless is simply not the right technology for your use case.
Because it is still less management effort than taking full control of the whole infrastructure.
Usually a decision factor between more serverless, or more DevOps salaries.
I would doubt that this is categorically true. Serverless inherently makes the whole architecture more complex with more moving parts in most cases compared to classical web applications.
> Serverless inherently makes the whole architecture more complex with more moving parts
Why's that? Serverless is just the generic name for CGI-like technologies, and CGI is exactly how classical web application were typically deployed historically, until Rails became such a large beast that it was too slow to continue using CGI, and thus running your application as a server to work around that problem in Rails pushed it to become the norm across the industry — at least until serverless became cool again.
Making your application the server is what is more complex with more moving parts. CGI was so much simpler, albeit with the performance tradeoff.
Perhaps certain implementations make things needlessly complex, but it is not clear why you think serverless must fundamentally be that way.
Depends pretty much where those classical web applications are hosted, how big is the infrasture taking care of security, backups, scalability, failovers, and the amount of salaries being paid, including on-call bonus.
Serverless is not a panacea. And the alternative isn't always "multiple devops salaries" - unless the only two options you see are server serverless vs outrageously stupid complicated kubernetes cluster to host a website.
There's a huge gap between serverless and full infra management. Also, IMO, serverless still requires engineers just to manage that. Your concerns shift, but then you need platform experts.
A smaller team, and from business point of view others take care of SLAs, which matters in cost center budgets.
For a thing, which permanently has load it makes little sense.
It can make sense if you have very differing load, with few notable spikes or on an all in on managed services, where serverless things are event collectors from other services ("new file in object store" - trigger function to update some index)
Agree, it seems like they decided to use Cloudflare Workers and then fought them every step of the way instead of going back and evaluating if it actually fit the use case properly.
It reminds me of the companies that start building their application using a NoSQL database and then start building their own implementation of SQL on top of it.
Ironically, I really like cloudflare but actively dislike workers and avoid them when possible. R2/KV/D1 are all fantastic and being able to shard customer data via DOs is huge, but I find myself fighting workers when I use them for non-trivial cases. Now that Cloudflare has containers I'm pushing people that way.
Hey! Bet I can guess who
In that scenario, how do you keep cold startup as fast as possible?
The nice thing about JS workers is that they can start really fast from cold. If you have low or irregular load, but latency is important, Cloudflare Workers or equivalent is a great solution (as the article says towards the end).
If you really need a full-featured container with AOT compiled code, won't that almost certainly have a longer cold startup time? In that scenario, surely you're better off with a dedicated server to minimise latency (assuming you care about latency). But then you lose the ability to scale down to zero, which is the key advantage of serverless.
Apparently not nice enough, given that they rewrote the application in Go.
Serverless with containers is basically managed Kubernetes, where someone else has the headache to keep the whole infrastructure running.
Cloudflare has containers now too, and having used AppRunner and Cloud Run, it's much easier to work with. Once they get rid of the container caps and add more flexibility in terms of container resources, I would never go back to the big cloud containers, the price and ease of use of Cloudflare's containers just destroy them.
I doubt that the bill would be that much cheaper, nonetheless thanks for making me aware they are a thing now.
They're much cheaper, they're just DOs, and they get billed as such. They also have faster cold start times and automatic multi-region support.
What does DO mean in this context?
Durable Object
Indeed.
They get to the bottom of the post and drop:
> Fargate handles scaling for us without the serverless constraints
They dropped workers for containers.
You're saying serverless can have really low latency and fast 24/7?
Isn't serverless at the base the old model, of shared vms, except with a ton of people?
I'm old school I guess, baremetal for days...
Yes, check Cloud Run, AWS Lambda, Azure Functions with containers.
I think someone should make a timeline of software technology eras, each beginning with 'why XYZ is the future' and ending with articles like this.
It sounds like you picked the wrong platform because you didn’t understand what you were doing.
That’s not a technology issue.
"Self-Hosting : Being tied to Cloudflare's runtime meant our customers couldn't self-host Unkey. While the Workers runtime is technically open source, getting it running locally (even in dev mode) is incredibly difficult.
With standard Go servers, self-hosting becomes trivial:"
A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.
After building my first Serverless/Cloudflare worker app, this is why I migrated to Deno. Deno enables you to run the same codebase in deno (self-hosted/local) and in deno deploy (serverless platform from deno).
I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo).
Here is a link to my reasoning from back then: https://github.com/K0IN/Notify/pull/77#issuecomment-16776070...
I ported my worker project into Django since cloudflare workers wouldn’t allow selection of region for hosting workers which is generally required due to data compliances. This is something all cloud providers provide from day one yet cloudflare made it an enterprise feature.
Also the vendor lock-in doesn’t help with durable objects and D2 instead of simply doing what supabase and others are doing by providing Postgres or standard SQLite as a service.
Linux servers running Go apps? Would be nice to see server cost and specs, backup strategy, etc.
What do you find so peculiar about it? A lot of people are running Go apps on VPSs.
TFA states that they’re running on AWS Fargate.
That said, as an example, an m8g.8xlarge gives you 32 vCPU / 128 GiB RAM for about $1000/month in us-east-1 for current on-demand pricing, and that drops to just under $700 if you can do a 1-year RI. I’m guessing this application isn’t super memory-heavy, so you could save even more by switching to the c-family: same vCPU, half the RAM.
Stick two of those behind a load balancer, and you have more compute than a lot of places actually need.
Or, if you have anything resembling PMF, spend $10K or so on a few used servers and put them into some good colo providers. They’ll do hardware replacement for you (for a fee).
Next article - why we switched from our own servers to serverless for reliability. A small performance hit was worth it.
They just use two servers and configure a loadbalancer within Cloudflare. Come on. Self-Hosting is no rocket science. You don‘t have to make it seem complicated. People have been doing this decades before AWS invented serverless.
Yet, idiots remain.
Backup strategy? What do you mean by that?
Servers go down. What is the plan to get them "backup" and running ;)
They probably don't need one for the application servers. And they probably already have a backup strategy for their DBs.
Most server outages are caused by hardware failures which EC2 MOSTLY abstracts from you.
Also, if it's just Golang, point Ansible or whatever deploys at new server and trigger a deploy.
But is this not needed with the so-called cloud systems?
I'm assuming "High Availability" is what is really meant here.
Incredible that these kinds of services were hosted like this.
I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering.
Why didnt they go with a self hosted backend right away?
Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.
We did initially but thought cloud flare was a better solution for scalability and latency.
We believed their docs/marketing without doing extensive benchmarks, which is on us.
The appeal was also to use the same typescript stack across everything, which was nice to work with
Where did their marketing or documentation say this service is perfect for low latency APIs?
I doubt they literally said “perfect for low latency APIs” but their messaging is definitely trying to convince you that they’re fast globally, just look at the workers.ckoudflare.com page
Author of that blog here, happy to answer any questions :)
Have you done new benchmarks since Cloudflare announced their latest round of performance improvements for Workers?
Just curious if this workload also saw some of the same improvements (on a quick read it seems like you could have been hitting the routing problem CF mentions)
Really great writeup. The charts tell the story beautifully, and the latency gains are surely a win for your company and customers. I always wonder about the tradeoffs. Is there a measurable latency difference for your non-colocated customers? What does maintenance look like for your Go servers? I assume that your Cloudflare costs dropped?
It’s faster for non-colocated customers too weirdly
I think cause connections can be reused more often. Cloud flare workers are really prone to doing a lot of TLS handshakes cause they spin up new ones constantly
Right now were just hang aws far hate for the go servers, so there really isn’t much maintenance at all. We’ll be moving that into eks soon though cause we are starting to add more stuff and need k8s anyways
Not a question: thanks for the writeup and for the honesty of saying that serverless is not inherently bad, just not the right fit for your usecase!
Unfortunately too many comments here are quick to come to the wrong conclusion, based only on the title. Not a reason to change it though!
Thanks
It’s totally fair criticism that the title and wording is a bit clickbaity
But that’s ok
Do you have a clearer picture of what use-cases you would use serverless functions for in the future (if any)?
We all love a vendor lock-in, don’t we? Until it backstabs us and we go back to VMs
Interesting writeup. The serverless approach helped with GTM. (I speculate) raising capital afforded them extra devs who noticed the cache latency.
> The serverless approach helped with GTM
Unlikely? They could've just as well deployed their single go binary to a vm from day 1 and it would've been smooth sailing for their use case, while they acquire customers.
The cloudflare workers they chose aren't really suited for latency critical, high throughput APIs they were designing.
30ms P99 does not a cache make.
Source work somewhere where you easily get 1ms cached relational DB reads from outside the service.
30ms makes me suspect it went cross region.
I think this is what is being said:
"Down with serverless! Long live serverless!"
If they use Cloudflare that automatically disqualifies them from me reading whatever they wrote, or caring about it.
Somewhere in Denmark, DHH is smiling
I often don't know what to make of DHH. He's a living contradiction. On one hand he will continually rant about how bad the overhead and waste of cloud services is, and on the other hand he will staunchly defend the most inefficient programming language that is regularly used for backend development, as well as defend the enourmous overfetching that active record leads to.
Really I think DHH just likes to tell others what he likes.
Gives him a break from writing out of touch screeds about countries he knows nothing about I guess.