I do what most responsible cloud platform users do—I check my billing reports regularly to make sure nothing unexpected is happening. It’s become a monthly ritual: export the billing data, scan for unusual spikes, maybe dig into a few line items that look different from last month. Most months, everything looks reasonable. Last March, it didn’t. There was a charge labeled “Network Internet Data Transfer Out from Sydney to Australia” that had grown from a few dollars to over $300. The SKU belonged to Compute Engine, which confused me immediately because most of my infrastructure runs on Cloud Run, Google’s serverless platform. I do have one legacy service running on Kubernetes, but that’s been stable for months with no changes to how much data it moves around.

Here’s what made the charge so baffling: my web service is small. We serve about 8,000 users per day, handle maybe 25 requests per second at peak times, and run lean by design. The billing report claimed we were generating 220GB of outbound traffic every single day. To put that in perspective, that’s roughly 27 megabytes of data for every user, every day. If we were actually serving that much data, I’d know about it—our application doesn’t work that way. We serve mostly text and small images, not video streams or large file downloads. Something was fundamentally wrong with either the billing data or my understanding of what my infrastructure was doing, and I had no clear way to figure out which.

The first thing I did was check the metrics for all my services. Cloud Run has decent monitoring built in, and you can see how much network traffic each service sends and receives. The numbers were nowhere close to 220GB per day. Same with the Kubernetes cluster—the GKE metrics showed normal traffic patterns that matched what I expected based on our user load. I spent an evening cross-referencing every metric I could find, looking for something that would add up to 220GB. Nothing came close. The gap between what the billing report claimed and what the monitoring tools showed was so large that it felt like we were talking about two different systems. That’s when I realized I needed help from someone who understood GCP’s billing infrastructure better than I did.

My assumption was that the billing support team would be able to explain this quickly. Billing issues happen, metrics get miscategorized, maybe there’s a known issue with how certain types of traffic get reported. But the first few conversations with billing support didn’t go anywhere useful. They looked at the same data I was looking at, confirmed that yes, the charges were real and correctly reported, but couldn’t tell me where the traffic was actually coming from. The SKU pointed to Compute Engine, so they suggested I open a technical support case with that team. Fair enough—I figured the Compute Engine folks would have access to deeper metrics that could trace the traffic back to its source.

The Compute Engine support team was thorough, I’ll give them that. We spent over a week examining VM instances, looking at GKE node traffic, checking network flow logs, trying to correlate the billing timestamps with actual usage patterns. Their working theory was that one of my VM instances or GKE nodes was somehow generating this traffic without my knowledge—maybe a background process, maybe some kind of data replication I’d forgotten about. But every time we dug into a specific hypothesis, the numbers didn’t line up. The total egress from all my VMs and GKE pods combined was maybe 30-40GB per day, and that matched what I expected. We were missing 180GB per day, and nobody could explain where it was coming from.

At this point, two weeks had passed, and I was starting to question my own understanding of how GCP works. Maybe there’s some hidden layer of traffic I don’t know about? Maybe the metrics are only showing application-level traffic and there’s infrastructure traffic that doesn’t appear in the standard dashboards? The Compute Engine team started looking at my Cloud Run services, thinking maybe the traffic was coming from there but being billed under Compute Engine for some reason. They asked me to open a separate support case with the Cloud Run team, which I did, and that team came back saying their metrics showed everything was normal—no unusual egress, nothing that would explain the charges. Now I had three support teams looking at this, and none of them could find the source of the traffic.

The next phase involved video calls where we’d all stare at the same dashboards together, each team insisting their part of the infrastructure was working correctly. I started to feel like I was stuck in some kind of support loop, being passed from one team to another with nobody taking ownership of the problem as a whole. The billing team said it’s a technical issue, the Compute Engine team said it might be Cloud Run, the Cloud Run team said it’s definitely not them. Meanwhile, I’m still getting charged $300+ every month for traffic that, as far as I can tell, doesn’t exist. The frustration wasn’t just about the money—it was about the fact that I couldn’t trust my own monitoring. If 220GB per day could be happening without showing up in any of the standard metrics, what else was I not seeing?

One of the support engineers eventually suggested something that should have been obvious from the start: export the detailed billing data to BigQuery. GCP lets you set up billing export so every single charge gets recorded as a row in a BigQuery table, with much more granular detail than what shows up in the Cloud Console billing interface. I’d known this feature existed but had never bothered to set it up because the standard billing interface seemed sufficient. That was a mistake. As soon as I had the data in BigQuery, I could write SQL queries to slice it however I wanted—group by service, by SKU, by region, by time of day, by any of the dozens of dimensions that GCP tracks.

It took me maybe thirty minutes to find the culprit once I had the data in the right format. The traffic wasn’t coming from Cloud Run, wasn’t coming from my VMs, wasn’t coming from Kubernetes. It was coming from a Cloud Load Balancer that I’d set up months earlier to serve static assets from a Cloud Storage bucket. And here’s the thing that made this so hard to diagnose: the load balancer traffic doesn’t show up in the Compute Engine metrics, doesn’t show up in Cloud Run metrics, doesn’t even show up clearly in the networking dashboards unless you know exactly where to look. It’s billed under Compute Engine because load balancers are part of the Compute Engine service, but the monitoring for load balancer egress is in a completely different place from where you’d monitor VM or container egress.

Once I knew where to look, the whole picture became clear. The load balancer was serving assets that should have been cached but weren’t, so every request was pulling data from the Storage bucket and sending it out to users. The bucket was in Sydney, the users were in Australia, and even though that’s technically in-region traffic, it still gets billed as egress. The reason it had spiked recently was because I’d made a change to the cache headers on those assets without realizing it would prevent caching. So instead of users downloading each asset once and reusing it, they were downloading it on every page load. A simple configuration mistake, but one that was completely invisible in the monitoring tools I was using.

The relief of finally understanding what was happening was enormous, but it was mixed with frustration about how long it had taken to get there. Two months of back-and-forth with support, hours spent on video calls examining the wrong metrics, all because the standard monitoring tools don’t give you a clear view of load balancer egress. And the only reason I found it was because I happened to export the billing data to BigQuery and could query it directly. That shouldn’t be necessary. The tools should make this kind of debugging straightforward, not something that requires data export and SQL queries.

What bothers me most about this experience isn’t the $300 per month or even the wasted time. It’s that the architecture of cloud monitoring makes certain problems invisible until you dig into the raw billing data. As a CTO, I’m supposed to have visibility into what my infrastructure is doing, but that visibility is only as good as the tools provide. And in this case, the standard tools—the ones that most people use, the ones that seem comprehensive—simply didn’t show the traffic that was costing me hundreds of dollars a month. That’s a design flaw, not a monitoring gap. If you’re billing me for egress, I should be able to see that egress in my metrics without having to export to BigQuery and write custom queries.

The immediate fix was simple enough: correct the cache headers, verify the traffic dropped, watch the billing charges return to normal. But the broader lesson is that cloud billing is fundamentally different from cloud monitoring, and the two don’t always align the way you’d expect. Billing is comprehensive because cloud providers track everything that costs money. Monitoring is selective because showing everything would be overwhelming. The gap between those two realities is where problems like mine hide. And the only way to bridge that gap is to take the billing data—which is accurate but hard to interpret—and analyze it directly until you find what doesn’t match your expectations.

If you run infrastructure on any cloud platform and you see billing charges that don’t make sense, don’t spend weeks going back and forth with support teams looking at standard metrics. Export your billing data to wherever you can query it properly, and dig into it yourself. The support teams are helpful, but they’re constrained by the same monitoring tools you are. The billing data is the source of truth, not because it explains what’s happening, but because it records everything that costs money. Your job is to translate those charges back into actual infrastructure behavior, which means you need the raw data and the ability to slice it however makes sense for your specific situation.

This whole episode reinforced something I’ve been thinking about a lot as both a CTO and a parent trying to manage a reasonable budget: complexity has a cost that goes beyond the money you spend on infrastructure. Every additional service, every load balancer, every layer of abstraction you add makes your system harder to understand and harder to debug when things go wrong. Sometimes that complexity is necessary, but often it’s just accumulated over time because it seemed like a good idea at the moment. The load balancer I’d set up was solving a real problem—serving static assets efficiently—but I’d never thought about how it would show up in billing or monitoring. That oversight cost me two months of investigation and over $600 in unnecessary charges. Not catastrophic, but completely avoidable with better visibility and better tooling.