Or, On Becoming a Cloudflare Fanboy.
What’s wrong with GitHub Pages?
I’d been using GitHub Pages for hosting static sites, including this one, for years. It’s easy and free-ish (I believe they’re free for public repos, and if you’re paying $4/mo, private repos as well).
One thing you don’t get, though, is any kind of visitor metrics. I know I can get this if I add something like Google Analytics to my sites, but I’m loath to do that for a number of reasons. For one, to be GDPR compliant, I’d have to add one of those annoying cookie banners to all my sites. Also, ad blockers can block the GA Javascript and skew results. Finally, I don’t want to have to add the GA code to every site.
If I were hosting the sites myself, I could get the information I need from the web server logs, but if GitHub is hosting the content, they have that info—but they aren’t sharing it.
Enter Cloudflare Pages
I’d been using Cloudflare for a while, as early on it was the only way to enable SSL on your GitHub Pages sites (GH now provides SSL for all Pages). But I didn’t know they had their own “Pages” offering.
Turns out it’s super easy to use: just point it to your GitHub repo, tell it what branch you want to deploy from and if there’s any build commands you want to run (plus other optional settings like output directory, environment variables,etc.), and it will publish your site on a pages.dev subdomain. Then you can add your own domain name via a CNAME record to that subdomain. If Cloudflare is hosting your DNS it will even update your records for you. And everything gets SSL.
(Another advantage of Cloudflare’s Pages offering is I can easily set up a dev/staging site by just creating another Pages site and pointing it to a branch in my repo.)
The analytics it gives you is decent: total requests, unique visitors, bandwidth, % cached, requests by country/region, etc. You can even see breakdown by source IP, browsers, OS, desktop vs mobile (although oddly it’s under the Security section, not Analytics).
One thing that seems to be missing is a breakdown of traffic by page. It would be nice to see, for example, if particular blog posts are getting more traffic than others, but I’m not seeing that data anywhere. Update: They do have this, in a separate section (they really need a good UX person to clean up their menus), and it has to be enabled per site. If you are using their DNS proxy, it will automatically inject their Javascript into your site. Otherwise you can manually insert their JS snippet. They don’t use cookies, and it doesn’t look like they are exposing IP addresses anywhere, so no GDPR concerns.
Going All-In
After moving a few of my sites over and being generally impressed with the Cloudflare experience, I decided to go all-in and consolidate all my domains (more than 20…yes, I have a domain problem) under Cloudflare. This meant having them be my registrar, DNS, and, where appropriate, hosting the content. Previously my domains were spread out among a few different registrars, and my DNS split between registrars, Digital Ocean, and Cloudflare. Having everything under one roof made a lot of sense.
The process was pretty straightforward. The only real snag I hit is that by default Cloudflare sets up “Flexible” SSL/TLS, where traffic is encrypted between visitors and Cloudflare, and then connections to your backend are made over HTTP. But if you already have a cert on your backend, and that backend is configured to redirect HTTP traffic to HTTPS, you’ll end up with a redirect loop. The fix is to just go into Cloudflare and change the SSL/TLS mode to “Full”.
So What Does This Cost?
Nothing. It’s all free…which has been bugging me a bit. Other than registrar fees when transferring domains to them, I haven’t paid them anything. They do have paid plans, of course, but the additional features offered—mostly around security—are targeted towards much bigger sites (bot detection, image optimization, cache analytics, web application firewall, etc.). I would actually be happy to pay them something, but I haven’t hit anything yet to warrant it. However, it seems like the paid plans are all per domain, so if I did end up needing to pay for something across all my sites, it would get unreasonably expensive (their cheapest paid plan, “Pro,” is $20/mo/domain).
Fingers Crossed
I do feel a little nervous about going all-in on Cloudflare, as I would going all-in with any company/service, but I’ve been really impressed with them so far. They’ve had some controversies in the past, but generally seem like a pretty good company, both technically and ethically. I’ll certainly keep you posted if anything changes.
It’s been a bit over a year since I was laid off from Etsy, where I’d been for 11½ years. Although everyone will tell you not to take a layoff personally, it still stings knowing that someone decided you were not worth keeping around. That said, the layoffs seemed to hit old-timers and remotes particularly hard, so the deck was stacked against me.
It wasn’t hard for me to decide not to go looking for another full-time job. I’d long fantasized about being an indie software developer, working on my own things and setting my own schedule, so I guess that’s what I was now.
I started an LLC for all the amazing apps I was going to start writing, but then some consulting fell into my lap. (Coincidentally, this same thing happened to me the last time I was laid off, nearly 25 years ago. I had been working in San Francisco during the dot-com boom and when the company went under I was all set to move to Bishop and become a climbing bum for a year, when a friend pinged me and I ended up spending the next four years consulting.)
I spent a couple months with Wikimedia Foundation creating data modeling guidelines. I learned that WMF has some really interesting and unique challenges, both technically and politically. It’s a really great group of folks.
That was followed a by a much bigger consulting opportunity, which lead to starting another LLC with some friends. We spend about six months tackling a pretty hairy data engineering challenge. What I learned from that is if you see NetSuite, run.
Around the paid gigs I also spent some time helping out one friend with their startup, another with his app, and wrote a web app to help another friend give away free bikes to kids.
I’ve been keeping myself pretty busy since that last gig ended. I spent a good amount of time doing a deep dive into MacOS aerial screensavers/wallpapers. I wrote a Safari extension to redact news articles. I wrote a little web game. I also spent a good amount of December doing Advent of Code, working on my Swift and SwiftUI skills in the process.
I’ll write separate posts about all of those projects, but I’m currently mulling over what to do next. I recently got a 3D printer, so playing with that and learning a bit of CAD has been taking a lot of my time, but that’s not nearly as fulfilling as a good project is.
So after a year of being “indie,” I’m still pretty sure I’m never going back to a full-time jobby-job. I do miss my colleagues at Etsy, and there’s something to be said about being part of a team and a larger mission, but I do not miss all the overhead that comes with working at a “large” company (“work about work”). I love waking up every day knowing that I have zero meetings and what I want to accomplish that day is entirely up to me. Literally whatever.
Now to just come up with that killer app idea….
My friends Sunah Suh, Anthony Hersey, and I finally figured out the magic incantations needed to automate sending “STOP” to political text messages. Setting it up is completely non-obvious, so I’ll walk through it step-by-step.
Note that this is for iOS only. If someone with an Android phone wants to write up their process, I’d be happy to link to it.
Also, this was done on iOS 17.4. Surely this process will change some day. If you notice something is off, let me know and I’ll update it.
First, open the Shortcuts app and tap the Automation tab at the bottom.
Tap the plus (+) in the top right corner.
Start typing "Message" in the search bar, and select Message.
On the When page, hit Choose next to Message Contains.
Type in the text to look for, such as "STOP to end". It appears to be case-insensitive.
Hit Done and back on the When page, select Run Immediately. Then tap Next in the top-right corner.
Now select New Blank Automation.
Under Next Action Suggestions, you should see Send Message. (If not, hit Add Action and search for "Send Message".) Tap Send Message.
Tap on the Message field and type "STOP".
Now comes the first tricky part. Hold down (long press) on Recipients and select Shortcut Input. (If instead it pops open a "To:" page, you didn't hold down long enough. Cancel and try again.)
Now the next tricky bit. Tap the Shortcut Input field and select Sender, then hit Done.
That's it! Now repeat this for different variations of "STOP to...." ("STOP to quit", "STOP2END", etc.)
One of the engineers on my team at Etsy, who is working on getting our clickstream events into protobuf, noticed that we were getting some values that were too large for protobuf’s largest integer, uint64. E.g: 55340232221128660000.
This value was coming from our frontend performance metrics-logging code, and is supposed to represent the sum of the encodedBodySizes of all the JavaScript or CSS assets on the page. But given that 55340232221128660000 is around 50,000 petabytes, it’s probably not right. There had to be a bug somewhere.
We were only seeing this on a small percentage of visitors to the site—less than 0.1%. I looked at the web browsers logging these huge values and couldn’t see any obvious pattern.
My first thought was that we had code somewhere that was concatenating numbers as strings. In JavaScript, if you start with a string and add numbers to it, you get concatenated numbers. E.g.:
> "" + 123 + 456
< '123456'
But I didn’t see anyway that could happen in our code.
Looking at the different values we were getting, I saw a lot of similar-but-not-quite equal values. For example:
18446744073709552000
18446744073709556000
18446744073709654000
36893488147419220000
36893488147419310000
55340232221128660000
55340232221128750000
55340232221128770000
73786976294838210000
92233720368547760000
The fact that they all end in zeros was curious, but I realized that was because they were overflowing JavaScript’s max safe integer.
I started googling some of the numbers and got a hit. I thought it was very odd that one of our seemingly random numbers was showing up on a StackOverflow post. Then I googled “18446744073709551616”. Any guesses what that is?
264
36893488147419220000? Approximately 2 * 264
55340232221128660000 =~ 3 * 264
etc.
I realized what was probably happening was when we were adding up all the resources on the page, some of them were returning 264-1 (the maximum 64-bit unsigned integer), but some were normal, so that explained the variation.
Now encodedBodySize is an unsigned long long, so my first thought was there is a browser bug that was casting a -1 into the unsigned long long, which would give us 264-1.
I started digging through browser source code. I found that in WebKit, the default value for responseBodyBytesReceived is 264-1, although that value should be caught and 0 returned instead.
Then I went spelunking in the Chromium source, and found this commit, from less than two weeks ago. (Although the bug was reported back in May 2022.) Sure enough, it was a default value of -1 getting cast to an unsigned value. Boom! Mystery solved.
I pushed a fix to our JavaScript to catch these bogus values and hopefully our protos will live happily ever after. Gone are my dreams of finding and fixing a major browser bug, but I’m happy that I can move on with my life.
The Framework for ML Governance by Kyle Gallatin
ML is becoming commoditized, but there hasn’t been much in the way of scalable frameworks to support the delivery and operation of models in production (of course, Algorithmia has one, hence their sponsoring of this report).
ML needs to follow the same governance standards for software & data as other more traditional software engineering does in your organization. In the development phase, this includes validation and reproducibility of your model, and documentation of your methods and rationale. The delivery and operational phases are much more complex, including things like observability, auditing, cost visibility, versioning, alerting, model cataloging, security, compliance, and much more.
MLOps is the set of best practices and tools that allow you to deliver ML at scale. ML governance is how you manage those practices and tools, democratizing ML across your organization through nonfunctional requirements like observability, auditability, and security. As ML companies mature, neither of these are “features” or “nice-to-haves”—they are hard requirements that are critical to an ML strategy.
Data scientists can’t do this all by themselves. They need support from the larger organization. The value of ML governance needs to be understood at the highest levels. They should involve infrastructure, security, and other domain experts early on, and be sure to be open in their communication with the rest of the company.