The Making of SSDR
In Internet Services (link), besides reading papers, we also had a final group project (GP) to develop a new idea in the field of internet services, or evaluate an existing one. I worked with my good friend and roommate Adolfo and we ended up producing SSDR, a dependency-inlining reverse proxy (available on Github), which improves page loads for users with high last-mile latencies (like 3G mobile users) by reducing the number of round-trip-times (RTTs).
This actually wasn't directly related to any of the papers we read in class, but was instead inspired by the "A cloud-based content gathering network" post from Adrian Colyer's The Morning Paper blog. Fun fact: if the proposal deadline had been a bit later, we might have ended up doing a sillier project like a DNS-based filesystem.
Working on the SSDR implementation and paper was a cool experience. I read a small ream's worth of papers, wrote some questionable code with cool tools like mitmproxy, learned a lot about how browser networking works (High Performance Browser Networking by Ilya Grigorik was incredibly helpful), and ended up in some pretty deep rabbit holes a couple times.
Rabbit hole #1: reinventing server push
At one point, Adolfo and I were talking about caching resources. Normally websites share a good number of resources from page to page: common CSS, maybe a Javascript library like jQuery, a header image, and so on. When you inline these resources, you lose the source of the resource; if a user's browser receives something like
<img src="data:image/png;base64 iVBORw0KGgo..."/>So we ended up drafting out this system where the user would also run a special user-side SSDR-aware client proxy. The SSDR proxy would then send dependencies with an extra identifier for caching. Now we can push data and the client can cache, but there's no way to make us not push if the client knows they have a cached copy. This is important: if we're always push everything, then we can't take advantage of caching.
So maybe the SSDR proxy can keep track of who's received what, and expect those cache hits. But what if the client clears their cache? Then they'll end up with a broken site. Even if it works, the SSDR proxy needs to maintain a lot of state, which becomes harder when it serves more clients.
To move this complexity to the client-side, we could use a mechanism like HTTP conditional GET. When the user makes a request, the client-side proxy can send a cookie or a header with a list of all the identifiers it has, and then the SSDR proxy can selectively in-line. We send a
<dependency id="dependency-id">dependency-id so that it can be cached.
Somewhere around here, we stepped back and said, "Wow, it's basically just HTTP/2 Server Push, but worse". Which was fun, since it's pretty validating to see that you are trying to address the right use cases (caching + the ability to decline pushes), but also a real timesink as the semester winds down and project deadlines loom.
Working with HTTP/2 really made it clear how complicated web services are these days. I also took 18-741 Computer Networks this semester and around the same time wrote a little HTTP/1.1 video streaming server in C. In comparison to these HTTP/2 servers, that was completely simple, even trivial. HTTP/1.1 is all text and pretty straightforward request-response exchanges (even with keepalives). HTTP/2 on the other hand is binary frames with potentially interleaved requests and responses, prioritization, compressed headers - and this is even before you get to server push. I'm very impressed that these things work at all, let alone work enough that I barely notice them as a user - there's an incredible amount of work that I had no idea about.
Rabbit hole #2: browser behavior
If HTTP/2 and TLS are complicated, then browsers must be even more complicated, since they handle all the network stuff, and also need to render the resulting pages.
I found some weird browser behavior over the course of this project. At first I was going to use Firefox (which is my usual browser), and found that Firefox does not resolve iframes nested more than 10 deep. So fine, I'll switch to Chrome, only to find that Chrome doesn't apply network throttling correctly for nested iframes, i.e., it doesn't apply it at all (no bug report link here, I'll file one eventually).
Sure, so it's back to Firefox, except it seems like HTTP/2 server push with my server isn't working. But it seems to work with other public servers that are using server push? Of course, it doesn't help that Firefox does not show what resources are server pushed, so I'm just here squinting at the waterfall chart to see if something loads too... slowly? It's pretty clear that server push is working because dependencies are loading before the original page, but if they load after, it doesn't really mean anything.
Eventually I tried to curl the request directly, and of course curl needs to be recompiled to handle HTTP/2 since everything is binary frames instead of plaintext, and I don't want to deal with that right now so I switch to nghttp2 and find that... the server push does work?
Wait no, actually, if I make the same request that Firefox does (with all the headers), the server push is never sent.
Binary search of the headers reveals that it's the Host header at fault: I can make two otherwise-identical requests, and the one without the Host header will receive push promises, and the other will not!
It's well past midnight at this point, but it seems like we have progress.
Debug logs from Hypercorn reveal that it's receiving two host pseudo-headers.
So I try using a Firefox extension to remove the Host header from Firefox's requests... which doesn't work.
(In hindsight, I don't think I inspected the stream to see that the header was actually removed... something to look into).
After a long time trying the same thing and hoping for different results (a sure sign of sanity), I try monkeypatching Quart to just delete the extra host pseudoheader.
I'm pretty sure I did this correctly, but turns out this also doesn't fix the issue, and I give up for the night to go to bed at 5 in the morning because I'm too tired to focus, and then in the morning with the clarity of rest I give up completely because I realize there's way more important things to be done (like, writing the actual paper), and so this remains a mystery to this day.
Maybe I'll take a look again soon - just writing out that sequence of events reminds me of how close I was.
Do I even care about the impact of the bug at this point? Honestly, I don't really care. I just want closure.
As a side note, there's also a bunch of throttling proxy tools like tc(8), Toxiproxy, Comcast, throttle-proxy, etc. In hindsight it probably would have been much easier to take the time to set one of these up instead of messing with the in-browser tools.
Besides that, there's also wacky rendering behavior: did you know that if you use the srcdoc attribute of an iframe, the resulting content is fixed to be like, less than 200 pixels tall?
Even if the iframe itself takes up the entire height of your browser.
Why does this happen? No one knows, of course.
Or maybe someone knows and I just don't know how to Google for it.
At least it's more-or-less consistent across Firefox and Chrome.
You can fix this by setting height: 100% as an inline style on the html and the body elements in the srcdoc, but who knows what else that will break.
The SSDR proxy probably breaks rendering in other ways, but "luckily", our testing was very limited and we didn't check, so I can truthfully say this is the only case with broken rendering that I know of.
After the deadline passed, Adolfo noticed that really, our particular evaluation method (using nested iframes) is biased in our favor: a nested iframe takes longer to render than, say, an image referenced from a CSS file. So when we inline iframes using srcdoc, we get even better performance than just reducing the number of RTTs - we also get a boost in rendering time. I'm not sure what causes this browser behavior, besides "an iframe is more complex", which isn't really an answer. This might be related to how loading an iframe might cause delays in the rendering of other items on the page - WProf has some examples of how these interact, and is notable because it's really the only place I've seen try to document this at all (since the browsers themselves don't).
Things I would do differently
The usual thing to say here is to start earlier, put in more time, be smarter. This is all true: the paper summary covers some limitations where it's basically just a matter of putting in more time: fixing same-origin behavior, using a headless browser, handling malformed HTML, etc. But I think these at best are small incremental improvements. They make the "product" slightly more usable, but they don't bring any additional insight. At the end of the day, if I was going to put in more time and effort into this class, amidst our final semester and in quarantine, I would want more to show for it.
I believe the best potential project changes fall into the following categories: real-world comparisons, more tooling, and asking for expert help.
More real-world comparisons
Every evaluation we did was pretty contrived and was often chosen because it was the most convenient option. So there's lots of room for improvement here.
Gathering this real-world data would have given us much stronger validation on if SSDR would actually be useful in the real world. It would produce lasting contributions that go beyond our fairly trivial SSDR implementation and could serve as a baseline for others to build upon.
Dependency graphs
A website where /5 nests 5 iframes and /10 nests 10?
That was incredibly easy to implement and made it really easy to generate graphs.
But I'd be very surprised if any real website actually had 10 nested same-origin iframes.
So, what does a real dependency graph look like?
- Out of a set of real websites like the Alexa Top 500, what does the distribution of max dependency depths look like?
- Are these graphs very deep, or wide and shallow?
- Are the dependencies same-origin, or from a third-party site?
- What are these composed of? Mostly images, or mostly iframes?
Network conditions
We simulated network latency using the in-browser Network Throttling features in Firefox and Chrome. But what do real-world latencies look like? We actually originally planned to evaluate this by snapshotting our DigitalOcean droplet and moving it to different locations, but we were thwarted by a lack of capacity at the time (May 2020), possibly due to COVID19-related scaling.
Drop-in usability
We claimed that SSDR was easy to drop-in and set up, but we didn't evaluate this. One interesting evaluation would be to create an HTTP/1.1 webserver, and watch webmasters try to set up our SSDR proxy versus upgrading it to HTTP/2 and configuring server push. "Usability" claims are easy to make, but they can only be confirmed with user testing.
More tooling
The current version of the repo has a nice Makefile that installs everything for you, with convenient commands to even run the servers. This was a relatively late addition; earlier in the project I would just run things manually, which was a little slower and more error-prone. In the end I had to set it up anyways to deploy the project in places besides my own computer, so it would have been better to just put in the time up-front.
All the timing evaluation was done by hand, with a human (Adolfo) painstakingly setting up the GUI through Chrome or Firefox DevTools and manually recording times. This could have been done using a Selenium script or similar instead. Although we might have saved time overall by not taking the time to figure out the script, it significantly reduced our flexibility. Basically, once we started testing, the implementation became "frozen": we couldn't make any significant changes to it because it could affect the test results, and then we would have to re-run these tests by hand. If the testing was script-driven, we could make changes whenever we wanted to and just re-run the script to get new numbers.
Asking for more expert help
Finally, we could have asked for more help. We did this project fairly independently using the resources we could gather online - as mentioned earlier, this topic was not directly related to anything we read in class. Looking back on our references list, there's a few sets of names which show up again and again in the topics of remote dependency resolution, mobile accelerators, and so on. I think if we had reached out to these researchers early on, they could have helped point us towards useful literature, warn about common pitfalls, and offer insight into other problems we had overlooked.
In hindsight, our project essentially asks the question "Does unconditionally inlining dependencies improve load times?" and the answer, of course, is yes, with penalties to caching and time-to-first-paint, which confirms existing knowledge. With more context about the field, earlier in the semester, we could have reworked our project to try and answer more interesting questions like when to conditionally push dependencies, or how to optimize for browser rendering behavior. Even though state-of-the-art systems would be out of reach considering our time constraints, I think we could have made meaningful progress for a subproblem. But, much like this year, hindsight is 20/20.
Other random comments
mitmproxy is super cool, and writing addons is really powerful, but wow, the addon documentation leaves a bit to be desired: "this is not an API manual, and the mitmproxy source code remains the canonical reference" is a pretty rough introduction. I wrote the SSDR add-on by copy-pasting an example add-on and then fiddling with it until it did the thing I wanted it to, which was sort of fun from a hacking perspective but also a frustrating experience. Lots of room to improve here, and maybe if I do more mitmproxy scripting I'll take a crack at it.
Quart, the Python web framework we used, has a ridiculous problem. When you Google for Quart help, it helpfully also searches for the abbreviation of quart... which gives a lot of results for Qt, the cross-platform GUI framework (not to be mistaken for you, the reader). I couldn't even get Google to stop doing this even with quotation marks, though at least DuckDuckGo let me tell it what I wanted. On one hand, this is pretty cool from a natural language search perspective. But on the other hand, sometimes I really just want to search for the things that I actually typed.
Trying to pair write the entire paper the day that it's due is not the most fun experience, so, uh, don't do that.
If you have to be stay-at-homed (for you future readers in a hundred-year plague), try to do group projects with your roommates. There were so many times that Adolfo and I had quick exchanges in the kitchen that resulted in progress, which probably wouldn't have happened at all if we were completely remote and needed to call or message to talk. The ability to do that casually, plus the ability to poke heads into each other's rooms to ask a quick question, was really valuable. Being friends helps too.
Thanks to Andy Tsai for feedback on this post.