iPhone iOS 9 Redirect Caching Bug?
Recently, while I was investigating reports from a customer of 404 errors from our site, I found what I believe is an iOS 9.0 – 9.0.2 bug that I haven’t seen any reports of elsewhere. Given this, I thought I’d share a bit of a write-up about what I found.
But first, a little bit of background.
We have some bespoke software responsible for serving landing pages, known as “page server”. To allow the use of consistent relative URLs for various things, page server responds to requests for URLs like “http://domain.com/somepage” with a 301 redirect to “http://domain.com/somepage/”. Note the trailing slash. That’ll become very important in a moment.
Also of relevance is how we track clicks. Our customers want to know whether visitors to their pages are clicking on links or not, so we track this by having the links link to special URLs handled by page server, and then redirecting to the real destination URL. These links take the form of They look something like “http://domain.com/somepage/clkn/http/someotherrdomain/someotherpage”, but are in relative form when they’re on the page. This means the page source actually has “clkn/http/someotherdomain/someotherpage” in it.
We heard from the customer that a number of their landing page visitors were receiving a 404 when they clicked on certain links on their landing pages. The links in question were our “click tracking“ links. The real oddity in this report though was that this was only happening on the 2nd time a visitor visited the page. A visitor could come to the page once, click the link, and it worked fine. Upon returning to a page, however, clicking the link would sometimes result in a 404 error.
Diagnosing the Bug
At the time of the report, our customer had already narrowed down that this was happening to customers using iPhones. Unsurprisingly, in a tech company office, there are a lot of iPhones.
But after testing on various iPhones around our office, we were only able to find a single iPhone that was able to reproduce the problem. By sheer coincidence, this happened to be my iPhone. I was able to reproduce the bug on a number of landing pages, not just the one the bug was originally reported on, but only with my Phone. My phone has been bugging me for days to update to iOS 9.1, but I’ve been putting it off. My immediate thought was that maybe I was running into a bug in iOS. Before I spent much time on that train of thought, I made sure I could reproduce the bug with some reliability.
The process for reproducing it was:
Navigate to testdomain.com/testpage, and click on the link:
[27/Oct/2015:17:29:34 +0000] testdomain.com"GET /testpage HTTP/1.1" 301 [27/Oct/2015:17:29:34 +0000] testdomain.com"GET /testpage/ HTTP/1.1" 200 [27/Oct/2015:17:29:39 +0000] testdomain.com"GET /test3/clkg/http/othertestdomain.com/images/test.jpg HTTP/1.1" 301
So far, those access logs show exactly what I would expect to see if everything’s working properly.
Close the tab, then open up another one and navigate to lp.lewisd.com/testpage (after backspacing the automatically added trailing slash), then click on the link again:
[27/Oct/2015:17:29:48 +0000] testdomain.com"GET /testpage/ HTTP/1.1" 200 [27/Oct/2015:17:29:51 +0000] testdomain.com"GET /clkg/http/wwwothertestdomain.com/images/test.jpg HTTP/1.1" 404
Well! That’s not at all what I expected. Despite having typed “/testpage”, the browser made a request to “/testpage/”. What’s even more insidious is that the URL in the browser bar still shows “/testpage”, and this is what’s used to resolve relative links. So instead of resolving to “/testpage/clkn/…” it resolves to “/clkn/…”. This is clearly not right.
Reloading the page appears to always send the request to the correct url, which then sends a 301, as expected:
[27/Oct/2015:17:30:02 +0000] testdomain.com"GET /testpage HTTP/1.1" 301 [27/Oct/2015:17:30:02 +0000] testdomain.com"GET /testpage/ HTTP/1.1" 200
This seems fairly clearly to be a browser bug.
Assessing the Impact
The most obvious difference between my phone and all the others we tested on was that I hadn’t yet upgraded to iOS 9.1. I was still on 9.0.2. Given that this was only happening on iPhones, and apparently only on certain OS versions, the initial theory was that it was an iOS bug.
To further narrow this down, I checked all the access logs for the customer page, to see the breakdown of browsers that were getting 404 errors. The results were pretty clear: only iOS 9.0 and 9.0.2 were getting 404s. Before jumping the gun and making assumptions, I wanted to see if, perhaps, those were perhaps the most popular (by a long shot) browsers that were visiting this customer’s page. It turned out that iOS 9.0.2 was the most popular, but iOS 9.1 wasn’t far behind, and was followed by iOS 9.0. iOS 9.0.1 wasn’t even make the list. Given this distribution, one would expect iOS 9.1 to be showing up in the 404’s, but it wasn’t. This further reinforced the theory that it was a bug in iOS 9.0 – 9.0.2, which was fixed in 9.1.
I was able to look at our logs for accesses to all pages over the last 30 days to get an idea of how many users might be encountering this issue, based on their iOS version (of which there are a lot). I’ve excluded anything older than 8.0 from this graph, since they’re all lost in the noise of old versions at the bottom.
I also looked at https://en.wikipedia.org/wiki/IOS_version_history to determine what date each new version was released. We can see from the graph that adoption climbs quite rapidly after each new release, and that use of the older versions drops similarly, but then falls into a long tail of people who either choose not to upgrade, or are unable to for various reasons. Based on this, we can expect iOS 9.0 – 9.02 usage to be lost amongst the noise of all the older versions within about a week, and with it, the bug we’ve seen.
There were a few take-home’s for me, out of this graph, which I found absolutely fascinating:
iOS updates seem to reach saturation after about 3 weeks. After that point, the rate of adoption slows to a crawl.
Old iOS versions hang around forever. Omitted from the graph are all the old versions of iOS that we see in our access logs. They go all the way back to 3.2!
Let’s look at some stats from the last 24 hours:
55.2% of our iPhone/iPad traffic is on iOS 9.x, but a total of 15.9% of the iPhone/iPad is on versions older than 8. For reference, the last iOS 7 release was on June 30, 2014, 15 months ago. By contrast, the first iOS9 release was on September 16, 2015, only 42 days ago.
Of the visitors that are on iOS 9.x, the majority of them (60.3%) are on the latest version, 9.1. This works out to 32% of the total iOS traffic. As shown in the very first graph (the line graph), the other 9.x versions are on their way down, and 9.1 is on the way up.
iOS 8 visitors are more all over the map. There’s no majority, and actually the largest segment (though not by much) are on an older version.
The iOS 7 visitor breakdown looks like a cross between the 8.x and 9.x charts.
Overall, it looks like somewhere close to 15% of traffic is on versions of iOS that are at least 15 months old, but the majority is on a version that was released at most 42 days ago.
Senior Software Developer