TTL and propagation

June 26, 2007

Every DNS record is equipped with a TTL. The TTL (Time To Live) is basically the expiration date on that record. Long story short, it's a countdown from when it was initially received until when it is marked as invalid and discarded for a replacement record. This is a very important piece of information that I've run into often as being either outright ignored or misunderstood.

Let's say you have a domain-something awesome like awesomedomain.wow--and awesomedomain.wow has a TTL of 24 hours. When I go to visit awesomedomain.wow as a new visitor (and you know I would, because it sounds awesome) I'm going to receive a record translating awesomedomain.wow to an IP address that will be valid for 24 hours. Any other time I visit that domain in the next 24 hours, I'm going to use that cached address because the record hasn't expired yet. In 24 hours regardless of if awesomedomain.wow has moved IPs, I'm going to trash that old DNS record I've cached locally and go look it up again. The new record will then be referred to by me for the next 24 hours, at which time I'll do it all over again.

But what happens when you have to change your IP, but you want your visitors to see the smallest amount of downtime possible? My first suggestion is to mirror your sites on both IPs, but that is a different discussion entirely. The second is to manipulate your TTL. First lower it to something smaller-from a day to an hour perhaps. Then give that new record with that new TTL at least 24 hours to propagate. Now you can be certain that at the 25th hour, all of your visitors now have a record that will expire in one hour. Next, change your IP for awesomedomain.wow, the record that your visitors have cached locally will expire in an hour, and then they will have your new record with your new IP. Feel free to bump your TTL back up to what it was originally in this step, since they have the new IP. Now your visitors have only had an old record for an hour rather than 24, and they probably missed that hour it was inaccessible while they were posing for a painting or having their top hats heightened. Because all of your visitors are terribly classy.

-Joshua

Comments

June 28th, 2007 at 3:46am

Ah yes, one of the fun things to do when moving servers. Being around small sites so long I've come familiar with "We're moving servers! If you see this page, please wait a few days to clear or go to newserver.site.wow". The few times I move servers I try to minimize the use of pages like that.

With mirroring the site, the problem is always databases and forums. Either you can use purely static pages for a bit on the old server, leave the database as is and risk some posts and updates not getting transferred, or have the old server use the new server for database for a little while. [Or even have the old server act as a proxy to the new server, and many other options like database syncing/linking] Obviously the most desirable option would be having the old server connect to the new server's database.

The problem with that becomes encryption. I'm sure you don't want all those database queries sent over the Internet in plain text. You have a few solutions to that however: Set your SQL server up to use SSL, use a SSL tunnel, or use secure VPN. For VPN I've used Hamachi, though I don't know how well Linux support is anymore. For SSL tunneling, you can use Stunnel. Both are free [Hamachi has a "Premium" version however]. If you want to set your server to use SSL, good luck with that.

Syncing files between servers can also be useful, or you can disable user uploads on the old server, if possible. I'm not sure what you can use for that though. If anyone has some ideas, please post and let us all know.

One major issue with DNS which you didn't mention. Some ISP's DNS servers ignore the TTL and sometimes cache it up to a week. While rare, it can affect some customers. It is rare though, so it's not a -really big- issue.

Note: If you have old server use new server's database, the queries may take a while to go through, so page generation time may increase a bit.

June 29th, 2007 at 9:08am

We do a lot of migration work. A common problem we see is that ISPs often have internal caching DNS servers that do not honor your TTL. We've found this to be the most problematic in Europe.

Another key problem is search engines. We have seen it take as long as a week for a search engine to pick up your DNS change. Google used to take even longer but now they are pretty fast. Anytime you migrate a site, we advise leaving the old site in place, even if you must shutdown things like forums. You can then monitor the logs on the new and old location to assure all search bots are hitting your new site.

We have seen pages get dropped from search engines from time to time because of internal DNS caching.

June 29th, 2007 at 6:35pm

Agreed, and that is a very good point. By far the best way to migrate a site is to leave both IPs live during propagation and monitor your logs. However, I've run into many, many instances where (typically due to finance issues) a company simply can't. If you are moving toward a strict cut over date in a situation such as that, this is the way to go.

And for all those admins that aren't expiring records like they should--stop it. ;)

November 20th, 2007 at 9:31am

For the least possible amount of downtime, setting the TTL extremely low is a great first step, but here's a near-instant way of getting a new server operational with little user impact. (And it makes an allowance for DNS admins that ignore the TTL).

1. Lower the TTL on all DNS involved to an extremely low number.
2. Register a temporary secondary domain (i.e. www2), point it at the new server.
3. Set up the new server as if it were an exact copy. Test, test, test.
4. Make the switch. Flip on the new server, change IPs in the DNS.
5. Pull a final db copy from the old server to the new one for forums etc, shut down SQL on the old server to prevent orphans.
6. On the old server, enable a 307 redirect to the www2 URL.

This way, while DNS propagates around the net, your users are getting redirected to the new server. 307 is Google-safe, and it also instructs web browsers to re-post any form data to the new URL. In a few days, traffic to the old server will taper off to nothing and you can take it offline.

The only db-driven content downtime is the amount of time it takes to transfer the db and reattach it.

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Comments

June 28th, 2007 at 3:46am

Ah yes, one of the fun things to do when moving servers. Being around small sites so long I've come familiar with "We're moving servers! If you see this page, please wait a few days to clear or go to newserver.site.wow". The few times I move servers I try to minimize the use of pages like that.

With mirroring the site, the problem is always databases and forums. Either you can use purely static pages for a bit on the old server, leave the database as is and risk some posts and updates not getting transferred, or have the old server use the new server for database for a little while. [Or even have the old server act as a proxy to the new server, and many other options like database syncing/linking] Obviously the most desirable option would be having the old server connect to the new server's database.

The problem with that becomes encryption. I'm sure you don't want all those database queries sent over the Internet in plain text. You have a few solutions to that however: Set your SQL server up to use SSL, use a SSL tunnel, or use secure VPN. For VPN I've used Hamachi, though I don't know how well Linux support is anymore. For SSL tunneling, you can use Stunnel. Both are free [Hamachi has a "Premium" version however]. If you want to set your server to use SSL, good luck with that.

Syncing files between servers can also be useful, or you can disable user uploads on the old server, if possible. I'm not sure what you can use for that though. If anyone has some ideas, please post and let us all know.

One major issue with DNS which you didn't mention. Some ISP's DNS servers ignore the TTL and sometimes cache it up to a week. While rare, it can affect some customers. It is rare though, so it's not a -really big- issue.

Note: If you have old server use new server's database, the queries may take a while to go through, so page generation time may increase a bit.

June 29th, 2007 at 9:08am

We do a lot of migration work. A common problem we see is that ISPs often have internal caching DNS servers that do not honor your TTL. We've found this to be the most problematic in Europe.

Another key problem is search engines. We have seen it take as long as a week for a search engine to pick up your DNS change. Google used to take even longer but now they are pretty fast. Anytime you migrate a site, we advise leaving the old site in place, even if you must shutdown things like forums. You can then monitor the logs on the new and old location to assure all search bots are hitting your new site.

We have seen pages get dropped from search engines from time to time because of internal DNS caching.

June 29th, 2007 at 6:35pm

Agreed, and that is a very good point. By far the best way to migrate a site is to leave both IPs live during propagation and monitor your logs. However, I've run into many, many instances where (typically due to finance issues) a company simply can't. If you are moving toward a strict cut over date in a situation such as that, this is the way to go.

And for all those admins that aren't expiring records like they should--stop it. ;)

November 20th, 2007 at 9:31am

For the least possible amount of downtime, setting the TTL extremely low is a great first step, but here's a near-instant way of getting a new server operational with little user impact. (And it makes an allowance for DNS admins that ignore the TTL).

1. Lower the TTL on all DNS involved to an extremely low number.
2. Register a temporary secondary domain (i.e. www2), point it at the new server.
3. Set up the new server as if it were an exact copy. Test, test, test.
4. Make the switch. Flip on the new server, change IPs in the DNS.
5. Pull a final db copy from the old server to the new one for forums etc, shut down SQL on the old server to prevent orphans.
6. On the old server, enable a 307 redirect to the www2 URL.

This way, while DNS propagates around the net, your users are getting redirected to the new server. 307 is Google-safe, and it also instructs web browsers to re-post any form data to the new URL. In a few days, traffic to the old server will taper off to nothing and you can take it offline.

The only db-driven content downtime is the amount of time it takes to transfer the db and reattach it.

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.