Make a New Year’s Resolution to start backing up your data regularly. Not just local files, but even data from the cloud. Here’s how to backup your data from a few of the most common online services. More importantly, I’ve also included instructions on how to restore from that backup.

Twitter

How to backup?

Copy this URL to a new browser window.

http://twitter.com/statuses/user_timeline/USERNAME.xml?count=10000

Then replace the string USERNAME with your actual Twitter username. Press enter to start downloading. If your browser does not prompt you with a file download box, but instead opens the file showing a bunch of text, choose File > Save As to save your backup to a secure location.

How to restore?

You cannot restore this data into Twitter (neither to your own account, nor to a different account.) But you will have access to your witticisms and interesting web links that you posted to amuse your friends. Do with it as you please.

Gmail Messages

How to backup?

Use an IMAP client such as Mail.app on the Mac, or Thunderbird on any platform. Make sure it’s configured to download and cache every email and every attachment.

How to restore?

You can access your messages from these programs even if Gmail is down. If you need to transfer messages to another account, add that new account in the same program as a new IMAP account, then drag-and-drop messages from your old account to your new account to transfer them there.

Gmail Contacts

How to backup?

Login to your Gmail / Google Apps email account, then open the Contact Manager. Click on the Export button in the top-right corner. For maximum compatibility with other applications, choose the third option for data format, vCard format. (It’s a standard format for contact information exchange.)

How to restore?

The vCard format is fairly standard. Gmail itself can read back the same file without trouble. To import into Mac OS X Address Book, simple double-click the .vcf file and let the import proceed. Microsoft Outlook also supports importing addresses from vCard files.

Google Calendar

How to backup?

Google Calendar publishes feeds of your calendar in the iCal format. If you save this feed to a file, you can use it as a backup. On the left side of your main calendar, there is a list labeled “My Calendars”. For each calendar that you want to backup, click on the little downward-pointing arrow next to the calendar name, and select “Calendar Settings”.

On the Settings page, under Calendar Details, locate the section labeled “Private Address”. Click on the button labeled ICAL and copy the URL there. Open a new browser window and paste the URL there. This will start downloading a file; save it to a safe location — this is your calendar backup. Lather, rinse, repeat for each calendar you want to backup.

How to restore?

The iCalendar format (also abbreviated as iCal or .ics) is a standard calendar format. You can import the backed up calendar file into Google Calendar, Apple iCal or Microsoft Outlook simply by opening it.

Google Reader

How to backup?

Login to Google Reader, then come back here and click on this link: Export Google Reader subscriptions as OPML. Save the file that your browser will prompt you to download. This is your backup.

How to restore?

Google Reader and lots of other feed readers know how to import OPML files. In case of Google Reader, go to Settings > Import/Export to import it back. For desktop software, try looking for an “Import from OPML” menu item somewhere.

Need instructions for more services? Write a comment and I’ll try to provide them.

This is a quick snippet I put together for an academic project. To be able to write this, I had to go through several documentation resources, for what is essentially a single web service method call. I figured it would help if I shared my PHP code.

<?php
/**
 * Query Amazon about a particular book by ISBN and obtain metadata.
 * The author disclaims all copyright and places this in the public domain.
 *
 * Amazon's Terms of Use for this service require you to:
 * - Send no more than 1 request every second
 * - Direct traffic to them in some way. You can use the URL provided in the
 *   resulting metadata to achieve this.
 */
class ISBN {
  function getMetadataFromIsbn($isbn) {
    // Get your own accesskey at http://aws.amazon.com/
    $awsAccessKeyID = 'YOUR_ACCESS_KEY_ID_HERE';
    $awsSecretKey = 'YOUR_SECRET_KEY_HERE';
    $awsAssociateTag = 'YOUR_ASSOCIATE_TAG_HERE';

    $host = 'ecs.amazonaws.com';
    $path = '/onca/xml';

    $args = array(
      'AssociateTag' => $awsAssociateTag,
      'AWSAccessKeyId' => $awsAccessKeyID,
      'IdType' => 'ISBN',
      'ItemId' => $isbn,
      'Operation' => 'ItemLookup',
      'ResponseGroup' => 'Medium',
      'SearchIndex' => 'Books',
      'Service' => 'AWSECommerceService',
      'Timestamp' => gmdate('Y-m-d\TH:i:s\Z'),
      'Version'=> '2009-01-06'
    );

    ksort($args);
    $parts = array();
    foreach(array_keys($args) as $key) {
      $parts[] = $key . "=" . $args[$key];
    }

    // Construct the string to sign
    $stringToSign = "GET\n" . $host . "\n" . $path . "\n" . implode("&", $parts);
    $stringToSign = str_replace('+', '%20', $stringToSign);
    $stringToSign = str_replace(':', '%3A', $stringToSign);
    $stringToSign = str_replace(';', urlencode(';'), $stringToSign);

    // Sign the request
    $signature = hash_hmac("sha256", $stringToSign, $awsSecretKey, TRUE);

    // Base64 encode the signature and make it URL safe
    $signature = base64_encode($signature);
    $signature = str_replace('+', '%2B', $signature);
    $signature = str_replace('=', '%3D', $signature);

    // Construct the URL
    $url = 'http://' . $host . $path . '?' . implode("&", $parts) . "&Signature=" . $signature;
    $rawData = file_get_contents($url);

    $metadata = simplexml_load_string($rawData);
    if (isset($metadata->Items->Request->Errors)) {
      return $metadata->Items->Request->Errors;
    } else {
      return $metadata->Items->Item;
    }
  }
}
?>

Setting up WebDAV on Leopard

The good news is that all the bits and pieces of software that you need to run a WebDAV server on Mac OS X 10.5 Leopard are already installed. You only need to configure them correctly and turn them on. Some experience with Terminal is preferred, and you should be familiar with executing UNIX commands. Let’s start!

1. Start Apache. (If you haven’t already) You will need to enable Web Sharing, since the WebDAV service will be provided by Apache, the web server on Mac OS X. You do not necessarily need to have a web site running, but you will need to activate and run Apache. Go to System Preferences > Sharing, and turn on the box labeled Web Sharing.
   
2. Enable WebDAV support in Apache. Edit the file /etc/apache2/httpd.conf, (remember to use sudo to edit it) and locate this line:

   LoadModule dav_module libexec/apache2/mod_dav.so

   Make sure it is not commented (there should be no "#" at the beginning of the line.) Then locate this line (towards the bottom of the file):

   Include /private/etc/apache2/extra/httpd-dav.conf

   Again, make sure it is not commented out. It is disabled by default, so you need to remove the "#" from this line.

3. Configure WebDAV. Next, edit the file /etc/apache2/extra/httpd-dav.conf. Add a section in it to create our new WebDAV share. Here’s what the new section should look like. As a security precaution, you should also go ahead and delete the /usr/uploads share that is set by default.

   Alias /webdav "/Library/WebServer/WebDAV"
   
   <Directory "/Library/WebServer/WebDAV">
     Dav On
   
     Order Allow,Deny
     Allow from all
   
     AuthType Basic
     AuthName WebDAV-Realm
     AuthUserFile "/usr/webdav.passwd"
   
     <LimitExcept GET OPTIONS>
       require user YourUserName
     </LimitExcept>
   </Directory>
   

   On line 1, the name following the Alias keyword is the URL you’d like for your new WebDAV share. If you want the share to be located at http://your-server-name/your-fancy-webdav-share, then line 1 should read:

   Alias /your-fancy-webdav-share "/Library/WebServer/WebDAV"

   On line 9, we specify the authentication scheme as Basic, not Digest. The security conscious will note that this sends unencrypted passwords over plain text. In my tests, OmniFocus was not able to communicate with the server with the Digest authentication scheme. Remember not to use a particularly important password for this account.

   On line 14, substitute the username you would like to use for your WebDAV account. Note this down, because you will need this again in the next step.

4. Create user accounts and passwords. Use the htpasswd tool to create your password file.

   sudo htpasswd -c /usr/webdav.passwd "YourUserName"
   New password:
   Re-type new password:
   Adding password for user YourUserName

5. Create the necessary directories.

   sudo mkdir -p /Library/WebServer/WebDAV
   sudo mkdir -p /usr/var

6. Setup permissions correctly.

   sudo chown -R www:www /Library/WebServer/WebDAV
   sudo chown -R www:www /usr/var
   sudo chgrp www /usr/webdav.passwd

7. Restart Apache gracefully.

   sudo apachectl graceful

8. Test your server. Optionally, you can test your WebDAV configuration using litmus, a WebDAV server test tool. It is distributed as source code with no binaries, so you will need to compile it first, for which you will need Apple’s Developer Tools. You can test your server manually by using a graphical client such as Goliath. Try uploading a file and see if you can access it again.

That’s it, you can now point OmniFocus to http://your-server-name/webdav and provide the credentials you created earlier. With this setup, you will immediately be able to access your WebDAV server over your local network. If your machine has a static public IP address, you will also be able to sync from outside your local network.

If, on the other hand, your machine is behind a router, you will need to configure port forwarding on your router. If you do not have a static IP, you will need to set a dynamic hostname via services like DynDNS.

Possible Error Messages

This is by no means a zero error configuration, and sometimes things might go wrong. Here are some of the common error messages and how to fix the relevant errors:

• The locks could not be queried for verification against a possible "If:" header.
No such file or directory: Could not open property database
  The web server is not able to access the password file. In our example, you need to ensure that /usr/webdav.passwd can be read by the Apache user, www. To do that, run the following chmod command.

  sudo chgrp www /usr/webdav.passwd

• Client used wrong authentication scheme: /webdav/
  You probably set the Authentication scheme to Digest instead of Basic. Try changing to Basic. Also note that you need to regenerate the password file using htpasswd instead of htdigest.

OmniFocus and WebDAV

To use WebDAV with OmniFocus, simply configure it as in the screenshots below.

A hat tip to Vivek for helping test these instructions on a clean Leopard installation.

After spending about an hour configuring what should, in theory, be a simple matter, I figured I’d write a blog post that might one day save another soul an hour or so from his or her life. So, for good karma, basically. In the past, I have set up port forwarding on Linux, Mac OS X and Windows, so I was a little worried that it took me about an hour trying to appease the SSH deities (and daemons).

The command itself is just a single line; the devil is in the parameters. I’m splitting the command over several lines and adding line numbers to illustrate the details and separate the parts of the long-ish command for easier explanation. Feel free to type it all on a single line (after removing the line numbers and the line-break markers ("\") of course!)

1.     ssh \
2.       -L local_port:service_host:service_port \
3.       -p ssh_server_port \
4.       -l ssh_server_username \
5.       -N \
6.       ssh_server_host

Parameters

Now for the various parameters used in the command above. Some of them may be omitted if the defaults are used, but I have included all of them in the example above to cover the most general case.

local_port

The port on your local machine that your local program expects to be able to connect to. If this is one of the reserved ports (i.e., under 1023), you will have to run your ssh tunnel command as root (using sudo). Ports above 1024 are freely available for any user to listen on.

service_host

The fully-qualified domain name or the IP address of the server that is hosting the service that you wish to connect to. For example, if this is a web site, it could be google.com or yahoo.com. It does not have to be under your control, nor does it have to be the machine that you’re SSHing into. It is just any host on the Internet that you can access from ssh_server_host. Often this is a server you are not allowed to access from your own machine, e.g. a chat server or IRC server. Or you may wish to hide the fact from the administrator of your local network that you are connecting to this server (e.g. when you’re out at a coffee shop on a sniffable insecure wireless network, or in a country with laws forbidding access to free information.)

Important: If you’re trying to access a service running on the same machine as ssh_server_host, remember to use 127.0.0.1, not localhost. What’s the difference, you say? Well, since IPv6 is here to stay, localhost can map to either 127.0.0.1 (IPv4) or ::1/128 (IPv6). If your applications aren’t all IPv6-compliant, this can cause some headache. Hopefully, we will all be on IPv6 in the near future, but till then, this is a way to make things work. If you’re trying to use IPv6, you need to use local_port/service_host/service_port (slashes instead of colons.)

service_port

The port number on which the desired service is running. Here are some common port numbers:

Service	Port
Web: HTTP	80
Web over SSL: HTTPS	443
Outgoing email: SMTP	25
Incoming email: POP3	110
Incoming email: IMAP	143
VNC	5900
iTunes Music Sharing	3689

ssh_server_host

The machine that you’re SSHing into. This is the one that is running sshd, the SSH daemon.

ssh_server_port

The port number on which the SSH daemon is listening on ssh_server_host. This is most likely 22; you should only use a different value if your sysadmin has told you that the SSH server is running on another port (or if you’re a sysadmin yourself and you set up your SSH server to run on a non-standard port for security through obscurity.)

ssh_server_username

The username you would use to connect to ssh_server_host in a regular SSH session. This may or may not be the same as the username you currently use on your local machine.

The Entire Command, Line by Line

1. Line 1 simply calls the ssh program;
2. Line 2 sets up the port forwarding. The -L parameter specifies that this is a remote-to-local tunnel. If you wanted to create a local-to-remote tunnel, you’d have used -R instead of -L. The next three parameters are from our list above, separated by colons. (Use slashes instead of colons for IPv6.) If you want to set up multiple tunnels from the same host, simply repeat line 2 as many times as you’d like, once for each set of local_port:service_host:service_port.
3. Line 3 selects a port on the ssh_server_host to connect to. Omit this line if you’re connecting to the default port 22.
4. Line 4 specifies the username to use on the ssh_server_host. It is also possible to use the ssh_server_username@ssh_server_host syntax instead of the -l parameter.
5. Line 5 indicates to ssh that no commands be run on the remote system. Since you’re using this SSH connection simply for tunneling, this is a useful option to set.
6. Line 6 contains the most basic parameter of this entire process. Please don’t get this wrong.

Common Errors and Solutions

If you’ve not been able to debug this so far, try passing the -v parameter to ssh to see verbose output. Add another -v for more verbose output.

If you’re reading this, and come across any specific source of trouble, please let me know so I can add it to this mini HOWTO.