HOWTO Obtain metadata for a book given its ISBN using Amazon Web Services in PHP

17 Jul, 2008 — HOWTO, Release

This is a quick snippet I put together for an academic project. To be able to write this, I had to go through several documentation resources, for what is essentially a single web service method call. I figured it would help if I shared my PHP code.

<?php
/**
 * Query Amazon about a particular book by ISBN and obtain metadata.
 * The author disclaims all copyright and places this in the public domain.
 *
 * Amazon's Terms of Use for this service require you to:
 * - Send no more than 1 request every second
 * - Direct traffic to them in some way. You can use the URL provided in the
 *   resulting metadata to achieve this.
 */
class ISBN {
  function getMetadataFromIsbn($isbn) {
    // Get your own accesskey at http://aws.amazon.com/
    $awsAccessKeyID = 'YOUR_ACCESS_KEY_ID_HERE';
    $awsSecretKey = 'YOUR_SECRET_KEY_HERE';
    $awsAssociateTag = 'YOUR_ASSOCIATE_TAG_HERE';

    $host = 'ecs.amazonaws.com';
    $path = '/onca/xml';

    $args = array(
      'AssociateTag' => $awsAssociateTag,
      'AWSAccessKeyId' => $awsAccessKeyID,
      'IdType' => 'ISBN',
      'ItemId' => $isbn,
      'Operation' => 'ItemLookup',
      'ResponseGroup' => 'Medium',
      'SearchIndex' => 'Books',
      'Service' => 'AWSECommerceService',
      'Timestamp' => gmdate('Y-m-d\TH:i:s\Z'),
      'Version'=> '2009-01-06'
    );

    ksort($args);
    $parts = array();
    foreach(array_keys($args) as $key) {
      $parts[] = $key . "=" . $args[$key];
    }

    // Construct the string to sign
    $stringToSign = "GET\n" . $host . "\n" . $path . "\n" . implode("&", $parts);
    $stringToSign = str_replace('+', '%20', $stringToSign);
    $stringToSign = str_replace(':', '%3A', $stringToSign);
    $stringToSign = str_replace(';', urlencode(';'), $stringToSign);

    // Sign the request
    $signature = hash_hmac("sha256", $stringToSign, $awsSecretKey, TRUE);

    // Base64 encode the signature and make it URL safe
    $signature = base64_encode($signature);
    $signature = str_replace('+', '%2B', $signature);
    $signature = str_replace('=', '%3D', $signature);

    // Construct the URL
    $url = 'http://' . $host . $path . '?' . implode("&", $parts) . "&Signature=" . $signature;
    $rawData = file_get_contents($url);

    $metadata = simplexml_load_string($rawData);
    if (isset($metadata->Items->Request->Errors)) {
      return $metadata->Items->Request->Errors;
    } else {
      return $metadata->Items->Item;
    }
  }
}
?>

HOWTO Setup WebDAV on Mac OS X

10 Jul, 2008 — Apple, HOWTO

Setting up WebDAV on Leopard

The good news is that all the bits and pieces of software that you need to run a WebDAV server on Mac OS X 10.5 Leopard are already installed. You only need to configure them correctly and turn them on. Some experience with Terminal is preferred, and you should be familiar with executing UNIX commands. Let’s start!

  1. Start Apache. (If you haven’t already) You will need to enable Web Sharing, since the WebDAV service will be provided by Apache, the web server on Mac OS X. You do not necessarily need to have a web site running, but you will need to activate and run Apache. Go to System Preferences > Sharing, and turn on the box labeled Web Sharing.
    Mac OS X Preferences Screenshot -- Enabling Web Sharing
  2. Enable WebDAV support in Apache. Edit the file /etc/apache2/httpd.conf, (remember to use sudo to edit it) and locate this line:
    LoadModule dav_module libexec/apache2/mod_dav.so

    Make sure it is not commented (there should be no "#" at the beginning of the line.) Then locate this line (towards the bottom of the file):

    Include /private/etc/apache2/extra/httpd-dav.conf

    Again, make sure it is not commented out. It is disabled by default, so you need to remove the "#" from this line.

  3. Configure WebDAV. Next, edit the file /etc/apache2/extra/httpd-dav.conf. Add a section in it to create our new WebDAV share. Here’s what the new section should look like. As a security precaution, you should also go ahead and delete the /usr/uploads share that is set by default.
    Alias /webdav "/Library/WebServer/WebDAV"
    
    <Directory "/Library/WebServer/WebDAV">
      Dav On
    
      Order Allow,Deny
      Allow from all
    
      AuthType Basic
      AuthName WebDAV-Realm
      AuthUserFile "/usr/webdav.passwd"
    
      <LimitExcept GET OPTIONS>
        require user YourUserName
      </LimitExcept>
    </Directory>
    

    On line 1, the name following the Alias keyword is the URL you’d like for your new WebDAV share. If you want the share to be located at http://your-server-name/your-fancy-webdav-share, then line 1 should read:

    Alias /your-fancy-webdav-share "/Library/WebServer/WebDAV"

    On line 9, we specify the authentication scheme as Basic, not Digest. The security conscious will note that this sends unencrypted passwords over plain text. In my tests, OmniFocus was not able to communicate with the server with the Digest authentication scheme. Remember not to use a particularly important password for this account.

    On line 14, substitute the username you would like to use for your WebDAV account. Note this down, because you will need this again in the next step.

  4. Create user accounts and passwords. Use the htpasswd tool to create your password file.
    sudo htpasswd -c /usr/webdav.passwd "YourUserName"
    New password:
    Re-type new password:
    Adding password for user YourUserName
  5. Create the necessary directories.
    sudo mkdir -p /Library/WebServer/WebDAV
    sudo mkdir -p /usr/var
  6. Setup permissions correctly.
    sudo chown -R www:www /Library/WebServer/WebDAV
    sudo chown -R www:www /usr/var
    sudo chgrp www /usr/webdav.passwd
  7. Restart Apache gracefully.
    sudo apachectl graceful
  8. Test your server. Optionally, you can test your WebDAV configuration using litmus, a WebDAV server test tool. It is distributed as source code with no binaries, so you will need to compile it first, for which you will need Apple’s Developer Tools. You can test your server manually by using a graphical client such as Goliath. Try uploading a file and see if you can access it again.

That’s it, you can now point OmniFocus to http://your-server-name/webdav and provide the credentials you created earlier. With this setup, you will immediately be able to access your WebDAV server over your local network. If your machine has a static public IP address, you will also be able to sync from outside your local network.

If, on the other hand, your machine is behind a router, you will need to configure port forwarding on your router. If you do not have a static IP, you will need to set a dynamic hostname via services like DynDNS.

Possible Error Messages

This is by no means a zero error configuration, and sometimes things might go wrong. Here are some of the common error messages and how to fix the relevant errors:

  • The locks could not be queried for verification against a possible "If:" header.
    No such file or directory: Could not open property database
    The web server is not able to access the password file. In our example, you need to ensure that /usr/webdav.passwd can be read by the Apache user, www. To do that, run the following chmod command.

    sudo chgrp www /usr/webdav.passwd
  • Client used wrong authentication scheme: /webdav/
    You probably set the Authentication scheme to Digest instead of Basic. Try changing to Basic. Also note that you need to regenerate the password file using htpasswd instead of htdigest.

OmniFocus and WebDAV

To use WebDAV with OmniFocus, simply configure it as in the screenshots below.

Omnifocus Sync Settings

OmniFocus WebDAV Authentication

A hat tip to Vivek for helping test these instructions on a clean Leopard installation.

Free as in freedom, not as in beer

I received a request today in the mail about one of my projects that is available under a free software license. It’s a web template system that I wrote to scratch a personal itch. Its hallmark feature is that it has no features, at least none that contribute to the bloat that’s rampant in Drupal and Joomla and their ilk.

His email was very well-written, asking about some of the specifics of the license, and how he could undertake projects for his clients building upon my framework. After I wrote a detailed reply to him, it seemed like a good idea to post it to my blog, for there are many who’re not particularly clear on how free software licensing works.

I want to use your templating system to build static websites for personal and commercial projects. I don’t have lots of money so I can’t promise much now but later if I will be able to make any money I will happily donate for this project.
[...]
I like open source projects because it is fun to learn how magic happens. I don’t want to use your code without permission because I just personally don’t think it is right thing to do. I have no problem giving you credit for this system but I need your permission to use it for commercial use?

Sincerely,
[redacted]

And my reply:

Hi [redacted],

I’m glad you found the site and my projects interesting and useful, and thanks so much for writing back to let me know!

I think free software is a great way to learn and understand other people’s code, that’s why almost all of my projects are open-source with the license to tinker and play. All I ask in return (via the Creative Commons license) is attribution back to me if you use it in a project. I’ve licensed this as an Attribution-ShareAlike-Noncommercial license, so you’re free to use it as you wish in any personal project as long as it is non-commercial: e.g. for an organization you belong to, or an academic department or program.

Things get a little more interesting when money enters the picture. While I’m not doing this (releasing my software) for the express purpose of making money from it, it does not seem right to me that someone else benefit financially from my work with no benefit to me. So, I politely ask that if you’re planning to use this commercially, you should contact me for a separate license (the code will then be dual-licensed, and you can pick either the paid commercial license, or the default un-paid non-commercial one.)

You don’t have to pay anything right away, and can play with the code as much as you want. But when you bag a client who wants to use a system based on my code, we can talk about royalties. That way, you retain the freedom to examine and modify my code as well as get a paying client, and I do not feel that someone has taken undue advantage of my generosity. This is how the open-source model was intended to work, and the free really refers to freedom, not free as in no-charge.

I’m glad you contacted me to check for permission first, and I got the opportunity to clarify. Often it’s quite tricky, and lots of people have lots of misunderstandings about how free software licenses work.

Regards,
Manas.

SSH Port Forwarding on Mac OS X

30 May, 2008 — Apple, HOWTO

After spending about an hour configuring what should, in theory, be a simple matter, I figured I’d write a blog post that might one day save another soul an hour or so from his or her life. So, for good karma, basically. In the past, I have set up port forwarding on Linux, Mac OS X and Windows, so I was a little worried that it took me about an hour trying to appease the SSH deities (and daemons).

The command itself is just a single line; the devil is in the parameters. I’m splitting the command over several lines and adding line numbers to illustrate the details and separate the parts of the long-ish command for easier explanation. Feel free to type it all on a single line (after removing the line numbers and the line-break markers ("\") of course!)

1.     ssh \
2.       -L local_port:service_host:service_port \
3.       -p ssh_server_port \
4.       -l ssh_server_username \
5.       -N \
6.       ssh_server_host

Parameters

Now for the various parameters used in the command above. Some of them may be omitted if the defaults are used, but I have included all of them in the example above to cover the most general case.

local_port

The port on your local machine that your local program expects to be able to connect to. If this is one of the reserved ports (i.e., under 1023), you will have to run your ssh tunnel command as root (using sudo). Ports above 1024 are freely available for any user to listen on.

service_host

The fully-qualified domain name or the IP address of the server that is hosting the service that you wish to connect to. For example, if this is a web site, it could be google.com or yahoo.com. It does not have to be under your control, nor does it have to be the machine that you’re SSHing into. It is just any host on the Internet that you can access from ssh_server_host. Often this is a server you are not allowed to access from your own machine, e.g. a chat server or IRC server. Or you may wish to hide the fact from the administrator of your local network that you are connecting to this server (e.g. when you’re out at a coffee shop on a sniffable insecure wireless network, or in a country with laws forbidding access to free information.)

Important: If you’re trying to access a service running on the same machine as ssh_server_host, remember to use 127.0.0.1, not localhost. What’s the difference, you say? Well, since IPv6 is here to stay, localhost can map to either 127.0.0.1 (IPv4) or ::1/128 (IPv6). If your applications aren’t all IPv6-compliant, this can cause some headache. Hopefully, we will all be on IPv6 in the near future, but till then, this is a way to make things work. If you’re trying to use IPv6, you need to use local_port/service_host/service_port (slashes instead of colons.)

service_port

The port number on which the desired service is running. Here are some common port numbers:

Service Port
Web: HTTP 80
Web over SSL: HTTPS 443
Outgoing email: SMTP 25
Incoming email: POP3 110
Incoming email: IMAP 143
VNC 5900
iTunes Music Sharing 3689

ssh_server_host

The machine that you’re SSHing into. This is the one that is running sshd, the SSH daemon.

ssh_server_port

The port number on which the SSH daemon is listening on ssh_server_host. This is most likely 22; you should only use a different value if your sysadmin has told you that the SSH server is running on another port (or if you’re a sysadmin yourself and you set up your SSH server to run on a non-standard port for security through obscurity.)

ssh_server_username

The username you would use to connect to ssh_server_host in a regular SSH session. This may or may not be the same as the username you currently use on your local machine.

The Entire Command, Line by Line

  1. Line 1 simply calls the ssh program;
  2. Line 2 sets up the port forwarding. The -L parameter specifies that this is a remote-to-local tunnel. If you wanted to create a local-to-remote tunnel, you’d have used -R instead of -L. The next three parameters are from our list above, separated by colons. (Use slashes instead of colons for IPv6.) If you want to set up multiple tunnels from the same host, simply repeat line 2 as many times as you’d like, once for each set of local_port:service_host:service_port.
  3. Line 3 selects a port on the ssh_server_host to connect to. Omit this line if you’re connecting to the default port 22.
  4. Line 4 specifies the username to use on the ssh_server_host. It is also possible to use the ssh_server_username@ssh_server_host syntax instead of the -l parameter.
  5. Line 5 indicates to ssh that no commands be run on the remote system. Since you’re using this SSH connection simply for tunneling, this is a useful option to set.
  6. Line 6 contains the most basic parameter of this entire process. Please don’t get this wrong.

Common Errors and Solutions

Problem Solution
Error message: channel 3: open failed: connect failed: Connection refused Change localhost to 127.0.0.1 in the ssh -L parameter.
Cannot listen on port X on local machine because of network policies. Try to use another port locally. Ports such as 3306 (MySQL) may have been left open. These are good to use for SSH tunneling if you aren’t already running MySQL.
Error message: Privileged ports can only be forwarded by root. Use a port above 1024, or try to set up the SSH tunnel as root.
Error message: bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: xxxx
Could not request local forwarding.
Some local server process is already listening on the local port you’re trying to forward to. Pick a different local port and configure your program to connect to th at port instead. If your program cannot be configured to listen to a different port, try to find what server process is occupying that port (netstat -a on Linux or lsof -i -P on Mac OS X) and stop it. Retry setting up the tunnel.
I want other hosts on my network to be able to use the tunnel I established. (By default, only local clients can connect to SSH tunnels established this way.) Use the -g option when setting up the tunnel. Realize that this is insecure, but it may make sense in certain scenarios.
I don’t know what local port is available for me to use. Linux: netstat -a | grep LISTEN
Mac OS X: lsof -i -P | grep LISTEN
will show you the ports that are in use. Generally, you can pick any that’s not already taken. To make sure you’re not breaking some other unknown protocol, check the IANA Well-known Port Numbers list and pick one that’s not taken.

If you’ve not been able to debug this so far, try passing the -v parameter to ssh to see verbose output. Add another -v for more verbose output.

If you’re reading this, and come across any specific source of trouble, please let me know so I can add it to this mini HOWTO.

A Bad Mother’s Day for Mother Nature

12 May, 2008 — Life, Thoughts

Let’s see:

  1. Cyclone in Irrawaddy Delta, Myanmar

    Myanmar Cyclone
  2. Volcano in Chaiten, Chile

    Chaiten Volcano
  3. Earthquake in Sichuan, China

    Sichuan Earthquake
  4. Tornadoes in Central United States

    US Tornadoes
  5. Wildfires in Central Florida, United States

    Florida Wildfires

May the souls of all the victims of all these disasters rest in peace. May the survivors find the courage to get back to their lives, even as they grieve their loved ones.

And yes, we can help them in their efforts.

« Previous PageNext Page »