Thursday, December 25, 2008

Troubleshooting Python's BaseHTTPServer

Everytime I run into some really weird issue I tend to like to write about it if only because it will help me remember it if I ever encounter it at a later date.

One the things I am working on is a standalone Eye-Fi server that receives HTTP SOAP requests from a small wireless device. My previous attempts at implementing one have been mostly successful: I used Apache and PHP to hack something quick together.

Looking back at the code I decided to try my hand at Python and implementing a simple web server. I looked up a tutorial on BaseHTTPServer and went on my merry way. The server worked pretty well when I loaded it through a web browser (IE/Firefox) but seemed to hang when the Eye-Fi card contacted it.

The error was rather interesting in that Python exceptioned indicating the remote host forcibly closed the connection. I was pretty lost as to why that was.

At first I thought it was something wrong with how the BaseHTTPServer was written. So I went looking around the source trying to find anything about TCP timeouts or how to disable them. I eventually broke down and ran WireShark. This showed me that the request was infact getting through to the web server but never making it up to the POST/GET handlers. So I started looking at buffers and how python knows to finish a line. It turns out that there is a bug in the Eye-Fi card. One of the requests it makes does not end with a carriage return or line feed. This causes the server to block on input. Eventually the TCP connection times out and resets are sent.

The solution was simply to upgrade to a later version of the firmware. Firmware 2.0001 seems to work well.

Manually authenticating to a proxy server with netcat or telnet

Sometimes when testing a web application you want to have total control of what you are sending to the web server. However, sometimes that web application is behind an authenticating proxy. This setup is pretty common in large enterprises.

Authentication is basically accomplished with an HTTP header called "Proxy-Authorization". The header's value is typically a username/password pair that is Base64 encoded.

Suppose your username is "Thomas" and your password is "crown". To form the header's value you would concatenate the username, a colon, and the password in one big string. This would turn out as:

Thomas:crown


You would then Base64 encode this value:
VGhvbWFzOmNyb3du

The final header that you would type in a netcat or telnet session would be:
Proxy-Authorization: Basic VGhvbWFzOmNyb3du

A full example of this in use:

telnet proxy.example.com 80
CONNECT somewebsite.example.com:80 HTTP/1.1
Proxy-Authorization: Basic VGhvbWFzOmNyb3du

GET / HTTP/1.1

Also don't forget to watch the carriage returns. The RFC concerning Proxy-Authorization can be found here: http://www.freesoft.org/CIE/RFC/2068/195.htm

Thursday, December 11, 2008

How to determine if a file is 32 or 64 bit on AIX

I was working on an AIX box recently and had to figure out if a binary was 32 or 64 bit. Took me a bit since I am not that familiar with AIX but I eventually figured out two methods:

The first is using the 'file' command. For a 32-bit binary this command produced the following output:

# file libmod_sm20.so
libmod_sm20.so: executable (RISC System/6000) or object module

However on a 64-bit binary it actually stated 64 bits:

# file libmod_sm20.so
libmod_sm20.so: 64-bit XCOFF executable or object module not stripped

The other way was to use the 'nm' command. Not entirely sure what nm stands for. Perhaps name mangling or just 'name'.

In any case this command has the '-X' switch which allows you to specify the type of file. So if you wanted to examine only 32 bit object files you would use '-X 32'. For 64-bit you would use '-X 64'. And for both you would use '-X32-64'.

Running nm with the 32-bit flag:

# nm -X32 libmod_sm20.so
0654-210 libmod_sm20.so is not valid in the current object file mode.
Use the -X option to specify the desired object mode.

Basically gives me an error. However

# nm -X64 libmod_sm20.so

will produce copious amounts of output. This means that libmod_sm20.so is a 64-bit file because the libraries it is linking to are 64-bit.

Monday, December 8, 2008

Tutorial for cURL, libcurl, and Python

When troubleshooting websites it often comes down to what tools you know and if you know how to use them. One of the most valuable tools in my internet troubleshooting toolkit is cURL. I usually use curl to emulate a web browser. It gives me fine grained control over all the traffic that is being sent to the webserver.

cURL (or curl) usually refers to a binary based on libcurl. The binary is basically a very well designed swiss army knife for working with HTTP type connections. It supports other types of connections but for someone just learning curl the most useful is its HTTP capabilities.

The other extremely useful thing about curl is libcurl. The actual library allows you to script a lot of HTTP processes. The language you use to script really depends if there exists a binding for libcurl in your language. While libcurl is written in C it doesn't mean you need to use that language.

Libcurl is available in a number of languages from this website: http://curl.haxx.se/libcurl/bindings.html

So far I've used the bindings in PHP, Java, Perl, and Python. So basically pick the language you are most familiar with and go from there. The best way to learn a new tool is by example.

The following is a code snippet for Python that posts to http://www.google.com/.

import pycurl
import StringIO

proxyHostAndPort = 'localhost:8888'
proxyAuthentication = 'username:password'


buffer = StringIO.StringIO()

c = pycurl.Curl()
c.setopt(c.URL, 'http://www.google.com/')
c.setopt(c.COOKIEJAR, 'cookies.txt')
c.setopt(c.COOKIEFILE, 'cookies.txt')

c.setopt(c.POST, 1)
c.setopt(c.POSTFIELDS, "User=smith&Password=password")
c.setopt(c.VERBOSE, 1)
c.setopt(c.REFERER,'')
c.setopt(c.USERAGENT,'Curl')
c.setopt(c.WRITEFUNCTION, buffer.write)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.SSL_VERIFYPEER, False)

c.setopt(c.PROXY, proxyHostAndPort)
c.setopt(c.PROXYUSERPWD, proxyAuthentication)

c.perform()
c.close()

print buffer.getvalue()

Not that useful but the main things you want to glean from the code are the use of cookies, the POST method, an authenticating proxy, and results in the form of a string. Curl can also set the referrer and useragent headers to arbitrary values. This whole package is what makes curl so powerful.