Thursday, December 25, 2008

Troubleshooting Python's BaseHTTPServer

Everytime I run into some really weird issue I tend to like to write about it if only because it will help me remember it if I ever encounter it at a later date.

One the things I am working on is a standalone Eye-Fi server that receives HTTP SOAP requests from a small wireless device. My previous attempts at implementing one have been mostly successful: I used Apache and PHP to hack something quick together.

Looking back at the code I decided to try my hand at Python and implementing a simple web server. I looked up a tutorial on BaseHTTPServer and went on my merry way. The server worked pretty well when I loaded it through a web browser (IE/Firefox) but seemed to hang when the Eye-Fi card contacted it.

The error was rather interesting in that Python exceptioned indicating the remote host forcibly closed the connection. I was pretty lost as to why that was.

At first I thought it was something wrong with how the BaseHTTPServer was written. So I went looking around the source trying to find anything about TCP timeouts or how to disable them. I eventually broke down and ran WireShark. This showed me that the request was infact getting through to the web server but never making it up to the POST/GET handlers. So I started looking at buffers and how python knows to finish a line. It turns out that there is a bug in the Eye-Fi card. One of the requests it makes does not end with a carriage return or line feed. This causes the server to block on input. Eventually the TCP connection times out and resets are sent.

The solution was simply to upgrade to a later version of the firmware. Firmware 2.0001 seems to work well.

Manually authenticating to a proxy server with netcat or telnet

Sometimes when testing a web application you want to have total control of what you are sending to the web server. However, sometimes that web application is behind an authenticating proxy. This setup is pretty common in large enterprises.

Authentication is basically accomplished with an HTTP header called "Proxy-Authorization". The header's value is typically a username/password pair that is Base64 encoded.

Suppose your username is "Thomas" and your password is "crown". To form the header's value you would concatenate the username, a colon, and the password in one big string. This would turn out as:

Thomas:crown


You would then Base64 encode this value:
VGhvbWFzOmNyb3du

The final header that you would type in a netcat or telnet session would be:
Proxy-Authorization: Basic VGhvbWFzOmNyb3du

A full example of this in use:

telnet proxy.example.com 80
CONNECT somewebsite.example.com:80 HTTP/1.1
Proxy-Authorization: Basic VGhvbWFzOmNyb3du

GET / HTTP/1.1

Also don't forget to watch the carriage returns. The RFC concerning Proxy-Authorization can be found here: http://www.freesoft.org/CIE/RFC/2068/195.htm

Thursday, December 11, 2008

How to determine if a file is 32 or 64 bit on AIX

I was working on an AIX box recently and had to figure out if a binary was 32 or 64 bit. Took me a bit since I am not that familiar with AIX but I eventually figured out two methods:

The first is using the 'file' command. For a 32-bit binary this command produced the following output:

# file libmod_sm20.so
libmod_sm20.so: executable (RISC System/6000) or object module

However on a 64-bit binary it actually stated 64 bits:

# file libmod_sm20.so
libmod_sm20.so: 64-bit XCOFF executable or object module not stripped

The other way was to use the 'nm' command. Not entirely sure what nm stands for. Perhaps name mangling or just 'name'.

In any case this command has the '-X' switch which allows you to specify the type of file. So if you wanted to examine only 32 bit object files you would use '-X 32'. For 64-bit you would use '-X 64'. And for both you would use '-X32-64'.

Running nm with the 32-bit flag:

# nm -X32 libmod_sm20.so
0654-210 libmod_sm20.so is not valid in the current object file mode.
Use the -X option to specify the desired object mode.

Basically gives me an error. However

# nm -X64 libmod_sm20.so

will produce copious amounts of output. This means that libmod_sm20.so is a 64-bit file because the libraries it is linking to are 64-bit.

Monday, December 8, 2008

Tutorial for cURL, libcurl, and Python

When troubleshooting websites it often comes down to what tools you know and if you know how to use them. One of the most valuable tools in my internet troubleshooting toolkit is cURL. I usually use curl to emulate a web browser. It gives me fine grained control over all the traffic that is being sent to the webserver.

cURL (or curl) usually refers to a binary based on libcurl. The binary is basically a very well designed swiss army knife for working with HTTP type connections. It supports other types of connections but for someone just learning curl the most useful is its HTTP capabilities.

The other extremely useful thing about curl is libcurl. The actual library allows you to script a lot of HTTP processes. The language you use to script really depends if there exists a binding for libcurl in your language. While libcurl is written in C it doesn't mean you need to use that language.

Libcurl is available in a number of languages from this website: http://curl.haxx.se/libcurl/bindings.html

So far I've used the bindings in PHP, Java, Perl, and Python. So basically pick the language you are most familiar with and go from there. The best way to learn a new tool is by example.

The following is a code snippet for Python that posts to http://www.google.com/.

import pycurl
import StringIO

proxyHostAndPort = 'localhost:8888'
proxyAuthentication = 'username:password'


buffer = StringIO.StringIO()

c = pycurl.Curl()
c.setopt(c.URL, 'http://www.google.com/')
c.setopt(c.COOKIEJAR, 'cookies.txt')
c.setopt(c.COOKIEFILE, 'cookies.txt')

c.setopt(c.POST, 1)
c.setopt(c.POSTFIELDS, "User=smith&Password=password")
c.setopt(c.VERBOSE, 1)
c.setopt(c.REFERER,'')
c.setopt(c.USERAGENT,'Curl')
c.setopt(c.WRITEFUNCTION, buffer.write)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.SSL_VERIFYPEER, False)

c.setopt(c.PROXY, proxyHostAndPort)
c.setopt(c.PROXYUSERPWD, proxyAuthentication)

c.perform()
c.close()

print buffer.getvalue()

Not that useful but the main things you want to glean from the code are the use of cookies, the POST method, an authenticating proxy, and results in the form of a string. Curl can also set the referrer and useragent headers to arbitrary values. This whole package is what makes curl so powerful.

Wednesday, November 12, 2008

Compiling PyXMLSec for Windows

For some reason there doesn't seem to be anyone that compiled PyXMLSec for Windows. Not sure why but I tried searching for all sorts of terms and nada. So I went on a bit of an adventure to get this thing working. Fair warning: while this works for the most part I mixed and matched a few of the binary packages without really researching the platform they were compiled on and whether they really would be compatible.

If you are looking to just download the binary go here:
http://returnbooleantrue.blogspot.com/2009/04/pyxmlsec-windows-binary.html


List of everything I used:

- MinGW. I used version 5.1.4.
- PyXMLSec source code. For this I used pyxmlsec-0.3.0.
- libxml2-2.5.10.win32
- iconv-1.9.2.win32
- libxmlsec-1.2.11.win32
- libxslt-1.1.24.win32
- zlib-1.2.3.win32
- Win32OpenSSL-0_9_8i.exe and Microsoft Visual C++ 2008 Redistributable Package (x86) from the same site



First thing I did was look at the setup.py script included inside pyxmlsec.

A bit down in the file is a list of all the files that need to be compiled.

sources = ["utils.c", "wrap_objs.c",
"app.c", "base64.c", "buffer.c", "errors.c",
"keyinfo.c", "keys.c", "keysdata.c", "keysmngr.c",
"list.c", "membuf.c", "nodeset.c", "parser.c",
"templates.c", "transforms.c", "version.c",
"xmldsig.c", "xmlenc.c", "xmlsec.c", "xmltree.c",
"x509.c",
"xmlsecmod.c"]



Looked pretty straghtforward to me. Took out my compiler and proceded to start:

> gcc *.c

One can hope right? Obviously all this does is spew out a ridiculous number of errors and warnings. So I set about troubleshooting them one by one. I decide to just pick a single .c file to just get a small number of warnings. I was too lazy to pipe output to a file.

> gcc utils.c

In file included from utils.c:27:
utils.h:5:20: Python.h: No such file or directory

well this is simple enough to fix. utils.c is expecting the Python.h header file. So lets give the compiling the path to the headers using the -I option.

> gcc -IC:\Python25\include utils.c

utils.c:(.text+0x4f): undefined reference to `_imp__PyInstance_Type'

So these turn out to be linking errors. So let us link to the Python libraries. We need to include the -lpython25 to explicitly tell gcc that the library we want is python25.

> gcc -IC:\Python25\include -LC:\Python25\libs utils.c -lpython25

utils.c:(.text+0x8e): undefined reference to `xmlsec_error'

So now I get errors about linking to the xmlsec library. Well this makes sense since I didn't tell gcc where the xmlsec libraries were. PyXMLSec depends on xmlsec.

At this point I am dreading I will have to compile xmlsec for windows as well. Luckily windows binaries for the XMLSec Library (as well as LibXML2, LibXSLT and OpenSSL) are available from Igor Zlatkovic. I will make a big caveat here though. Some of the binaries that I got from Igor did not work for me. One of the errors was about newline characters being missing in his openssl binary package. To solve this I downloaded

Win32 OpenSSL v0.9.8i from Shining Light Productions (http://www.slproweb.com/products/Win32OpenSSL.html) and linked against that.

The other issue was with libxml2. Don't try to use the newest version or else you will run into issues.

Now here is the relatively time consuming part. Need to unpack them all and make sure gcc knows about the locations.

This is what we have so far.
> gcc -IC:\Python25\include -I -LC:\Python25\libs utils.c -lpython25

Add in all the libraries and we get:

gcc -IC:\Python25\include -IC:\Build\iconv-1.9.2.win32\include-IC:\Build\libxml2-2.7.2.win32\include
-IC:\Build\libxmlsec-1.2.11.win32\include
-IC:\Build\libxslt-1.1.24.win32\include
-IC:\Build\openssl-0.9.8a.win32\include
-IC:\Build\zlib-1.2.3.win32\include -LC:\Python25\libs
-LC:\Build\iconv-1.9.2.win32\lib -LC:\Build\libxml2-2.7.2.win32\lib
-LC:\Build\libxmlsec-1.2.11.win32\lib
-LC:\Build\libxslt-1.1.24.win32\lib
-LC:\Build\openssl-0.9.8a.win32\lib -LC:\Build\zlib-1.2.3.win32\lib
parser.c -lpython25 -llibxml2 -llibxmlsec -llibxslt -libexslt
-llibxmlsec-openssl -lbz2 -liconv -lzlib

Looks really ugly but running it gives us a glimmer of hope:

In file included from xmlsecmod.h:4,
from parser.c:25:
C:/Build/libxmlsec-1.2.11.win32/include/xmlsec/crypto.h:54:2: #error
No crypto library defined

So we need to pick a crypto library. We do this through the gcc option -D.

-DXMLSEC_CRYPTO_OPENSSL

While looking for define options I also happen upon a post that said
to define -DLIBXML_STATIC -DLIBXSLT_STATIC -DXMLSEC_STATIC.

Another try:

gcc -DXMLSEC_CRYPTO_OPENSSL -DLIBXML_STATIC -DLIBXSLT_STATIC
-DXMLSEC_STATIC -IC:\Python25\include
-IC:\Build\iconv-1.9.2.win32\include
-IC:\Build\libxml2-2.5.10.win32\include
-IC:\Build\libxmlsec-1.2.11.win32\include
-IC:\Build\libxslt-1.1.24.win32\include -IC:\OpenSSL\include
-IC:\Build\zlib-1.2.3.win32\include -LC:\Python25\libs
-LC:\Build\iconv-1.9.2.win32\lib -LC:\Build\libxml2-2.5.10.win32\lib
-LC:\Build\libxmlsec-1.2.11.win32\lib
-LC:\Build\libxslt-1.1.24.win32\lib -LC:\OpenSSL\lib\MinGW
-LC:\Build\zlib-1.2.3.win32\lib *.c -lpython25 -llibxml2 -llibxmlsec
-llibxslt -llibexslt -llibxmlsec-openssl -lbz2 -liconv -lzlib


base64.c:(.text+0x5a1): undefined reference to `xmlMalloc'
base64.c:(.text+0x605): undefined reference to `xmlFree'

This is what you'd get if you tried to use a newer libxml2. I tried these versions:

libxml2-2.7.2.win32 - Doesn't work.
libxml2-2.6.9.win32 - Doesn't work.
libxml2-2.5.10.win32 - Works.

The last error I got was:
/mingw/lib/libmingw32.a(main.o):main.c:(.text+0xbd): undefined
reference to `WinMain@16'

To fix that I added a -shared option to gcc. I also added a -o option
so I wouldn't just get a a.exe.

gcc -DXMLSEC_CRYPTO_OPENSSL -DLIBXML_STATIC -DLIBXSLT_STATIC
-DXMLSEC_STATIC -IC:\Python25\include
-IC:\Build\iconv-1.9.2.win32\include
-IC:\Build\libxml2-2.5.10.win32\include
-IC:\Build\libxmlsec-1.2.11.win32\include
-IC:\Build\libxslt-1.1.24.win32\include -IC:\OpenSSL\include
-IC:\Build\zlib-1.2.3.win32\include -LC:\Python25\libs
-LC:\Build\iconv-1.9.2.win32\lib -LC:\Build\libxml2-2.5.10.win32\lib
-LC:\Build\libxmlsec-1.2.11.win32\lib
-LC:\Build\libxslt-1.1.24.win32\lib -LC:\OpenSSL\lib\MinGW
-LC:\Build\zlib-1.2.3.win32\lib *.c -lpython25 -llibxml2 -llibxmlsec
-llibxslt -llibexslt -llibxmlsec-openssl -lbz2 -liconv -lzlib -shared
-o pyxmlsec.pyd

Now that I know everything compiles cleanly I decide to go back and try my luck editing setup.py. To be honest I just deleted a lot of stuff and added what I needed.


from distutils.core import setup, Extension
import sys, commands


define_macros = []
include_dirs = []
library_dirs = []
libraries = []

define_macros.append(("XMSEC_CRYPTO_OPENSSL",None))
define_macros.append(("LIBXML_STATIC",None))
define_macros.append(("LIBXSLT_STATIC",None))
define_macros.append(("XMLSEC_STATIC",None))

include_dirs.append("C:\\Build\\iconv-1.9.2.win32\\include")
include_dirs.append("C:\\Build\\libxml2-2.5.10.win32\\include")
include_dirs.append("C:\\Build\\libxmlsec-1.2.11.win32\\include")
include_dirs.append("C:\\Build\\libxslt-1.1.24.win32\\include")
include_dirs.append("C:\\OpenSSL\\include")
include_dirs.append("C:\\Build\\zlib-1.2.3.win32\\include")

library_dirs.append("C:\\Build\\iconv-1.9.2.win32\\lib")
library_dirs.append("C:\\Build\\libxml2-2.5.10.win32\\lib")
library_dirs.append("C:\\Build\\libxmlsec-1.2.11.win32\\lib")
library_dirs.append("C:\\Build\\libxslt-1.1.24.win32\\lib")
library_dirs.append("C:\\OpenSSL\\lib\\MinGW")
library_dirs.append("C:\\Build\\zlib-1.2.3.win32\\lib")

libraries.append("python25")
libraries.append("libxml2")
libraries.append("libxmlsec")
libraries.append("libxslt")
libraries.append("libexslt")
libraries.append("libxmlsec-openssl")
libraries.append("bz2")
libraries.append("iconv")
libraries.append("zlib")

em = Extension("xmlsecmod",
sources = ["utils.c", "wrap_objs.c",
"app.c", "base64.c", "buffer.c", "errors.c",
"keyinfo.c", "keys.c", "keysdata.c", "keysmngr.c",
"list.c", "membuf.c", "nodeset.c", "parser.c",
"templates.c", "transforms.c", "version.c",
"xmldsig.c", "xmlenc.c", "xmlsec.c", "xmltree.c",
"x509.c",
"xmlsecmod.c"],
define_macros = define_macros,
include_dirs = include_dirs,
library_dirs = library_dirs,
libraries = libraries
)

doclines = __doc__.split("\n")

setup(name = "pyxmlsec",
version = "0.3.0",
description = doclines[0],
long_description = "\n" . join(doclines[2:]),
author = "Valery Febvre",
author_email = "vfebvre@easter-eggs.com",
license = "GNU GPL",
platforms = ["any"],
url = "http://pyxmlsec.labs.libre-entreprise.org",
ext_modules = [em],
py_modules = ["xmlsec", "xmlsec_strings"]
)

Ran setup.py with the following command:

C:\Python25\python.exe setup.py build --compiler=mingw32 install

running build
running build_py
running build_ext
running install
running install_lib
copying build\lib.win32-2.5\xmlsec.py -> C:\Python25\Lib\site-packages
copying build\lib.win32-2.5\xmlsecmod.pyd -> C:\Python25\Lib\site-packages
copying build\lib.win32-2.5\xmlsec_strings.py -> C:\Python25\Lib\site-packages
byte-compiling C:\Python25\Lib\site-packages\xmlsec.py to xmlsec.pyc
byte-compiling C:\Python25\Lib\site-packages\xmlsec_strings.py to
xmlsec_strings.pyc
running install_egg_info
Writing C:\Python25\Lib\site-packages\pyxmlsec-0.3.0-py2.5.egg-info

Everything looks good. Startup python and try to import xmlsec. At this point you'll probably get a few errors about missing DLLs and such.

I had to install Libxml and Libxslt Python Bindings for Windows
http://users.skynet.be/sbi/libxml-python/


>>> import xmlsec
Traceback (most recent call last):
File "", line 1, in
File "xmlsec.py", line 46, in
import xmlsecmod
ImportError: DLL load failed: The specified module could not be found.

A few other things I had to do:

Copied C:\Build\libxmlsec-1.2.11.win32\bin\libxmlsec-openssl.dll to C:\Python25
Copied C:\Build\libxmlsec-1.2.11.win32\bin\libxmlsec.dll to C:\Python25
Copied C:\Build\libxmlsec-1.2.11.win32\bin\libxslt.dll to C:\Python25
Copied C:\Build\libxmlsec-1.2.11.win32\bin\libexslt.dll to C:\Python25
Copied C:\Build\libxmlsec-1.2.11.win32\bin\libxmlsec-openssl.dll to C:\Python25

I also downloaded a Windows dependency walker to troubleshoot some OpenSSL DLL issues. The solution was to reinstall OpenSSL and have it put its DLLs in my Windows system32 directory.

C:\Python25>python
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlsec
>>>

In the end I was rewarded with a nice clean import of xmlsec. The examples even work!