Django TypeError Exception

One of the central pieces of Django is urls.py:

http://docs.djangoproject.com/en/dev/topics/http/urls/

This file is well documented and serves as a central place to match what pages get served by what URLs. Seems simple enough since they accept regular expressions.

One thing I did run into though was getting a TypeError exception. It looked something like this:

TypeError at /applicationname/

index() takes exactly 1 argument (2 given)

To me I was puzzled why this was happening. The reason is that when Django parses the regular expression it not only does a match but saves each of the tuples in the regular expression. In the documentation it is this line:

"The view gets passed an HttpRequest as its first argument and any values captured in the regex as remaining arguments."

So what happens is that when you start to get fancy with your regular expressions the function signature in your views.py needs to change as well to accommodate the extra parameters.

PyXMLSec Windows Binary

Xmlsec Python Bindings for Windows

Getting the python xmlsec library to work on windows is a bit of work though it really shouldn't be. Provided here are windows installers for xmlsec. The installer itself is produced by python distutils.

In order to get it to work you will also need libxml2. I am providing a copy of it here as well.

pyxmlsec-0.3.0.win32-py2.6.exe
libxml2-python-2.7.3.win32-py2.6.exe

I used MinGW for compilation. The entire environment I used to compile pyxmlsec is available here. Look inside the pyxmlsec directory for build.bat.
PythonWindowsCompileEnvironment.zip

Eye Fi Standalone Server Version 2.0

I'd like to release an updated version of the Eye Fi standalone server in python that I have been working on. This version should work on Linux, Mac OS X, Windows, Solaris, or wherever else you can load a Python interpreter. As always I love comments so if you are using this feel free to e-mail me or drop me a note!

Source on GitHub: http://github.com/tachang/EyeFiServer

Download (zip): http://www.darkeneddesire.com/EyeFiServer/2.0/EyeFiServer-v2.0.zip

I know some people just like to browse around the source without having to download stuff (I'm one of those people):

http://www.darkeneddesire.com/EyeFiServer/2.0/Release/

This new version has the following features:

• The server can now execute an arbitrary command on each uploaded photo. This is a very dangerous feature and should be used with caution. On the other hand it is also very cool. You can have the server FTP files, display them using an image viewer, or even run sorting programs on the images.
  
• Improved security: the server now generates its own nonces instead of using one that was hard coded. The nonce is based on the random library provided by python. The INTEGRITYDIGEST field is also checked.
• Ability to read settings from a configuration file (there is a included DefaultSettings.ini for reference). The file allows you to configure the listen port, console output, logging, download location, and execute on upload, and upload key.

Some other notable improvements but not really features are the addition of regression tests and support for Python 2.5. The regression tests are interesting since I run them against the official Eye-Fi Manager to make sure my behavior is a close match.

Getting usage information as to how to specify a configuration file:

C:\EyeFiServer\Release>EyeFiServer.py -h
Usage: EyeFiServer.py [options]

Options:
-h, --help show this help message and exit
-c CONFIGFILE, --config=CONFIGFILE
Path to configuration file (example in
DefaultSettings.ini)

Actually specifying a configuration file:
C:\EyeFiServer\Release>EyeFiServer.py -c DebugSettings.ini

Converting LDIF/LDAP data into a CSV file

LDIF to CSV

I work with LDAP on a regularly basis. I frequently have to pull data using ldapsearch. While the data that ldapsearch spits out is a decent representation sometimes I want something a bit easier to work with.

The format I use most tends to be CSV. While the acronym stands for comma separated value the format may be used to describe data that is printed out line by line and separated by anything you can imagine.

The problem for me is finding a decent LDIF to CSV converter. The ones that I have tried tend to choke on any number of issues. Some of these include binary data or failing to normalize the multivalued attributes.

I finally got sick enough of these issues that I decided to write my own. You can download it here:

Python/Linux/Source

Windows

Update: Pushed the code to GitHub: https://github.com/tachang/ldiftocsv

Hopefully the Windows binary works for you. If it doesn't then just download Python 2.6 and run the source manually. I am providing the binary just for convenience.

Using LDIFtoCSV
Running "python LDIFtoCSV.py" should give you the usage text.

usage: LDIFtoCSV.py [options]

-o <filename> : File to write output. By default this is set to sys.stdout
-l <filename> : File to write logging output. By default there is no logging.
-F <char> : Character to separate the fields by. By default this is a comma. i.e. -F","
-D <char> : Character to delimit the text value by. By default this is a double quote. i.e. -D"""
-M <num> : The maximum number of columns a multivalued attribute should take up (default: 5). This is common with the objectClass attribute where it can have over 20 values. Do you want to have 20 columns each with the same heading objectClass or do you want to limit it.

Here are some common command lines that I use (assuming you have a test.ldif):

Outputs the CSV straight to standard output:
python LDIFtoCSV.py test.ldif

Outputs CSV to standard output with semicolons as the delimiter:python LDIFtoCSV.py -F";" test.ldif
Outputs CSV to standard output with pipes as the delimiter and text surrounded by carrots:
python LDIFtoCSV.py -F"|" -D"^" test.ldif

Downloading the Global Address List from Outlook/Exchange

Getting the Data

Recently I had a need to dump the entire Global Address List from within Outlook. Usually the address list will contain the contact information for your entire organization.

The most effective way I have found to do this is a straight LDAP search on the Active Directory and pull everything into an LDIF file. Conceptually it is straightforward but I have found the most difficult part is usually trying to get all the pieces together. So subscribing to the French idea of "mise en place" here is what you'll need:

Username
Your username is your fully qualified distinguished name. It is not your Windows login name. It commonly looks something like "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com".

The easiest way I have found to obtain your username is to login to a Windows machine and then put the following two lines in a text file with the extension .vbs:

Set objADSysInfo = WScript.CreateObject("ADSystemInfo")
result = InputBox("Active Directory Username (copy and paste as necessary)", "Active Directory Username", objADSysInfo.UserName, 100, 100)

Password
The password will be your domain password.

Domain controller hostname
Couple of ways to get this. This will be the Active Directory/LDAP server that you will extract the entries. An echo %logonserver% at the command prompt usually works.

Ldapsearch tool
Easiest way to get the tools is from Sun's iPlanet Directory SDK. Go to http://www.sun.com/download/products.xml?id=3ec28dbd and select the iPlanet Directory SDK downloads. After download the zipfile find the executable named ldapsearch.exe.

Performing the search
Here is an example syntax to extract out all the users:

Windows:
ldapsearch.exe -b OU=Users,OU=USA,DC=example,DC=com -h host.example.com -p 389 -D "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com" -w - (objectClass=*)

Unix:
ldapsearch -b OU=Users,OU=USA,DC=example,DC=com -h host.example.com -p 389 -D "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com" -W -x "(objectClass=*)"

The -b is for the base. In this example I knew the users were all stored in that branch. -D is for username. -w - (that is a dash w followed by a dash) means to read the password from the console. On Unix the big -W is to prompt and the -x indicates you want simple authentication (cleartext username/password).

You might want to add a " > output.txt" to send the output to a file. If you do that you will have to supply the password on the command line.

Eye Fi Standalone Server

Eye Fi Linux Hacking

So I spent some time over my vacation learning a bit more about Python. What better way to learn a language than to implement something you want or need, right?

I am releasing a standalone Eye-Fi server written in Python. Basically I saw Dave Hansen's post (http://dave-hansen.blogspot.com/2008/12/freestanding-server.html) and went ahead and did it. This software works on Windows with Python 2.6. I have not tested it on Linux/Unix yet but I assume it will work seeing as how it is written in Python. Please let me know if you try this software out and it works or doesn't. I personally would love any comments!

Python script is here: http://www.darkeneddesire.com/EyeFiServer/EyeFiServer.py

General Architecture Notes

This is a standalone Eye-Fi Server that is designed to take the place of the Eye-Fi Manager.

Starting this server creates a listener on port 59278. I use the BaseHTTPServer class included with Python. I look for specific POST/GET request URLs and execute functions based on those
URLs.

Currently all files are downloaded to the directory in which this script is run.

To use this script you need to have your Eye-Fi upload key.
It is in C:\Documents and Settings\[User]\Application Data\Eye-Fi\Settings.xml

Simple search for "eyeFiUploadKey" and replace it with your key.

--

Update (4/4/09) - http://returnbooleantrue.blogspot.com/2009/04/eye-fi-standalone-server-version-20.html

Troubleshooting Python's BaseHTTPServer

Everytime I run into some really weird issue I tend to like to write about it if only because it will help me remember it if I ever encounter it at a later date.

One the things I am working on is a standalone Eye-Fi server that receives HTTP SOAP requests from a small wireless device. My previous attempts at implementing one have been mostly successful: I used Apache and PHP to hack something quick together.

Looking back at the code I decided to try my hand at Python and implementing a simple web server. I looked up a tutorial on BaseHTTPServer and went on my merry way. The server worked pretty well when I loaded it through a web browser (IE/Firefox) but seemed to hang when the Eye-Fi card contacted it.

The error was rather interesting in that Python exceptioned indicating the remote host forcibly closed the connection. I was pretty lost as to why that was.

At first I thought it was something wrong with how the BaseHTTPServer was written. So I went looking around the source trying to find anything about TCP timeouts or how to disable them. I eventually broke down and ran WireShark. This showed me that the request was infact getting through to the web server but never making it up to the POST/GET handlers. So I started looking at buffers and how python knows to finish a line. It turns out that there is a bug in the Eye-Fi card. One of the requests it makes does not end with a carriage return or line feed. This causes the server to block on input. Eventually the TCP connection times out and resets are sent.

The solution was simply to upgrade to a later version of the firmware. Firmware 2.0001 seems to work well.

Tutorial for cURL, libcurl, and Python

When troubleshooting websites it often comes down to what tools you know and if you know how to use them. One of the most valuable tools in my internet troubleshooting toolkit is cURL. I usually use curl to emulate a web browser. It gives me fine grained control over all the traffic that is being sent to the webserver.

cURL (or curl) usually refers to a binary based on libcurl. The binary is basically a very well designed swiss army knife for working with HTTP type connections. It supports other types of connections but for someone just learning curl the most useful is its HTTP capabilities.

The other extremely useful thing about curl is libcurl. The actual library allows you to script a lot of HTTP processes. The language you use to script really depends if there exists a binding for libcurl in your language. While libcurl is written in C it doesn't mean you need to use that language.

Libcurl is available in a number of languages from this website: http://curl.haxx.se/libcurl/bindings.html

So far I've used the bindings in PHP, Java, Perl, and Python. So basically pick the language you are most familiar with and go from there. The best way to learn a new tool is by example.

The following is a code snippet for Python that posts to http://www.google.com/.

import pycurl
import StringIO

proxyHostAndPort = 'localhost:8888'
proxyAuthentication = 'username:password'


buffer = StringIO.StringIO()

c = pycurl.Curl()
c.setopt(c.URL, 'http://www.google.com/')
c.setopt(c.COOKIEJAR, 'cookies.txt')
c.setopt(c.COOKIEFILE, 'cookies.txt')

c.setopt(c.POST, 1)
c.setopt(c.POSTFIELDS, "User=smith&Password=password")
c.setopt(c.VERBOSE, 1)
c.setopt(c.REFERER,'')
c.setopt(c.USERAGENT,'Curl')
c.setopt(c.WRITEFUNCTION, buffer.write)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.SSL_VERIFYPEER, False)

c.setopt(c.PROXY, proxyHostAndPort)
c.setopt(c.PROXYUSERPWD, proxyAuthentication)

c.perform()
c.close()

print buffer.getvalue()

Not that useful but the main things you want to glean from the code are the use of cookies, the POST method, an authenticating proxy, and results in the form of a string. Curl can also set the referrer and useragent headers to arbitrary values. This whole package is what makes curl so powerful.