Monday, July 13, 2009

PySAML - A SAML 2.0 Toolkit for Python

Security Assertion Markup Language (SAML) can be a bit confusing to understand. At its core SAML is just a protocol with defined messages written in XML. The main purpose of SAML is to enable you to log in at one place such as a website and then jump over to another website without having to log in again. This setup is commonly called "federation".

It is similiar to OAuth. I see SAML continuing to make headroom in the enterprise space while OAuth stays strong in the consumer space. However in the future I hope these two technologies will end up playing well together.

In either case I am releasing a small Python library for generating SAML assertions. The main purpose of this is to learn SAML by doing (actually having to create an assertion gives me a good idea of the complexity of the protocol).

The library depends on M2Crypto so download and install it:

M2Crypto depends on SWIG so you might need that as well.

Once that is installed you can download my distribution of PySAML here:

Python 2.6 - Win32:

Unix (source):

For windows you just run the executable. For unix you should run the following commands:

# cd PySAML
# python build
# python install

There are examples in the "examples" folder if you download the full source.

The whole project is available on github as well:

Thursday, July 9, 2009

Trying to get Dual-SIM to work in T-Mobile G1

There is a product being sold called "Magic SIM".

Basically it is a SIM card that has two slots and a small memory chip. After physically cutting up your two SIM cards and putting them into the slots you insert the whole thing into your phone.

If your phone supports the GSM STK (SIM Tool Kit) standard then you can switch back and forth between the two SIM cards.

However for your phone to actually support the standard the manufacturer had to have built it into the operating system as well as provide a tool to "talk" using the STK.

I currently see that the Android development tree has some Stk development going on. I downloaded a precompiled Stk.apk and installed it on my phone with the card. However it does not seem to work.

While I do see the SIM Toolkit menu icon and it does seem to read from the custom card I am unable to obtain a carrier signal from either of my SIM cards. When I scroll to any of the menu options such as "SIM 1" or "SIM 2" and press the trackball the phone simply gives me a rotating circle on the upper right saying it is busy. And at that point it hangs.

Here is a quick and dirty image of my phone as well as what the SIM menu looks like:

Thursday, June 18, 2009

Using Github Through Draconian Proxies (Windows And Unix)

Here is a pretty standard scenario at most corporations:

- All access to the internet is restricted to a proxy
- The proxy only allows connections out on port 80 and 443
- CONNECT method is only enabled for 443
- Proxy Authentication is required (NTLM or Basic)

I like to use both Windows and Unix environments. On Unix tunneling to Github is a bit easier because lots of tools are included.


1. Download Git. At the time I was writing this I was using Ubuntu so I simply did apt-get install git-core

2. Download and install corkscrew ( This is a tool for tunneling SSH through HTTP proxies.

3. Edit or create the file ~/.ssh/config and put the following:

ProxyCommand /usr/bin/corkscrew 443 %h %p ~/.ssh/myauth

User git
Port 22
IdentityFile "/media/truecrypt1/Keys/GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

User git
Port 443
IdentityFile "/media/truecrypt1/Keys/GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

  • The ProxyCommand is invoked when ssh needs to make a connection. We are telling ssh to use /usr/bin/corkscrew. This is a 3rd party program that sets up a socket through the HTTP proxy.
  • The program /usr/bin/corkscrew takes as its 5th argument a file containing credentials for your HTTP proxy. Not all proxies need authentication but if you do just put in the file a single line formatted username:password.
  • The Host indicates to ssh that if we are connecting to to use these specific settings. There is nothing special here except we specify the location of the private key that corresponds to the public key we had over in
  • Notice we have another entry titled "Host" . This is to get around proxies that only allow the CONNECT command over 443 (the truly locked down ones). To get around this github setup a whole separate host that listens on port 443. We add both entries here since they are both valid.
4. If everything is setup correctly you should be able to run:
# ssh

Hi tachang! You've successfully authenticated, but GitHub does not provide shell access.
Connection to closed.

If this doesn't work you can run
# ssh

And get the exact same thing. If the first command didn't work it means you are using a proxy that blocks CONNECT on port 22. Almost no proxies block CONNECT on port 443 because you need that for SSL.

We get a no shell access message from github because we are trying to obtain a shell and github has it disabled. However this indicates everything is working. This concludes the setup for Unix.



1. Download msysgit

Some settings:

- "Run Git from the Windows Command Prompt"
- "Use OpenSSH" (this one is very important)
- Pick your line endings

2. Download connect.c

This tool deserves its own post mostly because of its utter simplicity. It mirrors the open source tool corkscrew and is used for tunneling through proxies. Yes the tool's name is really called "connect.c".

For Window's users, a pre-compiled binary is available:
I put my connect.exe in C:\Windows\connect.exe

3. Decide whether you like to use the Windows cmd.exe to do stuff or the Cygwin style shell. Or both.

Cygwin Git Bash Shell
For the Cygwin style shell start up the Git icon and edit the file ~/.ssh/config
*Make sure the file has no extension.

Put the following in that file:

ProxyCommand /c/windows/connect.exe -H %h %p

User git
Port 22
IdentityFile "/c/Keys/GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

User git
Port 443
IdentityFile "/c/Keys/GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

  • Notice the slash style in order to access the file system.
  • The proxy username is specified as part of the proxy setting. The password for the proxy is prompted for. Read more about connect.c to figure out how to get rid of this prompt.
At this point, using the Git Bash shell should yield:

$ ssh

Hi tachang! You've successfully authenticated, but GitHub does not provide shell access.
Connection to closed

Windows cmd.exe shell

Suppose you don't like the Git Bash shell. You prefer the cmd.exe interpreter.

- Go to your config file at C:\Documents and Settings\\.ssh\config
- Make a copy of it or make a new one. I called mine config-windows

Put the following in the file:

ProxyCommand C:/Windows/connect.exe -H %h %p

User git
Port 22
IdentityFile "C:\Keys\GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

User git
Port 443
IdentityFile "C:\Keys\GitHubKey.private"
TCPKeepAlive yes
IdentitiesOnly yes

  • Notice the mixture of slash styles. I find this rather odd but it is what works. We have a forward slash style for the ProxyCommand but for the IdentityFile a forward slash or backward slash both work.
Running the command (making sure we run Git\bin's ssh.exe and not some other one in the PATH):

C:\Program Files\Git\bin>ssh.exe -F "C:\Documents and Settings\\.ssh\config-windows"

Hi tachang! You've successfully authenticated, but GitHub does not provide shell access.
Connection to closed

General Git Cloning

- Make sure you are using the right Git URL:

Suppose your Public Clone URL is: git://

You should use the following URL that utilizes the SSH transport:

git clone ssh://
git clone ssh://

Monday, June 15, 2009

Django TypeError Exception

One of the central pieces of Django is

This file is well documented and serves as a central place to match what pages get served by what URLs. Seems simple enough since they accept regular expressions.

One thing I did run into though was getting a TypeError exception. It looked something like this:

TypeError at /applicationname/

index() takes exactly 1 argument (2 given)

To me I was puzzled why this was happening. The reason is that when Django parses the regular expression it not only does a match but saves each of the tuples in the regular expression. In the documentation it is this line:

"The view gets passed an HttpRequest as its first argument and any values captured in the regex as remaining arguments."

So what happens is that when you start to get fancy with your regular expressions the function signature in your needs to change as well to accommodate the extra parameters.

Thursday, June 11, 2009

SiteMinder R12 Admin GUI

If you are upgrading or installing SiteMinder R12 you may have noticed that the traditional admin GUI has changed. The Java applet now asks you for 4 fields.

Username: This is the username found in the "Administrators" list on the old version 6 GUI. The applet is not case sensitive.
Password: The password in the "Administrators" list.
Host Name: A 4.x compatible Agent's name
Passphrase: The Shared Secret of the 4.x compatible agent

The Host Name and Passphrase fields are where things get interesting. SiteMinder R12 is trying heavily to move the functions of this applet to a web based system. The web based system communicates with the SiteMinder policy server using the API. However, the API still uses 4.x agents as the method of authentication to the policy server. 4.x agents have an associated shared secret.

Thus to get into the traditional 6.0 GUI you need to have a 4.x compatible agent. But what happens when you upgrade to SiteMinder R12 and don't have one? You have two options:

1. Install the SiteMinder Web Access Manager GUI - Unfortunately the installer for Windows is over 2GB
2. Manually create a 4.x Web Agent

Option 2 is easiest. We are going to create a file with the proper parameters and then use smobjimport to import it directly to the policy store. To do this create a file called "Generic4xAgent" and put the following in the file:

objectclass: Agent
Oid: 01-39c83ef9-5c51-4fb4-ba13-193543b8a9d4
Name: siteminder
AgentType: 10-8d78bb96-ae15-11d1-9cdd-006008aac24b
RealmHintAttrId: 0

objectclass: TrustedHost
Oid: 24-5ea55269-d8a9-47a0-864c-c97026c00b99
Name: siteminder
Secret: password
Is4xHost: true
RolloverEnabled: false
SecretGenTime: 00000000-00000000-000000000000000000000000000000000000000000000000
SecretUsedTime: 00000000-00000000-000000000000000000000000000000000000000000000000

Save the file as "Generic4xAgent". Now run the command:
smobjimport -iGeneric4xAgent -dsiteminder -wpassword -c

-i is the filename
-d is the SiteMinder admin username
-w is the SiteMinder admin password
-c indicates that the file has cleartext passwords

This creates a 4.x agent named "siteminder" with a shared secret of "password".

At this point you should be able to login to the traditional admin UI.

Friday, May 29, 2009

Fixing pkg_add -r on FreeBSD (things no one tells you)

On FreeBSD when you want to install a program you have to make a choice. Your two choices are to install a package or compile from ports.

Installing a package means you download a binary package and copy all the files to the right places. Everything is already compiled. This is how Windows works. When you install a piece of software it has already been compiled.

Your second option is to compile from ports. What this means is that the source files are downloaded and then compiled on your system. You need to have all the prerequisites for the compile to be successful.

Why choose one over the other? Well with packages sometimes the binary doesn't work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn't match anymore with the author's compiled environment. However packages are fast to install. No need to break out gcc and compile source code.

Ports on the other hand will almost always work. They are targetted for your system because you are compiling them. It takes a bit longer but you also get to customize any of the build options.

To install a program from package you use this command:

pkg_add -r


pkg_add -r xorg
pkg_add -r nano
pkg_add -r curl

I like pkg_add -r because it fetches the package for me and installs it. It does that by looking at your system's uname (I think) and forms a URL to FreeBSD's software repository. The problem is that after a while repositories are taken offline. So if you are using some older version of FreeBSD and trying to do a pkg_add -r it will return that it can't find the package.

The thing is, the package is probably still around but in a newer form. The newer package is compatible with your version if you can find it. FreeBSD makes a statement that says packages in the same stable branch will always be compatible.

To force pkg_add -r to bend to your will you edit the PACKAGESITE environment variable. If you want to force pkg_add -to download FreeBSD 7-STABLE packages, set PACKAGESITE to

Be very careful though that you point to the right URL. I actually pointed to the wrong URL and downloaded 32-bit packages instead of the amd64 packages which I needed:

Tuesday, May 12, 2009

Ubuntu: Getting files off your phone using bluetooth

Sometimes you just want to get a file off your phone. One of the easiest ways to do this is bluetooth. The problem is that some carriers (such as Verizon) lock their phones down so you can't transfer files from your phone to your PC over bluetooth.

The real technicality is that while the phone can't initiate the sending of the file, you can still browse the phone for files and pull them.

So far the most useful tool I found to get files off my phone (which is an LG Chocolate) is obexftp.

To get started install the package using Synaptic then open up a console and type:

$ obexftp -b

When you find a device you want you connect to it like so:

$ obexftp -b 00:1E:75:ED:62:2C -l
Browsing 00:1E:75:ED:62:2C ...
Channel: 7
Receiving "(null)"...

The way obexftp works is that you basically have to run the command over and over with different options to get what you want. So after running it with this "list" option I say I want to browse the folder "MyPictures".

$ obexftp -b 00:1E:75:ED:62:2C -c MyPictures -l
Browsing 00:1E:75:ED:62:2C ...
Channel: 7
Sending "MyPictures"... done
Receiving "(null)"...

Now that I know the location of the file I can procede to actually get the file.

$ obexftp -b 00:1E:75:ED:62:2C -c MyPictures -g 0512091207.jpg
Browsing 00:1E:75:ED:62:2C ...
Channel: 7
Sending "MyPictures"... done
Receiving "0512091207.jpg"...|done

Easy enough. Probably not the most intuitive way to do things but it gets the job done.

Monday, April 27, 2009

Obtaining a Free Fully Trusted SSL Certificate

Ever get that popup when you visited a supposedly "secure" website that says the certificate is not trusted? Reason you get that popup is because the person who signed/issued the certificate is not trusted by default on your computer.

The business of selling certificates is a lucrative one. Large certificate authorities such as Thawte and VeriSign make a killing issuing digital certifcates. They can arbitrarily adjust the parameters on the certificate and charge for it on a case by case basis. The reason they are able to do so is because almost every computer by default trusts these companies. You can check this in Windows by going to start, run, mmc.exe and adding the Certificate snap in. Look under Trusted Root Certification Authorities.

In general you have to pay quite a bit of money to get a certificate signed by one of these authorities. However there is one way to get a virtually "free" one.

Thawte offers a program called their Web of Trust:

Basically after going through the steps Thawte will give you a free certificate signed by an authority called "Thawte Personal Freemail CA". Because this CA is installed by default on all Windows installations it is trusted everywhere.

Getting this certificate involves a combination of sending in documents to Thawte or finding people who have already gone through the process to vouch for you. Once you have it though you keep it forever (as far as I can see) and can generate certificates with your e-mail address in it.

Wednesday, April 15, 2009

PyXMLSec Windows Binary

Xmlsec Python Bindings for Windows

Getting the python xmlsec library to work on windows is a bit of work though it really shouldn't be. Provided here are windows installers for xmlsec. The installer itself is produced by python distutils.

In order to get it to work you will also need libxml2. I am providing a copy of it here as well.


I used MinGW for compilation. The entire environment I used to compile pyxmlsec is available here. Look inside the pyxmlsec directory for build.bat.

Thursday, April 2, 2009

Eye Fi Standalone Server Version 2.0

I'd like to release an updated version of the Eye Fi standalone server in python that I have been working on. This version should work on Linux, Mac OS X, Windows, Solaris, or wherever else you can load a Python interpreter. As always I love comments so if you are using this feel free to e-mail me or drop me a note!

Source on GitHub:

Download (zip):

I know some people just like to browse around the source without having to download stuff (I'm one of those people):

This new version has the following features:
  • The server can now execute an arbitrary command on each uploaded photo. This is a very dangerous feature and should be used with caution. On the other hand it is also very cool. You can have the server FTP files, display them using an image viewer, or even run sorting programs on the images.
  • Improved security: the server now generates its own nonces instead of using one that was hard coded. The nonce is based on the random library provided by python. The INTEGRITYDIGEST field is also checked.
  • Ability to read settings from a configuration file (there is a included DefaultSettings.ini for reference). The file allows you to configure the listen port, console output, logging, download location, and execute on upload, and upload key.
Some other notable improvements but not really features are the addition of regression tests and support for Python 2.5. The regression tests are interesting since I run them against the official Eye-Fi Manager to make sure my behavior is a close match.

Getting usage information as to how to specify a configuration file:

C:\EyeFiServer\Release> -h
Usage: [options]

-h, --help show this help message and exit
Path to configuration file (example in

Actually specifying a configuration file:
C:\EyeFiServer\Release> -c DebugSettings.ini

Tuesday, January 27, 2009

Converting LDIF/LDAP data into a CSV file


I work with LDAP on a regularly basis. I frequently have to pull data using ldapsearch. While the data that ldapsearch spits out is a decent representation sometimes I want something a bit easier to work with.

The format I use most tends to be CSV. While the acronym stands for comma separated value the format may be used to describe data that is printed out line by line and separated by anything you can imagine.

The problem for me is finding a decent LDIF to CSV converter. The ones that I have tried tend to choke on any number of issues. Some of these include binary data or failing to normalize the multivalued attributes.

I finally got sick enough of these issues that I decided to write my own. You can download it here:



Update: Pushed the code to GitHub:

Hopefully the Windows binary works for you. If it doesn't then just download Python 2.6 and run the source manually. I am providing the binary just for convenience.

Running "python" should give you the usage text.

usage: [options]

-o <filename> : File to write output. By default this is set to sys.stdout
-l <filename> : File to write logging output. By default there is no logging.
-F <char> : Character to separate the fields by. By default this is a comma. i.e. -F","
-D <char> : Character to delimit the text value by. By default this is a double quote. i.e. -D"""
-M <num> : The maximum number of columns a multivalued attribute should take up (default: 5). This is common with the objectClass attribute where it can have over 20 values. Do you want to have 20 columns each with the same heading objectClass or do you want to limit it.

Here are some common command lines that I use (assuming you have a test.ldif):

Outputs the CSV straight to standard output:
python test.ldif

Outputs CSV to standard output with semicolons as the delimiter:
python -F";" test.ldif
Outputs CSV to standard output with pipes as the delimiter and text surrounded by carrots:
python -F"|" -D"^" test.ldif

Wednesday, January 21, 2009

Downloading the Global Address List from Outlook/Exchange

Getting the Data

Recently I had a need to dump the entire Global Address List from within Outlook. Usually the address list will contain the contact information for your entire organization.

The most effective way I have found to do this is a straight LDAP search on the Active Directory and pull everything into an LDIF file. Conceptually it is straightforward but I have found the most difficult part is usually trying to get all the pieces together. So subscribing to the French idea of "mise en place" here is what you'll need:

Your username is your fully qualified distinguished name. It is not your Windows login name. It commonly looks something like "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com".

The easiest way I have found to obtain your username is to login to a Windows machine and then put the following two lines in a text file with the extension .vbs:

Set objADSysInfo = WScript.CreateObject("ADSystemInfo")
result = InputBox("Active Directory Username (copy and paste as necessary)", "Active Directory Username", objADSysInfo.UserName, 100, 100)

The password will be your domain password.

Domain controller hostname
Couple of ways to get this. This will be the Active Directory/LDAP server that you will extract the entries. An echo %logonserver% at the command prompt usually works.

Ldapsearch tool
Easiest way to get the tools is from Sun's iPlanet Directory SDK. Go to and select the iPlanet Directory SDK downloads. After download the zipfile find the executable named ldapsearch.exe.

Performing the search
Here is an example syntax to extract out all the users:

ldapsearch.exe -b OU=Users,OU=USA,DC=example,DC=com -h -p 389 -D "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com" -w - (objectClass=*)

ldapsearch -b OU=Users,OU=USA,DC=example,DC=com -h -p 389 -D "CN=Tchang\, Jeff,OU=Users,OU=USA,DC=example,DC=com" -W -x "(objectClass=*)"

The -b is for the base. In this example I knew the users were all stored in that branch. -D is for username. -w - (that is a dash w followed by a dash) means to read the password from the console. On Unix the big -W is to prompt and the -x indicates you want simple authentication (cleartext username/password).

You might want to add a " > output.txt" to send the output to a file. If you do that you will have to supply the password on the command line.

Sunday, January 4, 2009

Eye Fi Standalone Server

Eye Fi Linux Hacking

So I spent some time over my vacation learning a bit more about Python. What better way to learn a language than to implement something you want or need, right?

I am releasing a standalone Eye-Fi server written in Python. Basically I saw Dave Hansen's post ( and went ahead and did it. This software works on Windows with Python 2.6. I have not tested it on Linux/Unix yet but I assume it will work seeing as how it is written in Python. Please let me know if you try this software out and it works or doesn't. I personally would love any comments!

Python script is here:

General Architecture Notes

This is a standalone Eye-Fi Server that is designed to take the place of the Eye-Fi Manager.

Starting this server creates a listener on port 59278. I use the BaseHTTPServer class included with Python. I look for specific POST/GET request URLs and execute functions based on those

Currently all files are downloaded to the directory in which this script is run.

To use this script you need to have your Eye-Fi upload key.
It is in C:\Documents and Settings\[User]\Application Data\Eye-Fi\Settings.xml

Simple search for "eyeFiUploadKey" and replace it with your key.


Update (4/4/09) -