Saturday, November 28, 2015

Troubleshooting rtorrent: Storage error: [File chunk write error: Invalid argument.]


So I decided to try and improve the performance of one of my VMs that was running torrents.

Instead of using samba to mount the filesystem from inside the guest I went with QEMU / KVM's v9fs.

The filesystem is actually called 9p or Plan 9 File System:

https://www.kernel.org/doc/Documentation/filesystems/9p.txt

After mounting and figuring out permission issues I started rtorrent and got the error

[File chunk write error: Invalid argument.]

Apparently rtorrent uses mmap and the Plan9 file system kernel module did not support memory mapping. However it was added in a later kernel release. So all I had to do was upgrade my kernel (I was on the default 14.04 LTS kernel):


sudo apt-get install linux-image-generic-lts-vivid linux-headers-generic-lts-vivid

Restarted and used the following mount command it all worked:

mount -t 9p -o trans=virtio,version=9p2000.L,cache=mmap,rw /mount/external /data

Before the cache=mmap did not work (only cache=loose or cache=none)

Monday, April 13, 2015

Making your Windows KVM guest boxes fly with VirtIO



In order to extract the maximum performance out of a Windows guest OS on KVM / qemu you should install VirtIO.

What is virtio?

There are quite a few articles about it but basically it is like the VMWare Tools for KVM. It is a bunch of libraries that basically speed up your guest OS by making it more efficient to communicate with the host.

What speed ups are we talking about? Mainly disk I/O and network. Without Virtio drivers installed your Windows guest will feel like molasses. Or more like the difference between traditional hard drives and SSDs. It will still work but you won't like it.

What tripped me up about install the drivers was that everywhere I looked it said to download them from the Fedora project here: http://alt.fedoraproject.org/pub/alt/virtio-win/stable/.

The problem is I was using Ubuntu so while you can install the drivers it just blue screens on you with a giant STOP error. Basically I was using the wrong drivers.

I am on Ubuntu 14.04 LTS so what I really wanted was this: https://launchpad.net/kvm-guest-drivers-windows/+download


The overall technique is setup a device that uses virtio in the libvirt configuration file for your guest. This is an xml configuration file you access by typing virsh and then "edit <name>".

After you do this you will want to specify a specific network card or disk that has type="virtio". This will cause Windows to detect a new type of device in Device Manager. You then update the driver by using the downloads found on launchpad.

If you are lucky you will see the following:


Both should say "packaged by Canonical". These are the major drivers you will want. The other is a serial driver and then a balloon driver. The balloon driver is for dynamic memory management.

If you are wondering why you don't need to install these drivers on Linux guests it is because linux guests have it compiled into the kernel by default.



Saturday, April 11, 2015

Generating your own GPG key on Windows



Generating your own GPG key on Windows


End to end encryption is important. One of the critical pieces is having your own public/private key pair. Here is how you generate one on Windows.

Download gpg4win here:

http://gpg4win.org/download.html

The latest release was Gpg4win 2.2.4. Go ahead and install it making sure you also install Kleopatra (a tool for managing keys and certificates).

Start Kleopatra and go to "File ---> New Certificate", Select "Create an OpenPGP key pair". Everything you put here will be publicly visible. Click on Advanced settings and make sure RSA 4096 bit is selected (stronger crypto) and select Authentication for Certificate Usage (as well as the others). Select a passphase to protect your private key (it is not needed but strongly suggested).

You should get along with a Fingerprint (yours will vary)

Certificate created successfully.
Fingerprint: 6E18EAD1C783F2B2964A949E9F4567CF161EA82D

At this point you should Make a backup of your keypair and upload the certificate to a public server. I personally use the MIT keyserver. After this you should public your PGP keyblock somewhere no one else can access and call it your own so other people can send you encrypted things. Example here is mine:



-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2

mQINBE6bcvsBEADaniL3XIia0PrCKMiNs7HFtI+YpVcxJKxMPVXwtuCY23i8knaU
apzb6L37uvt8pvMLlZbP84yXR+WATEfk2XO9HgvrDyVuLT1Fw584vbJ973t4Bg6H
lOeD/IEZoxIwjXGE68RMrPiwZCxv76AmC4RFv5nPBVAUFMyrMrQsutA1wfMcuCGn
US2QZaqjkItafbyi7yNfT9Ytrh5EZr6qFa/ABMg+UAJFgqvBgIcH3I8QfnI2s0O+
BK4h0DBlnBPPDJfKTxHuNK52gVFsys/Lba1tW8gnDF8aVV9Zw80xFndzX5ygruKx
yQBmmB/jxCEQv3zlO4uIiom28hNqxAkGI+Oucha0zt6/pY5+qm+GIL17/p/v7gxV
BHJ7YzJ4mrbMsZtJn7bclxVtbjzfGfwx/gmE9FNEUcP/Pfgd9yf4W23NgaplkmZ9
vD7zze1bVDq5MUNbrPdpwScvCoNKkTH4qRc3he9yR2nGSL9wgZdQIJnck0gr1yca
Dpog8f3Dq7dkcfQlmH5AQemcNNaXfdeSZFwk/VbWGwLnvb0v+OhpEtc831PapQgQ
t5JfUgebhc1xprhqOUdbdupo3uj+zsAquIZtOcHWuRpx1m2UGq95k7Qa9poGhbIp
QPYrPLLKlUzhZkbGD1CDFLwhUIOihr4jMHVhitCvK4pm6THw+bjU9yCgWQARAQAB
tCNKZWZmIFRjaGFuZyA8amVmZi50Y2hhbmdAZ21haWwuY29tPohGBBARAgAGBQJO
m3smAAoJEGTZD4idpdxl0nYAn1lJovJPDUN4IL4wJjkWUZgYwf0RAJ9t7HQE6Iux
1g2ZF6C5+eAV0643/YhGBBARAgAGBQJOm3wRAAoJEPTbJShaRqMbAA4An391L7Ra
p7RftgFFfJ4pFASaZLg9AJwJ66kA5pOXQ+glMjdwxmYDE7UHv4kBHAQQAQIABgUC
TpuK5QAKCRCpcHK63mj87EIBCACKaq131z5Ta4SiR54fO9HrM13hzeZLe0Qsozyo
M/IhxNve+SXwvZyOIMRSXnyDpkYAWcHwwuJAbq0CsgAY/nRK6BT77A3vwMCdwp2I
7VoGnWFuNHCzNq3Dibw+tr3QG2Gbb3VTOgGe08ZgwUVdv5a1Zmhgr+ERYyc2EnWn
9tN24nsFUpq5oQhJ92t1nANkY2xbJ1oFwW/J2BOEq0/5OXQEUKGlUQk02ZKr7yXo
LCYVRWCeDZ9e+sPOKqh+Q8st0Mz/wTMUXFddtbHAGsPRf3EGWyKrJJpm/mg52DG3
COKX+JIJFNbVBOCRqZ0Yg99Md7TN2DqxugNQyiazbhwNi8lJiQEcBBABAgAGBQJP
SuuIAAoJEEJEGMpLL4ygwskH/23zxlI0joeSGGcbqZw3EQZd4JeMW/WiXRcLQ81K
NeKK+bbE2nffx07+eSrLqN+6KAFvfXZp/xFVS+8TazOFYpSWn3TMgPekviTgkYR5
vMJnRACGHSwuVaR7yKGjYzkaOdat/IHtYIuWNFoLCaEyPWTT+EbCNzQMKCvTJ4wv
aZ/DnmGFODlwgoRYjhcHlElw4unkhmiHDCiAH7PVlLnD31wJsYJHI5X4PtWYAj5q
j2rJpl2/yc8mR0ultI80aNOHiWr0tP9we3rusVDBqGUePiUfMEARkJTdcznI5YXM
9n2RmnktdmEtucvpDRYQ7avc/Bwhp7ha4vc8jeRsl6cyy8qJARwEEwECAAYFAk6b
eo4ACgkQ2I7wYRrvkPQCSwgAxWlTMaDAWTJjxcRQlZkJchbOdOhWkvj9kdT34sNZ
9weIPoDVy/PB7BtPnPPEffR0mbA5GmZ12mzHpTfl8IPnG23M2hD7CFedpAtfCayX
pZ8sMN6qQvoL3J/kuRxxEI/7Nvbq2ysbG5HLHgOGQs590Yt3meqHuuVJ0BXJUN/L
Z3xCQvdku0vlcBglnXc8pw1uDWgPhqlJOajxEd8KdZuQu0g6wkXtCcdepBxpP8KM
cJRpG+L17b6E8fnTu5tBdzhoDBMCLm3u0SXK70+f0e0oVoXn1yISgIXG/4NaOT18
jk3ndJvqp84/f8qvussv+P2S0BExDKUi8waXtNT/xgY2t4kCHAQSAQIABgUCTpt7
gAAKCRC1dIlngK8H0+TOD/9oYGO+0/Vjsvq6RKux9l0h4fjr5mMU9wni9GQGOPwn
9FuxxF9WiXTQuDMpikX/us+SoaKtzuphMHzoEVXx2zhDB1UgBO5MUOfuW67st6WB
gKdwctmMDHZf5KqoeavklQjOLBt5YLyIL1w7gVf+tp/MbVCZ2gRfzgjJIZwlpg53
bGo06cTurLOnf+IhsqbwftE6bGODQFSAh0MVbA9yd6Wc1DEaZi6MRqNdAyRM9QLt
+CP99NPmvXATXipnuhs8/sFeN9QFdCd6l2KSC+Lpy3/pBxvBSdbfKRZbmNFDaker
1xQqHd32QUROIgsN5NAlOVlBtWJHFNh4fcWApvqt+BCU4MVJXMF0WkaPUg5m8KlO
/gGvRJeDkwQjufNFIqe6UB6zRyCk3N+hvIkh+Iq9DAqbHSXdGd3m7xhQVAQG9YF+
w7J3VUjVtQAw/rDrt74LjgsevRMbsiFvCov6KFwu/BRhd4HtYJucShGMM01jRL/D
ixcGNkU0KiBxFjL2GN5HauDuKEbmZ2/LHfiaCIc5b7IQyhyF2jn0Ec8ZMjaidP09
48YkhikzWUE7R4ACBXY22er0iGswL2t4J8IaklFY4E4Kl0jPYjFCf7PbrKtg6ckL
63M5yNK91pgnqM3ZOz0jJ8i0tiXGAXiyrakNitjKRx6krdlK9F14ef0so8lvClXt
zYkCOAQTAQIAIgUCTpty+wIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQ
3CqBdPG7zTX7VhAAsyhg+5xqwDRVxu8Gg30AoqTZFDbPF7ZHQtGzULPkyqqMlhBu
KMwXl+qiA9JfJWHBr828NiQlY0BFszxoqnWQkb74VXs8fZ7aMlnjyoHaR16YquJh
wyLnJwKX3suC7E6tUlbW97ENsxJeLaYbs+XekqUbKWeDa4iMdMRkN0Pl7K6DiqYw
+FKBqpHDqLJiW6SAwwQMM3gzCDOEf2hUw9rKr8ZYeEH4dLmWHNYg9PExBeapkSj8
ZcxOIv9fMUvbThnXk0UgLa0tnNGnzg5wLSAzDIUgAicFHwN/w3iDmL6Q2Bl2YajR
5I0YKKXNtbbifZVhvJQRepdND69POHxQj2qdcVq7qyzerf+myc4rmfRmIwE5V++0
oBOYewpLVUwhWEjnjind53S/97tTbM6QVkune7yL9dFqCa73ZBR/4U9aLlMDcmRR
p6eHdCQR2JI0sVadFPp/nqYYIvbsdxrAvIy9uNhwPc6L/bM71YmJrjBaBeeaD1YC
UTbUKlfrwZSoLrW6AUP+3rfeB2KDTSkDIbJUCuMuA4E2ZRbFcK+WogfScOw9iRkN
lrUjkVBSfdKx+aC6xUbh/lD77s3roL9D3p4dtWVqhsq2pk75EaMDYJ0Rz2kpzpkt
J3gPxK+Ib/0AOdxF6MvURNqlgZ5BR8NzV5SYqCuR7UXUZ01ky41Qu5XZ8z+5Ag0E
Tpty+wEQAPZuxuhd/vbcM7901jNUQA7c/NOsXczxS9o5JDqlwHu53AHujnBf6Ivv
HOdxvd+4t4HFA9i6V8xcZnngpLhC0oh939WLVu5ABJfvExKXYfYoaUdg/HeSePxU
b5tdahLHq4tJZEgt+5ZsNdejUbLSzQCA0SeeC+ENohYCqyrErZXV63pi8/D0bjTQ
1K3xDUhvSM0W6X6N7L74oV9V8eLmT9KNGuPYlNmQMxuKMO3FjM0wCyDJsjSgz+m+
n5CSFlltJYgJXt2ZGQfUuMJGbEfwOfsh8mgyqLXlH4erEin1BDpUqx9zwgg9HQWo
XXyabZRAAmhOY3Kfu2t9o7t523PfHn6RwH//YIa14+Z2QE9a074CSdaNOYqKp5Yd
xZxDBPw86/rq3KhsdruhXVZhYiTon2HNoQl8G9LFGm4XZN5eIq6GT+coun0824Bp
jrl13MtN0xxFBSqHRdAfRER2uta4COm03Yz/hwnIQxGx1T4cjWStRb7I26RBXSxh
K/RG489y3MCjPyP8gVv4gz5UUAlxkMl7a38vYDVZoqwAJZGI0+NoS7N149HPBYTS
WQXo1UgNtVtos1kgBKz01iq3/Ttb7iGeKMlEHeRYEyR0XpyJYFwCqEcwtdYrAF9n
GMPg6VK+5WfJyEhWtOvfCUmRaa8SZf6faqQEP8JXYzzk5Ri17jhJABEBAAGJAh8E
GAECAAkFAk6bcvsCGwwACgkQ3CqBdPG7zTU+ExAAgA5Gd7XS+NRzNmBFDWrLuQRn
yHX+iR9lEb+VrSAQWA3MbY1W5Fp9dfmPSG4+i/9OOVhmhkefCwYJit/0sGrL2fuP
LrHevLhQKNQj4wVZXel1RIBjsJH8zbdUCnqiBggcm2iEmp6DZT9/+uy2TqEOcOT/
AD9bC1v1jMS+rn8CykHjX+PHrZg2Sm7FwhX7qaFilub+q3ok6vKPUjilvVgduH/A
PDRgasjJYAWWtUcdYP6RYPRID5zWaDnPgX+8Lm3H/2s1rsifarOvpb2nN7r+rdUc
WJq3QUNrYSVEJ24PvXpDdIiMKW02LUuWYfpSFHqGOdjGY6xI+jCbmVSBu0O/Oe7n
uywHrWR9EtpLcBb0LvKDofQ5x6tMw2BnGJRtRXYxbJjV8qssUhG3Lkzu2g5bkXl5
A4vp2FzYrLCjXgFy69+hBypxxqp7Izb+eNGsiVmDNUC1SGRc1e6bKBPkBTdCP0GF
HjdqJuIm5ps1pj813mM7tPbTKJQi/Jm84U6YioDZfqJznBLsKrL5rLqpjDB6GaIZ
64GsY5Kwx469p7lg2v4S+yWM+72w2GV05q+tUpFgIYG312he1KSevFfW0+Anvs4p
kODqU0d1aJbqco56VPvikhTNy8jikha56Zti7SrdcXgH5dfnlpTqTVdZJAPeJO7O
RDHCLFVsnJ2guAkXEHY=
=FzKZ

-----END PGP PUBLIC KEY BLOCK-----

Friday, April 10, 2015

Backup a KVM host with CrashPlan and LVM snapshots


Backups. Everyone's favorite topic.

You've heard it before. You should back your shit up. I'm sure there are a number of reasons why you aren't backing things up.

I've recently been managing a box with a bunch of virtual machines on it. The virtual machines are of the KVM/libvirt kind so you manage them with the command line tool virsh.

The question is how in the hell do you get a "good" backup solution. I came up with this solution but am welcome to hear how you others do it as well.

I signed up for CrashPlan. Mostly because they support Linux and they are relatively cheap. BackBlaze is good but does not support Linux so they are out.

The general strategy is to "snapshot" the partition where you store all your KVM images and then backup that snapshot.

You must be using LVM. Most Ubuntu installs are using it by default. Doing a df -h you should look for stuff like /dev/mapper which is a tell tale sign you are using it:

root@example:/var/kvm# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/example--vg-root          832G   48G  742G   7% /

What I do is snapshot the partition with all the images on it. I then point CrashPlan to it and then run this script every so often:


This script recreates the partition so you get a fresh snapshot. Probably not the best situation but it works for me.






How to make a DHCP reservation for a kvm/libvirt host using virsh

It is very useful to reserve DHCP addresses for guest virtual machines. Here is a quick rundown of how to do it.

Run virsh

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

Once you are in virsh run net-list to figure out the list of networks you have. Usually you only have one as in this case. Go ahead and edit it.


virsh # net-edit default

This will pop you into your default editor. I changed mine to be nano. Edit the part that says dhcp or add it and make sure everything is correct. You can get the host mac by editing the specific vm's XML configuration file. To do that you can run the command "edit examplevm1" and just look for the MAC address.

<network>
  <name>default</name>
  <uuid>969e94f2-f6bb-4436-8195-51603fd79225</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:60:bc:01'/>
  <ip address='192.168.5.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.5.100' end='192.168.5.254'/>
      <host mac='52:54:00:11:11:11' name='examplevm1' ip='192.168.5.10'/>
      <host mac='52:54:00:22:22:22' name='examplevm2' ip='192.168.5.20'/>    </dhcp>
  </ip>
</network>

Now the most annoying thing is you need to make sure these changes take hold. You can reboot all the VMs (pain in the butt) or use this script to remove and reattach each VM's network interface. The VMs will lose connectivity for a bit though.



Wednesday, March 25, 2015

Downloading a Rackspace cloud vm and running it on KVM


Do you need to migrate a rackspace image and run it locally? Or perhaps you need to run it on another server.

There is a good article on how to run it on Xen here: http://serverfault.com/questions/394030/how-can-i-run-rackspace-images-locally

However my server is running KVM so I needed to make some small modifications.


The overall strategy:

  • Go to your rackspace cloud page and create an image.
  • Authenticate to rackspace's API using your api key generated on the web portal. This gets you an access token.

    curl -s -X POST https://identity.api.rackspacecloud.com/v2.0/tokens \
      -H "Content-Type: application/json" \
      -d '{
        "auth": {
          "RAX-KSKEY:apiKeyCredentials": {
            "username": "exampleusername",
            "apiKey": "1c9bde3d45804cce5bb24333bce81960"
          }
        }
      }' | python -m json.tool

    Basically the response comes back as JSON so you want to pipe it to the json.tool module of python to format it nicely.
  • Once you have your token put it in your environment.

    export TOKEN="d129535e3b6dcc4e4c3b0c809be341b8"
    export ENDPOINT="https://dfw.images.api.rackspacecloud.com/v2"
    I had to figure out where I took the image and what endpoint to use. This one is Dallas-Fort Worth.
  • Use the access token to get a list of images. One of them is the one you created:

    curl -s $ENDPOINT/images -H "X-Auth-Token: $TOKEN" | python -m json.tool
  • Generate a POST request to export the image to a swift container. This is basically your cloud files container name (in this case Container1):
    curl -s -X POST $ENDPOINT/tasks \
      -H "Content-Type: application/json" \
      -H "X-Auth-Token: $TOKEN" \
      -d '{
        "type": "export",
        "input": {
          "image_uuid": "11107aa8-3187-4e0d-abcd-4a5d0a4f084d",
          "receiving_swift_container": "Container1"
        }
      }' | python -m json.tool
  • Periodically call:

    curl -s -X GET $ENDPOINT/tasks \
      -H "Content-Type: application/json" \
      -H "X-Auth-Token: $TOKEN" \
      | python -m json.tool

    To see the status of the task. If it immediately errors you did something wrong. If it says processing then you are good to go. For a 6GB image it took about an hour while it was stuck processing. The way you know it is working is if you go to the web front end and refresh your cloud files you will start to see files pop up.
  • Use the swift tool to download the image from the server.

After you have the image some tools that will help you get it in the right direction to run:

qemu-img info server.img

Gives you nice information about the format. Example:

image: server.img
file format: raw
virtual size: 1.0T (1073905926144 bytes)
disk size: 945G

I then created a new server using virt-install:

virt-install \
  --name exampleserver \
  --vcpus=2 \
  --disk path=/data/rackspace/server.img,device=disk,bus=virtio,format=raw \
  --ram 2048 \
  --graphics vnc,password=examplepassword,listen=0.0.0.0,port=9999 \
  --accelerate \
  --os-type=linux \
  --noautoconsole \
  --network network=default \
  --boot cdrom,fd,hd,network,menu=off

The thing is though the server might start up but you will get lots of kernel panics because paths and devices are wrong. The easiest way I found was to convert the image into a raw format (it comes as something else) and then mount it. Then go into /etc/fstab and change it all around properly.

I also had to change one other file because it was looking for /dev/xvda1 I believe and that didn't exist. That's pretty much it.