Matt's Linux Blog
My linux problems (and their respective solutions)Sunday, May 22, 2011
Dancing with Erlang
So I decided to create a highly-available IRC bot in Erlang. This is probably overkill, but I was mesmerized by the prospect of on-the-fly code reloading, one of the major features of the OTP stack.
Some "gotchas" that I came across:
- The vocabulary: the terms "release" and "application" are a little skewed from how I originally conceived them. "application" really means a piece of your high-level application, something I would normally call a "module" (but that itself has a meaning to Erlang developers), and a "release" is what the high-level application really is — it's called a "release" just because it specifies versions of each of the constituent "applications".
- What you actually name the .rel file means nothing, aside from naming the .tar.gz file that systools gives you. Hence, I can actually let git manage the file — I was afraid I was going to have to copy it for each and every release I wanted to push out.
- Making sure the latest BEAM file is loaded: it's easiest just to use the shell's "c(modulename)" function, which compiles and loads the new version for you, or exit the interpreter completely. Trying to use code:purge, code:delete, and code:load_file just gets awkward and annoying. I don't suggest it.
- Creating the relup file to deploy a new version of an application requires both versions of the application in the code path, even if the appup file for the new version tells the whole story and only reloads functional modules. This is somewhat interesting to integrate into a workflow with Git; I'm still working on that part.
- Creating a system instance to deploy to is a pain in the butt, especially if you want the processes to run in the background. I ended up mostly following the Erlang System Principles guide, but I did most of the things manually instead of copying their script — I learned a lot about the target's filesystem structure.
- Remotely accessing an Erlang node is really really picky about the name you use to access it.
Biggest lesson: Erlang is an operating system by itself, with its own idea of package management and directory structure. I had to throw away what I normally think of as packaging to gain the power of the Erlang release handling system.
There is more on most of these points to come.
Friday, January 9, 2009
Windows 7
I’m posting this from a new Windows 7 installation, using Windows Live Writer. I really don’t have anything else to say right now.
Thursday, May 1, 2008
Moving my RAID array to XFS
I've got a 1TB RAID 5 array, which I use for backups. I use a script I made based on rsync, which pretty much mimics Time Machine functionality. That means in /storage/backup-lithium, there are directory entries for about 30 copies of my laptop's filesystem tree, even though most of them point to the same inode (i.e. data is shared between them). I've been using ext3 for this storage array, but I noticed it's really slow for searching through that crazy amount of directory entries, so I decided to try copying all my data to XFS to see whether it was any better.
Since I use LVM, this was pretty painless. First I shrunk my ext3 partition, which took a couple hours:
Then I created a new logical volume:
I copied all my files from the ext3 to the XFS volume, which took about 22½ hours:
Now that I had two filesystems equally populated with directory entries, I ran bonnie++ and got very similar benchmarks on both volumes, suggesting that the change didn't really affect raw speed much. Then I tried something that would have to traverse the directory tree, with pretty drastic results:
The time it took to determine the total size of that folder went from 56 minutes to 12 minutes. To be even more sure of that result, I ran it again, giving me the same numbers within just a few seconds difference. I also noticed that XFS seems a bit more efficient at allocating files, since I confirmed that the contents of the directories were the same.
All in all, I think I'm going to mount the XFS volume as /storage and let the change be pretty much transparent. I've already checked to make sure that there were no files on /storage that were modified during the move. I'm not going to wipe the ext3 volume quite yet, since I still have plenty of space in the XFS partition — I'll do that when I actually fill up the disk.
Since I use LVM, this was pretty painless. First I shrunk my ext3 partition, which took a couple hours:
# e2fsck -f -C 0 /dev/storage/main
# resize2fs -p /dev/storage/main 400G
# lvresize -L 400G storage/main
Then I created a new logical volume:
# lvcreate -l 100%FREE -n xfs storage
# mkfs.xfs /dev/storage/xfs
# mkdir /xfs
# mount /dev/storage/xfs /xfs
I copied all my files from the ext3 to the XFS volume, which took about 22½ hours:
# time cp -av /storage/* /xfs
Now that I had two filesystems equally populated with directory entries, I ran bonnie++ and got very similar benchmarks on both volumes, suggesting that the change didn't really affect raw speed much. Then I tried something that would have to traverse the directory tree, with pretty drastic results:
# time du -sh /storage/backup-lithium
90G /storage/backup-lithium
real 56m40.618s
# time du -sh /xfs/backup-lithium
86G /xfs/backup-lithium
real 12m0.409s
The time it took to determine the total size of that folder went from 56 minutes to 12 minutes. To be even more sure of that result, I ran it again, giving me the same numbers within just a few seconds difference. I also noticed that XFS seems a bit more efficient at allocating files, since I confirmed that the contents of the directories were the same.
All in all, I think I'm going to mount the XFS volume as /storage and let the change be pretty much transparent. I've already checked to make sure that there were no files on /storage that were modified during the move. I'm not going to wipe the ext3 volume quite yet, since I still have plenty of space in the XFS partition — I'll do that when I actually fill up the disk.
Labels: ubuntu
Thursday, April 24, 2008
LDAP Authentication in Ubuntu 7.10
I recently acquired a file server machine, and I wanted to be able to use NFS to share files between it and my laptop. To do so requires that all users on both machines have the same numeric user IDs. I have a bit of experience with LDAP, so I decided to go for it.
I set up a simple LDAP tree under "ou=elements,dc=mmlx,dc=us", storing user accounts in "ou=People" and groups in "ou=Groups". I used "uid=username" as the RDN for users, and "cn=groupname" for the groups. Users are all of objectClass person and posixAccount. I made a group (objectClass groupOfUniqueNames) for each machine, containing the users I want to be able to log in. I also made a posixGroup called "ldapusers", to which all users belong.
Since one of my machines is a laptop, I set it up as an LDAP mirror, so that I can authenticate against localhost even when I am not connected to a network.
I installed libnss-ldap and libpam-ldap. I put the following in /etc/ldap.conf, and symlinked it to /etc/ldap/ldap.conf:
Then, in /etc/nsswitch.conf, I changed the "passwd", "shadow", and "group" lines to:
At the top of /etc/pam.d/common-account, I added:
At the top of /etc/pam.d/common-auth:
At the top of /etc/pam.d/common-password:
Then I tested the NSS configuration by using getent passwd someuser, where someuser was in LDAP and not the local machine. As long as that responds properly, you're good!
I used the following sources for this:
Making a Debian or Ubuntu Machine an LDAP Authentication Client
Re: [pamldap] pam_ldap for groupdn access control only?--Got it.
Need HOWTO for Ubuntu as an Open Directory client (LDAP/Kerberos/XServe)
I set up a simple LDAP tree under "ou=elements,dc=mmlx,dc=us", storing user accounts in "ou=People" and groups in "ou=Groups". I used "uid=username" as the RDN for users, and "cn=groupname" for the groups. Users are all of objectClass person and posixAccount. I made a group (objectClass groupOfUniqueNames) for each machine, containing the users I want to be able to log in. I also made a posixGroup called "ldapusers", to which all users belong.
Since one of my machines is a laptop, I set it up as an LDAP mirror, so that I can authenticate against localhost even when I am not connected to a network.
I installed libnss-ldap and libpam-ldap. I put the following in /etc/ldap.conf, and symlinked it to /etc/ldap/ldap.conf:
host 127.0.0.1
base ou=elements,dc=mmlx,dc=us
ldap_version 3
bind_policy soft
pam_groupdn cn=beryllium,ou=Groups,ou=elements,dc=mmlx,dc=us
pam_member_attribute uniqueMember
pam_min_uid 1000
pam_password md5
nss_base_passwd ou=People,ou=elements,dc=mmlx,dc=us
nss_base_shadow ou=People,ou=elements,dc=mmlx,dc=us
nss_base_group ou=Groups,ou=elements,dc=mmlx,dc=us
Then, in /etc/nsswitch.conf, I changed the "passwd", "shadow", and "group" lines to:
passwd: files ldap
group: files ldap
shadow: files ldap
At the top of /etc/pam.d/common-account, I added:
account [ authinfo_unavail=ignore ignore=ignore success=ok default=bad ] pam_ldap.so ignore_unknown_user
At the top of /etc/pam.d/common-auth:
auth sufficient pam_ldap.so
At the top of /etc/pam.d/common-password:
I had read that adding references to pam_ldap.so in /etc/pam.d/common-session was also required, but I ended up commenting that out (unfortunately, I did that a while ago and don't remember why).
password sufficient pam_ldap.so
Then I tested the NSS configuration by using getent passwd someuser, where someuser was in LDAP and not the local machine. As long as that responds properly, you're good!
I used the following sources for this:
Making a Debian or Ubuntu Machine an LDAP Authentication Client
Re: [pamldap] pam_ldap for groupdn access control only?--Got it.
Need HOWTO for Ubuntu as an Open Directory client (LDAP/Kerberos/XServe)
Tuesday, October 23, 2007
D630 Gutsy 2.6.23.1 AMD64 Packages
Seriously, the title of this post should contain all the information anyone would want to know except for, maybe, the URL. I've built packages from the official release of 2.6.23.1. Here they are: http://www.owlnet.rice.edu/~mmullins/packages/linux-headers-2.6.23.1-mokomull_2.6.23.1-mokomull-10.00.Custom_amd64.deb
http://www.owlnet.rice.edu/~mmullins/packages/linux-image-2.6.23.1-mokomull_2.6.23.1-mokomull-10.00.Custom_amd64.deb
I don't have any of the Ubuntu "restricted" drivers built, since the only one I need is ndiswrapper, and I do that by hand. If you need nVidia modules, well, I'm not very qualified to help you. Integrated graphics ftw.
Oh, and hi Dustin :-D
http://www.owlnet.rice.edu/~mmullins/packages/linux-image-2.6.23.1-mokomull_2.6.23.1-mokomull-10.00.Custom_amd64.deb
I don't have any of the Ubuntu "restricted" drivers built, since the only one I need is ndiswrapper, and I do that by hand. If you need nVidia modules, well, I'm not very qualified to help you. Integrated graphics ftw.
Oh, and hi Dustin :-D
Thursday, August 23, 2007
D630 AMD64 Ubuntu Kernel Packages
Okay guys, I've finally made AMD64 kernel packages that fix the sound bug on the D630. They're available at http://www.owlnet.rice.edu/~mmullins/packages/
Thursday, August 16, 2007
Huge Ubuntu Kernel Packages
For all you D630 (and possibly D830) users running Ubuntu, I've been trying to make a kernel package that incorporates a working sound driver. I've been building packages with the stock Ubuntu configuration options, pared down a little bit (omitting IDE drivers and things that we'll never use), and I still end up with over 250MB of kernel images, which is about 80MB compressed! I can't figure out at all how Ubuntu kernels are compiled to have only 79MB of modules, uncompressed, giving a 17.6MB package. I'm *not* going to post an 80MB kernel package, so until I figure this out, you'll have to wait or DIY. I'll have access to webspace when I get to Rice on Sunday, so that's where I'll post what I build. This will all be unnecessary when 2.6.23 comes out, since the patch has already been committed.
Archives
May 2007 June 2007 August 2007 October 2007 April 2008 May 2008 January 2009 May 2011
Subscribe to Posts [Atom]