Note: These instructions worked for me, but I can’t guarantee there’s not something I did by “muscle memory” and didn’t document. I repeated this process about 6 or 7 times, and each time found something new I’d forgotten to write down. It also stands to reason that these instructions may stop working at any point, for any reason – I can’t provide support for these instructions as by the time you read this I will likely not have a Linode any longer.

First, you’ll need a Linode, if you don’t already have one – any size will do, depending on what you actually want to do with it… but there’s no minimum Linode size for FreeBSD itself. Then next thing you’ll want to do is file a ticket with support – Linodes default to vcpus=4, which will choke FreeBSD at the moment. They won’t increase this setting of course, but they will happily lower it for you. Simply give them your Linode # and ask them to reduce vcpus to 1 and they’ll make the change. You can continue setting everything up, they’ll just tell you that the Linode needs to be powered off and restarted before the setting will take effect – which we’ll do when we go to boot FreeBSD anyway – and chances are by the time they respond you’ll be about ready to reboot into FreeBSD for the first time.

Configuring the Disks

Go into the Linode manager, and trash any existing configuration it may have generated for you, along with any disks. Create a new disk image, 64MB should be plenty, and make it ext2 – we’ll be using this disk to store our FreeBSD kernels, PV-GRUB’s configuration, that sort of thing.

Next, create a ~512MB or so raw disk, which we’ll be dd’ing a FreeBSD image over in a rather nasty fashion. Make it whatever size you need for your image, most FreeBSD “minimal” images weigh in at around 200MB or so, so 512MB should be plenty.

Finally, use up the rest of your disk space with another “raw” disk image, which is where our main FreeBSD installation will live.

Setting up Profiles

Next, create a new profile… I called mine “finnix” but you can call it “Linux Rescue” or anything you like. Set it’s kernel as “Recovery (finnix)”, set xvda to the “recovery – finnix (iso)”, xvdb to your ext2 kernel storage disk, xvdc to your temporary FreeBSD root (the littler one), and set the initrd to “finnix (initrd)”.

Create another new profile, we’ll call this one “FreeBSD”. Set kernel to pv-grub, 32-bit (might be called x86_32 or something), xvda is your kernel store, xvdb is your main FreeBSD disk, xvdc is your “rescue” FreeBSD disk.

Note: laying the disks out this way will have the unfortunate side-effect of having your main disk live on xbd1 in FreeBSD – there’s no way around this, the ext2 disk has to be disk zero for GRUB to run unattended.

Booting into finnix

Finnix is a “live CD” style Linux, which will work from a RAMdisk and not occupy or otherwise hold any disks, leaving you free to do things like dd images over them and such. Now we need a kernel – you have two options here. One, you can build one yourself on another machine, put it on a webserver, and curl it across; or two you can use the one graciously provided by Aprogas in this forum thread. If you decide to use his images (I did to bootstrap the process), I’d suggest mirroring them someplace yourself in case you screw up and need to do it multiple times.

If you’re building your own kernel, see the notes below – at the time of writing the default XEN kernel won’t work.

Now we need to put our kernel in place:

# mount /dev/xvdb /mnt
# curl http://path.to/kernel.gz | gunzip > /mnt/kernel
# umount /mnt

Now extract and write the image over your temporary root disk:

# curl http://path.to/root.img.gz | gunzip > /dev/xvdc

Sync (I’m not sure if this is necessary, but it’s a habit) and reboot:

# sync
# reboot

From the Linode manager or LISH, boot your FreeBSD profile. You should end up at the grub menu, so point it directly at your kernel (I’m led to believe chain loading the BSD loader won’t work under paravirtualization):

grubdom> kernel (hd0)/kernel
grubdom> boot

Because we bypassed the FreeBSD loader, it’s extremely probably FreeBSD won’t know where to find it’s root partition. We’ll point it at our temporary root partition:

mountroot> ufs:/dev/xbd2s1a

You should now have booted FreeBSD, and be able to login as root (you’re on the console, and if you’re using Aprogas’s image, there will be no password on root at the moment).

Installation

There’s a lot of gotchas in this portion, so try and pay attention to what you’re doing (and don’t rely on me). First thing to do is fdisk /dev/xbd1 and note the geometry, because for some reason the fdisk inside sysinstall won’t be able to tell it, and will set it to 0/0/0, which will cause sysinstall to blow up when you try to partition the drive.

Start sysinstall, and go into fdisk. Set your geometry as noted before (mine was: cylinders=2015 heads=255  sectors/track=63), and then use the entire disk for a Freebsd slice. Write your changes, then back out to the label editor. Create your partitions how you’d like them… for testing I just made a 512MB swap (not sure if I’d need it or not), and then used the rest of the disk for one giant partition. Set the mountpoints for your partitions with root as /mnt, because / is already in use at the moment. Write your changes out, no boot loader (I’m pretty sure it doesn’t work anyway) then bail out of sysinstall.

When you’re out of sysinstall, mount all your partitions under /mnt, because for me they wouldn’t stay mounted inside sysinstall. You might need to newfs them, I did sometimes and didn’t others.

Back into sysinstall, go into options and pick your OS version (any kernel you’re likely to build is going to have -pX on the end of it, which will not be a valid FTP path for installation), and then make sure you set the install root to /mnt.

Back out, and do a standard install, without clobbering any of your partitions. I can’t recall if you need to set the geometry again if you’re not actually writing anything out. Select your distributions and install them, if you’re using Aprogas’s images the network is already configured (DHCP). Let it fly.

Once you’re out of sysinstall, make sure all your partitions are mounted, then configure /mnt/etc/fstab the way it should be… it’s very likely sysinstall didn’t wind up making one, but if it did, it may not be sane – it may have /mnt as the mount points for everything, and you’ll want to set them back to /.

Also ensure that /mnt/etc/rc.conf is correct – it may not even exist. You probably want at least sshd_enable=”YES” in it. Finally, chroot /mnt, set a root password, and create at least one non-root account. Make sure it’s in wheel.

Exit chroot, unmount everything and reboot. At the grub profile again:

grubdom> kernel (hd0)/kernel
grubdom> boot

Then this time at the kernel’s “where the hell is my root partition” prompt (actual location of your root partition may vary):

mountroot> ufs:/dev/xbd1s1d

If everything went according to plan, you should be inside of a working FreeBSD installation at this point… but it won’t boot unattended. If your kernel has ext2fs compatibility, you can fix that from inside FreeBSD (otherwise it’s back to finnix). Either way, mount your kernel store partition under say, /mnt, then:

# mkdir -p /mnt/boot/grub
# cat > /mnt/boot/grub/menu.lst
default 0
timeout 5
title FreeBSD
kernel (hd0)/kernel
^D
# umount /mnt

Now PV-GRUB should boot unattended (sidenote, i have “timeout 5″, but for some reason grub instantly boots freebsd). I haven’t figured out how to pass the root partition to the kernel yet, so see the notes below about compiling that into the kernel.

Compiling a new Kernel

8.0-RELEASE kernels will boot and apparently function fine in Linode if they’re built with the XEN configuration. Anything newer will panic on boot because of the new x86bios stuff, which is called from atkbd, but Linodes don’t actually use the atkbd device (console functions fine without it)…

… so copy the XEN config to something (I called mine “LINODE”) and edit it. I commented out the lines for atkbd, aktbdc, kbdmux, and psm – though I’m not sure all that is necessary.

If you haven’t already got it, you’ll likely want “options EXT2FS” so you can install new kernels without shipping them off to another machine and rebooting into finnix. Finally, you’ll need to tell FreeBSD kernel where to find it’s root partition so I used:

options ROOTDEVNAME=\"ufs:xbd1s1d\"

Build and install your new kernel, noting that it’ll land in /boot/kernel. You need to get it over to the kernel store where GRUB can find it – if you have EXT2FS compatability, that’s as simple as mounting the ext2 partition in /mnt, and cp /boot/kernel/kernel /mnt/kernel. Unmount all partitions and reboot.

At first I didn’t have EXT2FS compatibility, so I had to gzip the kernel, scp it to another webserver, boot into finnix and curl it back the same way we initially got the kernel up.

Finally, at the time of writing you’ll be spammed with messages (approximately one every 10 seconds) about the “hypervisor wallclock nudging TOD”. It doesn’t appear to effect anything, so I commented this message out in the kernel sources (sys/i386/xen/clock.c:344 at the time of writing)… that’s probably bad, but that’s up to you.

One more boot, and the Linode should boot completely unattended, including when it falls and can’t get up and lassie needs to restart it. Which will happen. A lot.

So here’s how to do it yourself in a few easy steps… first, right-click My Computer and choose properties. Then, go to the Hardware tab, and pick Device Manager.

Navigate to your unknown device, double-click it and then pick the Details tab. Find the Hardware Ids entry, and look for the most detailed entry. My shitty SiS network adaptor’s is “PCI\VEN_1039&DEV_0900“.

Navigate to the PCI Devices database in a browser on an internet-connected computer. In my case, I’m looking for vendor ID 1039, so I’ll click “1″ and scroll down… and I’ll find the Vendor “Silicon Integrated Systems [SiS]“, which is to be expected. Click into the Vendor entry and look for the Device ID.

That should hopefully give you the correct Google-snacks to track down a driver for the hardware.

Our most recent setup has been a Linksys WRT54G, which has been rather crap in it’s duties really. For some reason it’s consistently dropping packets and possibly rebooting (but with the garbage default firmware, there’s absolutely no way to tell if it’s rebooted or not) so I decided to replace it with a FreeBSD machine again – at least temporarily.

Networking in fwaggle-land these days is not without it’s new challenges, bumped well up from “lolsecurityissue” to “must have, design requirement” is Universal Plug and Play, or UPNP for short.

The Lowdown on UPNP

UPNP-IGD is both a terrible idea and an awesome idea at the same time – or more accurately: it’s an awesome idea, but a terrible implementation. Basically in it’s “Internet Gateway Device” form, it allows devices behind a firewall to arbitrarily set up port forwarding and open those ports temporarily, completely transparently to the user. For computing, this is probably unwanted – malware can easily make use of it and your firewall’s essentially useless.

But for gaming on consoles, it’s a godsend. It’s the difference between an awful experience on a Playstation 3, and an event-free, enjoyable gaming experience.

Setting up Routing

For the purposes of this article (which is already long enough, thanks to my story-telling) I’ll assume you’re familiar with setting up NAT under FreeBSD. We’re also going to assume you’re using the PF packet filter from OpenBSD – if you’re not and you’re clever enough to work out setting up NAT with one of the others, figuring out PF can be done in an afternoon.

We’ll assume at this point that your machine is up, routing and translating traffic correctly and is protected by some form of firewall ruleset in PF. We’re also going to assume you have a working DHCP server and all your machines are able to connect to the internet in some form. Now we make the top of your PF ruleset look like this:

scrub on dsl0 no-df
nat on dsl0 from lan0:network to any -> (dsl0) static-port
rdr-anchor miniupnpd

Obviously interface names need changing (or you can rename your interfaces, like I did – which requires only a one-line change in /etc/rc.conf if you happen to swap cards out for a different card/driver later on, instead of grepping for fxp0 in /usr/local/etc/. Reload the ruleset in PF if you haven’t already:

pfctl -f /etc/pf.conf

Let’s explain what’s going on here. First of all, it appears as though PF might scrub at least partially by default these days – I’m not sure about this, I haven’t confirmed it, but my Playstation was complaining about fragmented packets not going through and some Google searching appeared to indicate the PS3 likes to do dumb shit like sending fragmented packets with the DF bit set. This rule helped immensely and removed all complaints from my PS3 once miniupnpd was enabled.

Adding static-port prevents PF from re-arranging port numbers on the WAN side as it makes sense, which might break stuff occasionally, but for our purposes it’ll break far more if PF re-arranges port numbers on UDP traffic – internet games on consoles are stupid. If Modern Warfare 2 sends game data from port 3658, to another host on port 3658, and the packet doesn’t arrive with source port 3658, the host appears to drop it.

Configuring MiniUPNPd

Now that you have an anchor set up for MiniUPNPd to add rules to, it’s time to configure the daemon itself. Copy the miniupnpd.conf.sample to miniupnpd.conf in /usr/local/etc, and then start editing it with your favourite editor. The important bits I changed are:

ext_ifname=dsl0 # the interface name of your WAN device
listening_ip=10.0.0.1 # the LAN IP of your router
secure_mode=yes # this is default, but I wanted to point out what a great idea it is
allow 1024-65535 0.0.0.0/0 1024-65535 # access control, but you can do it via PF anyway

secure_mode is important – UPNP-IGD specs allow devices to set up firewall rules for other devices, which is where the broken-by-design issues with regards the UPNP spec come into play. You almost certainly do not want to allow this, and I can’t for the life of me figure out any reason you would need such functionality.

Access control can be configured based on IPs, you can also firewall the UPNP service (UDP port 5555) so that your network doesn’t appear to even support UPNP for hosts that you’d rather not have it. Our network is configured to assign all consoles into a /24 of their own, and to only allow those IPs to use the UPNP service. You can be as draconian as you like with UPNP access control, either via MiniUPNPd’s configuration or just by firewalling the UPNP service.

Testing it out

The first step was testing the internet configuration in the network settings menu. After correctly allowing UPNP, I successfully reached “NAT Type 2″ as far as the Playstation 3 is concerned. Next it was time to test if a game would actually work, so I loaded up Modern Warfare 2 and tried it out. At first I had “NAT Type: Strict” which basically meant that I was boned, until I found the above gotcha about source ports. Adding a static-port keyword to my ruleset in PF and reloading gave me “NAT Type: Open”, and I was all set.

With our PS3s in their own subnet, I’m able to enjoy hassle-free internet gaming without sacrificing the security of the rest of our network.

So how do we fix this? The libpq client is unfortunately very temperamental about it’s access, and it doesn’t complain at all when something’s wrong, it just won’t work. It’s actually rather trivial to fix.

The .pgpass file

There’s already plenty of documentation about the .pgpass file so we won’t go over it too much here. Suffice to say you want it to look like this:

localhost:5432:*:pgsql:muhpassword

Change “muhpassword” to whatever your pgsql user’s (which should have superuser privs inside the database server) password is and you should be set.

This file should live at ~pgsql/.pgpass, which on my FreeBSD with PostgreSQL 8.1 installed from ports is /usr/local/pgsql/.pgpass.

Now, and this is the critical part that kept me scratching my head for quite a little while… this file must be owned by pgsql user and it’s permissions must be 0600. If it’s anything else, the psql program will ignore the pgpass file, and prompt for a password on a terminal that doesn’t exist (because all the input/output from the daily scripts is /dev/null). Once the auth timeout hits, the script stops running, and the result is your databases don’t get the maintenance you probably want them to have.

This information may have been presented someplace else, but I’m posting it here for my benefit and I hope it helps someone else.

This HOWTO is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. ©2008 Strykar

The lowdown

I hate spammers; I mean I really hate their kind. It stems from a crevasse deep inside me since Sabir Bhatia sold Hotmail.

There are plenty of excellent methods for spam-control, none of them fool-proof, and almost all the good ones need you to ‘train’ the filters to differentiate between what’s spam and what’s not.

I am not going to delve into this here, suffice to say, that I use the same RBLs that sendmail.org does and some selective country/host based filtering keeps out most of the spam from my mail-servers. If you use Windows, try this: www.cloudmark.com/desktop – it’s by the author of Vipul’s Razor.

The subject of today’s rant is the burning desire we all have to smack a spammer, or bludgeon his gonads with a 9 Iron. We all wanna get back at spammers, and let’s face it, there’s very little chance that we actually will.

Here’s why YOU and I won’t:

• They hijack misconfigured mail-servers shifting identity and location.
• Tracking them takes time and effort and isn’t trivial, ask Spamhaus.
• It gets boring unlike watching Mplayer compile in Linux.
• The enemy only knows the system because Bruce Schneier wants him to know the system.

We typically secure our mail servers to relay using AUTH or whatever else takes your fancy. They still bombard us nagging your MTA to send email, VRFY email addresses and bloat our logs with useless details.

That’s in the past – It’s payback time.

Well, a start at least, remember that the concept of RBLs started with a bunch of SF email administrators sharing spammer host IP lists. I have no illusions that this will kick off like that, but take a gander, if it brings you the same joy it has brought me; follow the simple instructions to set it up yourself. At the end of the article, take a minute and imagine a 100’000 people running spamd and slowing spammers. Who would mind getting back at spammers?

This will only work on *BSD, the few Linux hacks floating around simply don’t work. This is NOT a method to prevent SPAM. This is NOT a How-to on using spamd and relaydb to gray-list spammers. This is intended for machines that have a public IP address and receive no external mail. You can be running Sendmail/Postfix/Exim locally for system mails etc. This typically means your MTA listens only on localhost.

If you’re not familiar with spamd, read these and come back:

All about spamd from the horse’s mouth – Daniel Hartmeier

Wikipedia – spamd

Read the spamd and spamd.conf manuals and my setup below will make sense to you. Edit spamd.conf to use your own blacklist file. If you’re not listed as an MX record anywhere, the ONLY people speaking to your SMTP port are spammers and zombies.

$ cat /etc/mail/spamd.conf
# spamd.conf for machine having no MX record.
#
all:\
:myblack:

# List everyone specifying 0.0.0.0/0 in /var/db/myblack.txt
#
myblack:\
:black:\
:msg="I'm not listed as an MX record anywhere. You are a zombie\n\
or a spammer. Die a slow stuttered death %A":\
:method=file:\
:file=/var/db/myblack.txt:


We ensure every IP address is included by using this CIDR notation.

$ cat /var/db/myblack.txt
0.0.0.0/0

Setup spamd to run from /etc/rc.conf The switches -bvh mean blacklist, log verbose and ‘display hostname as’.

$ cat /etc/rc.conf|grep spam
spamd_flags="-b -v -h tarpit.spamtrap.host" # for normal use: "" and see spamd-setup(8)
spamd_black=YES # set to YES to run spamd without greylisting
spamlogd_flags="" # use eg. "-i interface" and see spamlogd(8)


Now comes the easiest part. We tell pf not to send anything from our public/external interface to the SMTP port at localhost. This is a fail-safe for those running a mailserver locally. We also tell pf to route incoming connections to our SMTP port on the public interface to spamd running on localhost.

$ cat /etc/pf.conf|grep smtp -A 1
no rdr on $ext_if proto tcp from any to 127.0.0.1 port smtp
rdr pass log on $ext_if proto tcp from any to any port smtp \
-> 127.0.0.1 port spamd


Setup syslog to log spamd to /var/log/spamd so we have one log to rule them all.

$ touch /var/log/spamd
$ cat /etc/syslog.conf|grep spamd
!!spamd
daemon.err;daemon.warn;daemon.info;daemon.debug /var/log/spamd


Now you’re all set to waste spammer processor cycles. kill -HUP spamd and syslog; reload pf rules. Watch with unabated joy as the bastards get stuck in your tarpit by: $ tail -f /var/log/spamd

Conclusion

Here are some snipped results from my /var/log/spamd:

$ tail -f /var/log/spamd
Sep 17 18:22:27 barge spamd[14558]: listening for incoming connections.
Sep 18 11:24:04 barge spamd[14558]: 218.167.76.111: connected (1/0)
Sep 18 11:24:07 barge spamd[14558]: 218.167.76.111: disconnected after 3
seconds.
Sep 18 15:00:15 barge spamd[14558]: 211.74.105.53: connected (1/0)
Sep 18 15:04:35 barge spamd[14558]: (GREY) 211.74.105.53: gdnrpt@289.95.17.232
- abcc.1234@gmail.com
Sep 18 15:06:20 barge spamd[14558]: 211.74.105.53: From: gdnrpt@289.95.17.232
gdnrpt@59.95.17.232
Sep 18 15:06:20 barge spamd[14558]: 211.74.105.53: Subject:
289.95.17.232,25,webmaster,webmaster,,-SMTP-Dx1784E
Sep 18 15:06:20 barge spamd[14558]: 211.74.105.53: To: abcc.1234@gmail.com
Sep 18 15:06:20 barge spamd[14558]: 211.74.105.53: Body:
ULHs9076819R%webmaster#webmaster%289$95$17$232%PAVO5204412G
Sep 18 15:07:09 barge spamd[14558]: 211.74.105.53: disconnected after 414
seconds.


Now let’s see what and by how much.

# grep disconnected /var/log/spamd | awk '{print $9}' \
> | sort -rn | uniq -c | head
1 55574
1 40390
1 8888
1 909
1 898
1 872
7 805
1 803
2 801
1 800


The first at 55574 seconds took over 15.43 hours trying to finish delivering a single email! The second at 11 hours.

Hah haaa, bastards! Say EHLO to my personal spam decelerator.

Lookup my sendmail article for its default SMTP protocol timers. By seeing how much time each attempt took to finish the SMTP dialogue, we can make educated guesses as to which mailserver the spammer machine is running. Most run dowdy FreeSMTP or SmartSMTP on Windows. The rest are misconfigured *NIX SMTP servers being abused as open relays. Of course, you need to know the default configuations that they ship with.

All of the above is on a P-200Mhz with 32MB of EDO RAM and repeated top / vmstat runs with and without spamd showed absolutely no extra resources used. That’s how well it’s written. For those of you running BSD on a machine not as archaic as mine, you’ll have zero worries running spamd and resource hogging. You’ll forget you have it running, I have.

To sum up, just watch one spammer connect and waste time. The pleasure of watching is almost, ahem, anal. 50 bucks says you’ll keep it running.

All this is for personal machines. You definitely want to use spamd in front of mailservers, it doesn’t matter what it’s running, yes, even Exchange. The links at the bottom will get you up and doing that in no time.

Read Steve Williams’ October 20th, 2006 message to the OpenBSD-misc mailing list where he reports that a pure greylisting configuration immediately rid his company of approximately 95% of their spam load.

Video of OpenBSD spamd in action

(AVI 15m:19s 4.38MB)

The video shows you how spamd stutters the conversation with the spammer, backing up his mail queue. Spamd sets its socket receive buffer size to one character, forcing the sender to send one TCP packet for each byte of data, even if it’s a non-compliant “dump and disconnect” mailer, so the spammer wastes CPU cycles and network bandwidth in addition to a delayed queue.

The video link is me telnetted into a spamd install. The SMTP transaction is recorded at real-time speeds to show what a tarpitted, stuttered, SMTP conversation looks like. Download it and scroll ahead – you get the drift.

If the video seems dodgy, and you use Windows, download the MSU screen capture codec. Alternatively, you can download the larger DivX here.

TODO:

1. Use addresses logged by spamd to setup my own RBL. Have sendmail use it from our DNS servers as a private and guaranteed spammer-only list. Nobody should be talking SMTP to a dynamic home IP netblock!

2. Document the process to setup and test a private RBL with the required steps for BIND so this writeup can be a walkthrough for the process.

Additional reading:

• Wikipedia – Tarpitting
• Peter N. M. Hansteen – PF and spamd
• ONLamp.com – Greylisting with PF and spamd
• Andy Valencia’s ghetto method of fooling spammers.
• Project Honey Pot – No response to my registration. If anyone works with them, please drop me a line.