Nerd Notes

/dev/brain: no space left on device

How to extract a filesystem from a disk image

with 11 comments

You need to backup an entire hard disk to a single file. Supposing your disk is at /dev/hda and the backup file is image-file, you’d do:

# cat /dev/hda > image-file

or

# dd if=/dev/hda of=image-file

The file backup you get will hold a copy of every single bit from the hard disk. This means that you also have a copy of the MBR in the first 512 bytes of the file.

Because of this, you can see the partition table on the backup file:

# sfdisk -l -uS image-file
Disk image-file: 0 cylinders, 0 heads, 0 sectors/track
Warning: The partition table looks like it was made
for C/H/S=*/255/32 (instead of 0/0/0).
For this listing I'll assume that geometry.
Units = sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
image-filep1 32 261119 261088 83 Linux
image-filep2 261120 4267679 4006560 82 Linux swap / Solaris
image-filep3 4267680 142253279 137985600 83 Linux
image-filep4 0 - 0 0 Empty

Now, suppose you want to extract partition number 3. You can see that it starts at block 4267680 and is 137985600 blocks long. This translates into:

# dd if=image-file of=partition3-file skip=4267680 count=137985600

Now, peeking into the contents of the partition is as easy as:

# mount -t ext3 -o loop partition3-file /mnt/hack

Update (30 June 2011): as someone cleverly suggested in the comments, you can avoid using dd to extract the partition file by passing the offset option to mount as explained in this blog post.

Advertisements

Written by Mirko Caserta

April 18, 2008 at 5:10 pm

Basic transactions in Spring using TransactionProxyFactoryBean

with 25 comments

UPDATE – Aug 29, 2011: I see that quite a few sites link to this article. Please keep in mind that I wrote this at the beginning of year 2007. While it should still be valid, more recent versions of the Spring Framework have largely simplified the approach described here and the online documentation about transaction management is far more complete, precise and straightforward than what I wrote years ago. I strongly encourage you to read the Spring documentation for the most up-to-date details.

Scenario: you have a simple stand-alone java application which accesses a single database using spring DAOs and you want to make your application’s interface transaction aware.

Now, while there’s plenty of ways to achieve this, the fool-proof, working approach is to make use of Spring’s declarative transactions and the TransactionProxyFactoryBean.

Here is an example BeanFactory configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aop="http://www.springframework.org/schema/aop"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd"
>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost/foobardb"/>
<property name="username" value="dbuser"/>
<property name="password" value="dbpass"/>
</bean>
<bean id="fooDao"
class="com.acme.backend.dao.FooDAOJdbc">
<property name="dataSource" ref="dataSource"/>
</bean>
<bean id="fooBarServiceImpl"
class="com.acme.backend.FooBarServiceImpl"/>
<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource"/>
</bean>
<bean id="foorBarService"
class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean">
<property name="transactionManager" ref="txManager"/>
<property name="target" ref="fooBarServiceImpl"/>
<property name="transactionAttributes">
<props>
<prop key="get*">PROPAGATION_SUPPORTS,readOnly</prop>
<prop key="*">PROPAGATION_REQUIRED</prop>
</props>
</property>
</bean>
</beans>

Okay, this was a lot of configuration data but, don’t worry, we’ll get into each little bit, one at a time and, by the end of this article, you’ll have all figured out.

In the first part, you basically have a boilerplate xml declaration of the required schemas. Those are needed so that your xml editor knows how to do auto completion and so that, when the xml configuration file gets parsed, the parser knows what is syntactically and semantically correct and what’s not.

Then you have a datasource bean declaration:

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost/foobardb"/>
<property name="username" value="dbuser"/>
<property name="password" value="dbpass"/>
</bean>

This simply declares a configuration object with the datasource properties, which are the usual driver class name, connection URL and login credentials to the database. If you look closely, you’ll see that I’m using a commons-dbcp BasicDataSource. This means two things:

  1. the connections you’re going to get from this datasource are going to be pooled by the commons-dbcp driver
  2. you’ll have to put the commons-dbcp and commons-pool jars in the classpath

The next piece is quite simple:

<bean id="fooDao"
class="com.acme.backend.dao.FooDAOJdbc">
<property name="dataSource" ref="dataSource"/>
</bean>

This declares a DAO, plugs in a jdbc implementation class for it and injects the datasource as a property of the DAO.

<bean id="fooBarServiceImpl"
class="com.acme.backend.FooBarServiceImpl"/>

This declares the service implementation. In other words, this is the bean that implements the interface to our application. It is here merely because we need to declare a placeholder we’ll use in the transactional proxy declaration later.

<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource"/>
</bean>

Here we have declared a transaction manager using Spring’s default DataSourceTransactionManager. Then, we inject the datasource property into the transaction manager bean so that it becomes aware of what it should operate upon.

<bean id="foorBarService"
class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean">
<property name="transactionManager" ref="txManager"/>
<property name="target" ref="fooBarServiceImpl"/>
<property name="transactionAttributes">
<props>
<prop key="get*">PROPAGATION_SUPPORTS,readOnly</prop>
<prop key="*">PROPAGATION_REQUIRED</prop>
</props>
</property>
</bean>

This is a bit more complicated and needs some careful explanation. The bean we’re declaring here is a transaction proxy, a class provided by Spring which takes care of wrapping the service implementation with a transactional behavior.

The transaction proxy needs a few properties set. Let’s see what they are:

  • a transaction manager (remember we had declared that earlier, hadn’t we?)
  • a target class; this is the class that the proxy wraps and adds transactional behavior to
  • the transaction attributes

The transaction attributes are a set of properties. They describe what the transactional behavior should be. In this very simple but effective example, we are declaring that all methods of our service interface beginning with the name get are to be considered read-only and that they support transaction propagation. The other methods are considered as write methods in terms of transaction behavior and they require transaction propagation.

Now you’re gonna ask me: “And all of this is for what?”. Well, what you achieve with this kind of configuration is a fully transactional service. That means, if something goes wrong within your service operation, all you have to do is throw an exception and the Spring framework (and your database) will take care of rolling back all work.

For a simple, standalone application, this is just great. With a few lines of xml configuration data, you’ve got a fully transactional service. Ain’t that great? If you think it’s not, then you must take a long walk with an Oracle architect.

Written by Mirko Caserta

March 30, 2007 at 11:09 am

Backing up a vmware virtual machine

with 5 comments

I’ve learnt quite a number of new things today. One of these things is that an ISO9660 filesystem doesn’t allow files bigger than a couple of gigabytes. Another thing is that bzip2 kicks ass, even if it actually doesn’t implement the best compression algorithm available today (your milage may vary).

Another interesting thing is that the good old cdrtools package on Gentoo sucks. In my understanding that’s mainly because of licencing issues which came out during the software development. But, no worries, the new cdrkit ebuild has a modern cdrtools implementation with a compatible interface. That is, the good old mkisofs and cdrecord command lines you (and your GUIs) had learnt how to use are just the same. Different software, same interface. Great, uh? So, first of all, if you want the latest tools on a Gentoo box, you’ll need to:

# emerge -C cdrtools
# emerge -av cdrkit

Anyway, my job today was to backup on DVD-R media a large vmware virtual machine in order to regain free storage space on a server with serious shortage troubles. To begin with, we have a directory which holds the VM (virtual machine) files:

# du -sh my-very-cool-vm
21G my-very-cool-vm

Now, 21 gigabytes of data is a lot because this VM hosts a few extra virtual disks. But, if you’re lucky enough like me, some bzip2 goodness might help:

# tar cvf - my-very-cool-vm | \
bzip2 --compress --verbose --best --stdout > \
my-very-cool-vm.tar.bz2
# ls -lh my-very-cool-vm.tar.bz2
total 4.3G
-rw-r--r-- 1 root root 4.3G Mar 8 15:32 my-very-cool-vm.tar.bz2

Depending on your iron, the time between tar and ls might be enough for you to gain a few extra golds in World of Warcraft. Anyway, now that might actually fit on a DVD-R. But, since there’s a 2GBytes file size limit on ISO9660 filesystems, we still need to do a little something before we can fire up mkisofs. Our old unix friend split comes to the rescue:

# split -b 1024m -d my-very-cool-vm.tar.bz2 \
my-very-cool-vm.tar.bz2-part-
# rm my-very-cool-vm.tar.bz2
# ls -lh
total 4.3G
-rw-r--r-- 1 root root 1.0G Mar 8 16:04 my-very-cool-vm.tar.bz2-part-00
-rw-r--r-- 1 root root 1.0G Mar 8 16:05 my-very-cool-vm.tar.bz2-part-01
-rw-r--r-- 1 root root 1.0G Mar 8 16:05 my-very-cool-vm.tar.bz2-part-02
-rw-r--r-- 1 root root 1.0G Mar 8 16:06 my-very-cool-vm.tar.bz2-part-03
-rw-r--r-- 1 root root 295M Mar 8 16:06 my-very-cool-vm.tar.bz2-part-04

This is just perfect. Assuming the above files are in a directory called mydir, I can now run mkisofs like this:

# mkisofs -o dvd.iso -r mydir

This creates a file which holds the ISO9660 filesystem that we can later feed to cdrecord. My peculiar incantation for cdrecord is as follows:

cdrecord -v dev=/dev/hdc driver=mmc_mdvd -sao dvd.iso

Written by Mirko Caserta

March 8, 2007 at 4:54 pm

Windows shares in /etc/fstab

leave a comment »

Ok, this is pretty much an easy one, but I keep forgetting what the heck you need to specify in your /etc/fstab in order to mount an smb share (at least on a Linux system). Here is an handy example:

//host.lan/sharename /mnt/host/sharename smbfs defaults,rw,noauto,username=scott,password=tiger,uid=myuid,gid=mygid 0 0

All of the above needs to be on one line. Of course you need to replace a few things:

host.lan
hostname or ip address of the host you’re connecting to
sharename
the windows share name (pretty easy, uh?)
/mnt/host/sharename
a local directory where the share is to be mounted (this must exist before mounting so “mkdir -p” it)
username
the windows share username
password
the windows share password
uid
the user id the filesystem needs to be mounted as; this can be either numeric or symbolic and is usually your day-to-day user account
gid
the group id the filesystem needs to be mounted as; this can be either numeric or symbolic and is usually your day-to-day user account’s primary group

Once you’ve added that line into /etc/fstab, you can simply:

# mount /mnt/host/sharename

Written by Mirko Caserta

March 2, 2007 at 5:19 pm

Struttin’ from the CLI

leave a comment »

Suppose you want to write a shell script that calls every action in your struts-config.xml in order to, say, stress test every callable url in your application.

Assume you have the following struts-config.xml:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE struts-config PUBLIC
"-//Apache Software Foundation//DTD Struts Configuration 1.1//EN"
"http://jakarta.apache.org/struts/dtds/struts-config_1_1.dtd">
<struts-config>
<action-mappings>
<action
path="/page1"
forward="/WEB-INF/page/page1.jsp"/>
<action
path="/page2"
forward="/WEB-INF/page/page2.jsp"/>
</action-mappings>
</struts-config>

The point here is that you want to extract the path attribute value for each action and complete it with a full blown URI. You could do that with a lot of grep and/or awk or you could just use xmlstarlet. So, go now, download a copy and install it, or fire up your favourite package manager (be it apt-get, synaptic, portage or whatever) and let it do the job for you.

Once you have xmlstarlet installed, just do:

$ xmlstarlet sel -t -m "/struts-config/action-mappings/action" \
-o http://localhost/ctx -v "@path" -o .do -n struts-config.xml

and you should get the following output:

http://localhost/ctx/page1.do
http://localhost/ctx/page2.do

This basically means that you can, for instance, use wget to grab all those URIs:

$ xmlstarlet sel -t \
-m "/struts-config/action-mappings/action" \
-o "wget http://localhost/ctx" \
-v "@path" \
-o .do -n \
struts-config.xml | sh

This will fire up wget for each action path in the struts-config.xml. Neat, uh?

Under the hood, xmlstarlet is simply using some xslt transformation on the xpath expressions given in the above template to do its work. So, in case you want to sharpen your xml tools, you’ll need to get a better understanding of xpath at least, since it has to be used in the select templates.

Written by Mirko Caserta

November 15, 2006 at 3:34 pm

Posted in CLI, struts, XML

UNIX time to human readable

with one comment

The current issue of the Gentoo Weekly Newsletter has a good clue on how to easily convert a date from the standard UNIX time format (which basically is a number which represents the number of seconds elapsed since January 1, 1970 at midnight, UTC) to a human readable form. The best solution is using date -d on Linux systems such as in:

$ date -d @1161911504
Fri Oct 27 03:11:44 CEST 2006

However, on BSD derived systems, such as Mac OS X (and maybe Solaris), date -d won’t work so you have to used instead:

$ date -r 1161911504
Fri Oct 27 03:11:44 CEST 2006

Written by Mirko Caserta

November 1, 2006 at 8:10 pm

Posted in CLI, Console, Linux, UNIX

gpasswd and less +F

leave a comment »

Before I knew about gpasswd, I used to add and remove users from groups using a combination of “usermod -G” and some cut and paste in the terminal window. I was even using sed to automate the command. That is now a thing of the past.

Most modern linux distributions have a gpasswd command (at least Gentoo and Ubuntu do have it). Adding a user to a group is as easy as:

# gpasswd -a username groupname

Removing is just a matter of changing the -a switch (add) into -d (delete):

# gpasswd -d username groupname

Another useful thing I discovered recently is the “follow mode” of less, which works like a tail -f. If you want to launch less in “follow mode”, you just have to add the +F switch, as in:

# less +F /var/log/messages

If you want, at any time, to scroll back through the backlog, you just have to interrupt the follow mode typing C-c (that is: control+c). Also, you don’t need to start less in follow mode to use the follow mode: you can just press F at any time within less to turn the follow mode on.

Written by Mirko Caserta

October 31, 2006 at 11:10 am