Thursday, January 17, 2013

Seagate disk SMART values...

I was just looking at smartctl output from one of my disks, and it had a large number for Seek error rate attribute (note that I edited the output for readability):

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG   VAL WOR THR TYPE     UPDATED WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f 100 253 006 Pre-fail Always      -       0
  3 Spin_Up_Time            0x0003 098 098 000 Pre-fail Always      -       0
  4 Start_Stop_Count        0x0032 100 100 020 Old_age  Always      -       826
  5 Reallocated_Sector_Ct   0x0033 100 100 036 Pre-fail Always      -       0
  7 Seek_Error_Rate         0x000f 072 060 030 Pre-fail Always      -       17262017054
  9 Power_On_Hours          0x0032 087 087 000 Old_age  Always      -       11538
 10 Spin_Retry_Count        0x0013 100 100 034 Pre-fail Always      -       0
 12 Power_Cycle_Count       0x0032 100 100 020 Old_age  Always      -       838
187 Reported_Uncorrect      0x0032 100 100 000 Old_age  Always      -       0
189 High_Fly_Writes         0x003a 100 100 000 Old_age  Always      -       0
190 Airflow_Temperature_Cel 0x0022 067 043 045 Old_age  Always  In_the_past 33 (0 17 33 31 0)191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age  Always      -       0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age  Always      -       809
193 Load_Cycle_Count        0x0022 001 001 000 Old_age  Always      -       351945
194 Temperature_Celsius     0x001a 033 057 000 Old_age  Always      -       33 (0 11 0 0 0)195 Hardware_ECC_Recovere 0x0012 097 044 000 Old_age  Always      -       196225726
197 Current_Pending_Sector  0x0010 100 100 000 Old_age  Offline     -       0
198 Offline_Uncorrectable   0x003e 100 100 000 Old_age  Always      -       0
199 UDMA_CRC_Error_Count    0x0000 200 200 000 Old_age  Offline     -       1
200 Multi_Zone_Error_Rate   0x0032 100 253 000 Old_age  Always      -       0
202 Data_Address_Mark_Errs  0x0000 100 253 000 Old_age  Offline     -       0
254 Free_Fall_Sensor        0x0000 100 253 000 Old_age  Offline     -       0
It's not the first time I saw such a large raw values which, while not problematic (VAL/WOR/THR should be actually monitored) are nevertheless interesting, to say at least. While searching around I stumbled on a post Seagate's Seek Error Rate, Raw Read Error Rate, and Hardware ECC Recovered SMART attributes. In this post, the author explains that all the values are actually 48 bits, and due to the way they are encoded it follows that those values are large. More specifically, raw value of the Seek error rate attribute should be converted to hexadecimal and then upper 16 bits are number of errors, while lower 32 bits are total number of seeks.

In this concrete case the raw value for Seek error rate is 17262017054, or 0x000404E57A1E. The first 16 bits is 0x0004 and the last 32 bits are 0x04E57A1E. What this means is that there were 4 seek errors (meaning the head wasn't positioned correctly after being moved to some track) but there were 82147870 seeks in total. So, this is very very small fraction of errors.

For the meaning of Seek error rate attribute, and many others, I recommend Wikipedia's page about SMART.

Friday, January 4, 2013

Partition vs. whole disk and creating encrypted filesystem...

With a new laptop I got a 1T disk which I intend to use as a data disk. So, it will have a single encrypted partition. This is a new disk with a 4K sector size and because of that fdisk tool offers me to start partition on 2048th sector. This is some alignment stuff from the old days of MSDOS, and obviously I don't want to waste disk space for those reasons. You can read more about that on Linux ATA Wiki. Linux is the only OS I'll use with this disk. It is possible to start partition from 63rd sector but if you are using fdisk you'll have to first create a partition and then switch to expert menu (option x) where it is possible to move beginning of the partition from 2048th to 63rd sector (option b).

Now, it is also possible to use the whole disk for a filesystem, without partition table. I found some discussions of pros and cons of this approach. Additional question is if the Fedora will recognize such disks during a boot process. LWM HOWTO also talks about this issue. It seems that everything boils down to the problem if some other tools or operating systems, that expect disk to be partitioned, treat disk as unpartitioned and thus destroy data on it. Also, someone noted possible performance degradation, but this was not confirmed by simple testing (look at the first link I gave), and besides, why would that happen when you use the whole disk? It can not be better aligned, can it? Also, someone used the whole disk for his Gentoo OS and then he had to install GRUB. Since GRUB, during installation, asks you whether you want it to be installed on, e.g. /dev/sda or /dev/sda1, it seems that it isn't important if you don't have partition table. But, I didn't go more deeper in this.

In the end, I decided to use the whole disk, no partitions. This disk will hold a single partition, will have only data on it, it will never be used on anything other than Linux, actually, on anything other than my laptop. So, this is the way I decided to go.

So, from that point on everything was very simple:
  1. Encrypt the whole disk
  2. # cryptsetup luksFormat /dev/sdc

    WARNING!
    ========
    This will overwrite data on /dev/sdc irrevocably.

    Are you sure? (Type uppercase yes): YES
    Enter LUKS passphrase:
    Verify passphrase:
  3. Open crypted disk:
    # cryptsetup luksOpen /dev/sdc cryptodev1
    Enter passphrase for /dev/sdc:
  4. Create file system:
  5. # mkfs -t ext4 /dev/mapper/cryptodev1
    mke2fs 1.42.5 (29-Jul-2012)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    61054976 inodes, 244190134 blocks
    12209506 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=4294967296
    7453 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
            4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
            102400000, 214990848

    Allocating group tables: done
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done
  6. Remove reserved blocks (5% by default):
  7. # tune2fs -m 0 /dev/mapper/cryptodev1
    tune2fs 1.42.5 (29-Jul-2012)
    Setting reserved blocks percentage to 0% (0 blocks)
  8. Finally, mount a disk:
  9. # mount /dev/mapper/cryptodev1 /mnt
And that's basically it. When you want to use disk, and it is not mounted, then you first have to open crypted device (step 2) and then you mount newly created file system.

Thursday, January 3, 2013

Signing XML document using xmlsec1 command line tool

Suppose that you have some XML document you wish to sign. It turns out it's very easy to do so because there is xmlsec library, and in particular xmlsec1 command line tool that's standard part of Fedora Linux distribution. The only problem is that its very picky and not very informative when it comes to error logging, finally, there are a lot of small details that can catch you. Since I had to sign a document I spent some time trying to figure out how to do that. In the end, I managed to do it and I'll write here how to for a future reference. Before I continue you'll need certificate and a key to be used for verification and signing. They are not the topic of this post so I'll just give you them: private key, certificate, and CA certificate.

Ok, let's assume that you have the following XML document you wish to sign:
<?xml version="1.0" encoding="UTF-8"?>
<document>
  <firstelement attr1="attr1">
    Content of first element.
    <secondelement attr2="attr2">
      Content of the second element.
      <thirdelement attr3="attr3">
        And the content of the third element.
      </thirdelement>
    </secondelement>
  </firstelement>
</document>
Basically, you can take any XML document you wish. I'll suppose that this XML document is stored in the file tosign.xml. If you typed yourself XML document, or if you just want to be sure, you can check if XML is well formed. There is xmllint tool that surves that purpose. Just run it like this:
$ xmllint tosign.xml
And if you don't get any error messages, or warnings, that the XML document is well formed. You can also check if the document is valid by providing schema, or DTD, via appropriate command line switches.

In order to sign this document you have to add XML Signature fragment to the XML file. That fragment defines how the document will be signed, what will be signed, and, where the signature, along with certificate, will be placed. The fragment has the following form:
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
  <SignedInfo>
    <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
    <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
    <Reference>
      <Transforms>
        <Transform           Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
        <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
      </Transforms>
      <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
      <DigestValue />
    </Reference>
    </SignedInfo>
  <SignatureValue />
  <KeyInfo>
    <X509Data />
  </KeyInfo>
</Signature>
Note that this (quite verbose) fragment has to be placed somewhere within the root element. Now, lets sign this, newly created document. To do so invoke xmlsec1 command like this (this is one line in case it is broken into two due to the formatting):
xmlsec1 --sign --privkey-pem privkey.pem,cert.pem --output signed.xml tosign.xml
After this, the signed XML document will be in the file named signed.xml. Take a look into it, the placeholders within signature fragment are filled up with signature data, and with a certificate who's private key was used to sign the XML document.

Note that signature itself is generated using private key (privkey.pem) which, as its name suggest, has to be private for a signer. Otherwise, anyone can falsify signature.

Now, to verify signed XML document you have to specify trusted CA that will be used to verify signature. It has to be certificate of the certificate authority (CA) that issued signer's certificate. In my case that's cacert.pem, i.e.:
$ xmlsec1 --verify --trusted-pem cacert.pem signed.xml
OK
SignedInfo References (ok/all): 1/1
Manifests References (ok/all): 0/0
As you can see, the signature was verified OK. You can try now to change something in the XML document and see if the verification passes or not.

I'll mentioned one more thing before concluding this post. Namely, in the previous example the whole XML document was signed. But, you can sign only a part. To do so, you have to do two things. The first one is to mark the element that you wish to sign (its content will also be signed) and the second is to tell xmlsec1 to sign only that element.

The first step is accomplished by adding attribute to the element that should be signed. Let's assume that in our case we only want secondelement to be signed. Modify the appropriate opening tag to have the following form:
<secondelement attr2="attr2" id="signonlythis">
Note that I added attribute id, but basically any name can be used (unless you use some predefined schema or DTD).

The second step is to tell xmlsec1 that only this element should be signed. This is accomplished by modifying Reference element to have the following form:
<Reference URI="#signonlythis">
If you now try to sign this modified XML document using command I gave above, you'll receive an error message:
$ xmlsec1 --sign --privkey-pem cert.key,cert.pem --output test_signed.xml tosign.xml
func=xmlSecXPathDataExecute:file=xpath.c:line=273:obj=unknown:subj=xmlXPtrEval:error=5:libxml2 library function failed:expr=xpointer(id('signonlythiselement'))
func=xmlSecXPathDataListExecute:file=xpath.c:line=356:obj=unknown:subj=xmlSecXPathDataExecute:error=1:xmlsec library function failed:
func=xmlSecTransformXPathExecute:file=xpath.c:line=466:obj=xpointer:subj=xmlSecXPathDataExecute:error=1:xmlsec library function failed:
func=xmlSecTransformDefaultPushXml:file=transforms.c:line=2405:obj=xpointer:subj=xmlSecTransformExecute:error=1:xmlsec library function failed:
func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec library function failed:transform=xpointer
func=xmlSecTransformCtxExecute:file=transforms.c:line=1296:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec library function failed:
func=xmlSecDSigReferenceCtxProcessNode:file=xmldsig.c:line=1571:obj=unknown:subj=xmlSecTransformCtxExecute:error=1:xmlsec library function failed:
func=xmlSecDSigCtxProcessSignedInfoNode:file=xmldsig.c:line=804:obj=unknown:subj=xmlSecDSigReferenceCtxProcessNode:error=1:xmlsec library function failed:node=Reference
func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=547:obj=unknown:subj=xmlSecDSigCtxProcessSignedInfoNode:error=1:xmlsec library function failed:
func=xmlSecDSigCtxSign:file=xmldsig.c:line=303:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec library function failed:
Error: signature failed
Error: failed to sign file "tosign.xml"
The problem is that URI attribute references ID attribute of an element. But, ID element isn't recognized by name but has to be specified in DTD or in schema, depending what you have. In our case there is neither schema nor DTD and thus ID isn't recognized by xmlsec1. So, we have to tell it what is the name of the ID attribute, and that can be done in two ways. The first one is by using command line switch --id-attr, and so the command to sign this document is:
xmlsec1 --sign --privkey-pem privkey.pem,cert.pem --id-attr:id secondelement --output signed.xml tosign.xml
The name after the column is the attribute name that is ID. Default value is "id", but can be anything else. If it is "id", then it can be omitted. The argument to --id-attr is element whose attribute should be treated as an id. You should also be careful of namespaces. If they are used then the namespace of the element has to be specified too, and not shorthand but the full namespace name. Finally, note that XML is case sensitive!

The other possibility is to create DTD file and to give it as an argument to xmlsec1. In this case, DTD should look like this (I'll assume that this is the content of a file tosign.dtd):
<!ATTLIST secondelement id ID #IMPLIED>
And you would invoke xmlsec1 like this:
xmlsec1 --sign --privkey-pem privkey.pem,cert.pem --dtd-file tosign.dtd --output signed.xml tosign.xml
Note that you'll receive a lot of warnings (DTD is incomplete) but the file will be signed. To check the signature, you again have to specify either --dtd-file or --id-attr options, e.g.
xmlsec1 --verify --trusted-pem cacert.pem --id-attr:id secondelement signed.xml
Now, you can experiment to check that really only secondelement was signed and nothing else.

Final note. You have to put XML signature fragment in XML file you are signing. What can confuse you (and confused me) is that there is an option sign-tmpl that adds this fragment, but it is very specific and used only for testing purposes.

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive