Blu-ray Disc/ DVD+RW/+R/ -R[W] for Linux

by <appro@fy.chalmers.se>, March 2008
Japanese


Q. What is this page (not) about?
A.  Maybe to your disappointment it is not about video(*). The scope of this page is primarily computer storage applications of Blu-ray Disc and DVD±RW/±R, things like backup, archiving, data exchange... The downloadable files are an optional [and essentially historical] Linux 2.4 kernel DVD+RW patch and a couple of user-land utilities dubbed as dvd+rw-tools.

(*) Though it doesn't mean that you can't burn DVD-Video discs with dvd+rw-tools. [Unlike Video-CD] DVD-Video is "molded" in an ordinary data file system and therefore no explicit support by the burning program is actually required. In other words it is the DVD-Video content preparation which is beyond the scope of this page.

Q. Kernel patch? What would it be good for?
A. It should be explicitly noted that the user-land utilities, dvd+rw-tools, do suffice for BD/DVD recording without explicit kernel support. So if they fulfill your requirements, then patching the kernel is by all means optional. Kernel support would make it possible to house a read-write file system, e.g. udf, vfat, ext2, etc. and accomodate "live" data on suitable optical media. However, I have to recommend to deploy it with caution, see tutorial for further details. Above mentioned 2.4 patch is provided mostly for historical reasons. Linux kernel version 2.6>=10 is equipped with packet writing driver which supports even DVD rewritable media, but I haven't tested it myself, so don't ask:-)

Q. What are the dvd+rw-tools for?
A. As implied/already mentioned - to master the Blu-ray Disc and DVD media, both +RW/+R and -R[W]. I could simply refer to the tutorial below, but figured that couple of words about the [original] design ideas behind growisofs, the principal burning utility, wouldn't harm. Even though a modified kernel can let you put for example an ext2 file system on DVD+RW, it's probably not very practical, because you most likely want to access the data on an arbitrary computer. Or in other words you most likely want ISO9660. The trouble is that you might as well want to add data now and then. And what options do you have in the lack of multiple sessions (no, DVD+RW has no notion of multiple sessions)? Complete re-mastering which takes more and more time as data set grows? Well, yes, unless you employ growisofs! Growisofs provides the way to both lay down and grow an ISO9660 file system on (as well as to burn an arbitrary pre-mastered image to) all supported optical media.

Q. But if they support both + and - recording strategies, why are they called dvd+rw-tools?
A. For historical/nostalgical reasons, as originally they did support exclusively DVD+plus. On the other hand now, when the vast majority of DVD burners that are being introduced to the market today are DVD+capable, the name most likely refers to your unit in either case. And you can always consider the plus in the name as notion of a unique quality, such as "seamless" multisessioning, not as reference to some particular format:-)

Q. Do I still need cdrtools?
A. Yes. It should be explicitly noted that growisofs is a front-end to mkisofs, i.e. invokes mkisofs to perform the actual ISO9660 file system layout. Secondly, the DVD burners available on the market can burn even CD-R[W] media and cdrecord is the tool for this job.

Q. Does it work with my recorder unit?
A. If your unit is MMC compliant, then the answer is "most likely it just does." Well, as the probability of your unit being non-MMC compliant is virtually zero, the answer in practice is unconditionally "most likely." The [core] tools were reported to work with a wide range of drives, including [but not limited to] HP dvd[12345]x0i, Ricoh MP512x, Philips DVDRW[248]xx, SONY DRU-[157]x0, NEC ND-[1234]xx0, TDK indiDVD 4x0N, Plextor PX-[57]xx, Benq DW[48]00A, OptoRite DD0[24]0x, Lite-On LDW-[4816]xxS, as well as nonplus units such as Pioneer DVR-x0[45679], LG GxA-40[248]x, Toshiba SD-R[56]112, Panasonic UJ-811, LF-D[35]1x, and not the least all-mighty SW-5582...

Q. Is there a GUI front-end available for dvd+rw-tools?
A. K3b, version 0.10 and later, and nautilus-cd-burner, version 0.5.1 and later, are both hiding growisofs behind their pretty buttons and menus:-) Keep in mind that those are not directly related to dvd+rw-tools development effort and GUI users should turn elsewhere for end-user support. Oh! dvd+rw-tools 5.10.x is a minimum requirement for GUI frontends...

Q. I don't run Linux. What are my options?
A. Version 5.4 adds support for OpenBSD/NetBSD. Version 5.6 adds support for Solaris 2.x [commercial licensing terms for distribution on Solaris are to be settled with Inserve Technology]. Version 5.8 features FreeBSD port contributed by Matthew Dillon, FreeBSD Development Team alumnus. Hewlett-Packard Company has donated HP-UX 11 support for 5.14(*). IRIX 6.x support appears in 5.19, Win32 one - in 6.0, while Mac OS X - in 7.0...

¡ Common usage tip! Whenever separately available [and unless stated otherwise], do use character-type device entry with dvd+rw-tools, e.g. OpenBSD/NetBSD users should stick to /dev/rcdXc.
(*) As of 5.14 HP-UX support was classified as "initial." Version 5.18 in turn is the one which has undergone HP quality assurance testing and is delivered on HP software depot.


Foreword

As of May 2003 I've decided to advise users to turn to <cdwrite@other.debian.org> on support matters. It's an open list, meaning that you don't have to be subscribed to post a problem report. List archives can be found at both subscribe page and mail-archive.com. When submitting report, provide versioning information, exact command line, exact output generated by the program and complement it with dvd+rw-mediainfo output for resulting recording. Do check couple of last archived months, as the issue might have been discussed recently. If you've chosen to contact me personally and haven't heard back within a week or so, then you most likely overlooked something on this page. Please read it more attentively...

Special thanks for hardware donations [in chronological order]:
Inserve Technology  HP  LinuxFund  comm*tech 


Tutorial



Compatibility: caveat lector


This paragraph discusses "DVD-ROM compatibility," or playability of already recorded media in legacy units. Blank media compatibility issues, or cases such as failure to start or fulfill recording because of poor media support by burner firmware, are beyond the current scope. Turn to your vendor for list of supported media and/or to the public to share your experience.

In order to optimize seek times DVD[-ROM] players calibrate their mechanics every time the media is loaded by sliding the optical head some place, picking up the signal and noting the physical block address underneath the lens. In order for this procedure to work with re-writable/recordable media, that particular spot has to be written to [or de-iced in DVD+RW terms]. Some units slide the head to 30mm [radial] to calibrate, some to 35mm. In order to keep such players "happy," make sure that at least 1GB is written [before you attempt to mount it in DVD-ROM unit].

Other units attempt to seek to lead-out [or vicinity of it] for calibration purposes. Now the catch is that it's perfectly possible to produce a DVD+RW disc without lead-out. Most notably media initially formatted with dvd+rw-format [apparently] doesn't have any lead-out, not to mention that practically whole surface remains virgin. If you fail to mount/play DVD+RW media, attempt to

dvd+rw-format -lead-out /dev/scdN

which relocates the lead-out next to outermost written sector as well as makes sure there is no virgin surface before it. Previously written data is not affected by this operation.

Then non-finalized DVD+R and Sequential DVD-R[W] discs don't have lead-out either(*). If you fail to mount/play DVD+R media and wish to sacrifice the remaining space for immediate compatibility, just fill the media up(**). Alternatively if you master volume in a single take and don't plan to use it for multisessioning(***), you have the option to invoke growisofs with -dvd-compat option and cut the real lead-out directly after the first session.

(*) Well, there are lead-outs at the session edges, but the problem is that "End Physical Sector Number of Data Area" field in "Control Data Zone" of the lead-in contains address of the largest media sector, which makes affected DVD[-ROM] players calibrate at the outermost edge instead of the first session. Actually I fail to understand why don't they burn the address of last sector of the first session in the lead-in even on multisession discs...
(**) But beware the 4GB limit! If 4GB is already an issue, or if you don't feel like throwing unrelated data on the media in question, then invoke 'growisofs -M /dev/scd0=/dev/zero' (supported by 5.6 and later). Alternative is to re-master the whole volume, naturally with -dvd-compat option.
(***) E.g. when mastering DVD-Video disc:-) Note that -dvd-video option [passed to mkisofs] engages -dvd-compat automatically.


Then we have logical format compatibility issue(s). Probably the very ground for all the controversy around DVD+RW, rather around DVD+RW media not being playable in a whole range of players. DVD+RW Alliance was keen to blame on DVD-ROM vendors, even claiming that they deliberately block playback. But the fact is that format specifications don't explicitly say that unrecognized format [designated by "Book Type" field in "Control Data Zone" of the lead-in] should be treated as DVD-ROM and [in my opinion] it was rather naive of them to claim and expect that the media will be playable in "virtually all players." This deficiency was recognized by practically all DVD+RW vendors [well, apparently by "traditional" DVD+RW vendors and not "latest generation" vendors such as Sony, NEC, TDK...] and a secret vendor-specific command manipulating this "Book Type" field was implemented. So if you fail to mount/play DVD+RW media, you might have an option to

dvd+rw-booktype -dvd-rom -media /dev/scdN

Once again. Not all vendors support this and you can't expect this utility to work with all recorders.

It's naturally not possible to manipulate the "Book Type" field on DVD+R media, that is not after the lead-in is written [which takes place at the moment the first session gets closed]. But it might be possible to control how it [lead-in] is going to be branded by programming the drive in advance:

dvd+rw-booktype -dvd-rom -unit+r /dev/scdN

Meaning that if you fail to play DVD+R media, you can attempt to burn another disc with more appropriate unit settings. For more background information about dvd+rw-booktype, see "Compatibility Bitsettings" article at dvdplusrw.org.

There [potentially] are other logical DVD+RW(*) format incompatibilities, but the "Book Type" issue discussed above is the only one "officially" recognized. Well, it's actually understandable as it's the only one that can be recognized and addressed by a DVD+RW vendor alone. Recognition of other incompatibilities would require cooperation from DVD[-ROM] player vendors and that's something they apparently are not willing to show referring to the fact that DVD+RW format is not approved [and apparently never will be] by DVD Forum(**).

(*) Finalized DVD+R media branded with DVD-ROM "Book Type" is virtually identical to DVD-ROM.
(**) To which I say "so what?" DVD Forum is an alliance of manufacturers just like DVD+RW Alliance is. It [or any other party for that matter] has no authority to deny a technology development initiative.


Finally there is a physical incompatibility issue. They claim that there are optical pick-ups out there not being capable to decode the track because of low reflectivity of DVD+RW media surface. I write "they claim," because in the lack of cooperation from DVD[-ROM] vendors it's not possible to distinguish physical from logical format incompatibility, which I find important to tell apart in order to make sure at least logical format incompatibility issues don't persist over time. It might be as trivial as following. As you surely know [already], DVD+RW has same reflectivity as dual-layer DVD-ROM. Now the catch is that the linear pit density in turn is same as of single-layer one. Meaning that if player makes assumptions about linear pit density based on reflectivity, then it won't be able to trace the track... But either way, there is very little you can do about this one, but to try another player...


Technical Ramblings


As for multisession ISO9660 [DVD] recordings! Unfortunately, Linux ISOFS implementation had certain deficiency which limits interoperability of such recordings. In order to understand it, have a look at sample ISO9660 layout to the right... Now, the problem is that isofs i-nodes(*) are 32 bits wide (on a 32-bit Linux) and represent offsets of corresponding directory entries (light-greens), byte offsets from the beginning of media. This means that no directory (green areas) may cross 4GB boundary without being effectively corrupted:-( It should be noted that in reality it's a bit better than it looks on the picture, as mkisofs collects all the directories in the beginning of any particular session (there normally are no blues between greens). The first session is therefore never subject to i-node wrap-around, but not the subsequent ones! Once again, files themselves may reside beyond the 4GB boundary, but not the directories, in particular not in further sessions. Having noted that directory entries are actually specified to start at even offsets, I figured that it's perfectly possible to "stretch" the limit to 8GB. But in order to assure maximum interoperability, you should not let any session start past 4GB minus space required for directory structures, e.g. if the last session is to fill the media up, it has to be >400MB. As of version 5.3 growisofs refuses to append a new session beyond 4GB-40MB limit(**), where 40MB is pretty much arbitrary chosen large value, large for directory catalogs that is. Yet it doesn't actually guarantee that you can't suffer from i-node wrap-around. Interim fs/isofs 2.4 kernel patch was addressed to those who have actually ran into the problem and have to salvage the data. Even though permanent solution for this problem appears in Linux kernel 2.6.8 (thanks to Paul Serice effort), growisofs keeps checking for this 4GB limit in order to ensure broader compatibility of final DVD recordings. This check is not performed for Blu-ray Disc recordings, as probability that a member of such user community would run something elder than 2.6.9 is considered diminishingly low.
(*) i-node is a number uniquely identifying a single file in a file system
(**) well, as DVD+R Double Layer support was introduced in 5.20, -use-the-force-luke=4gms option was added to override this behaviour (naturally recommended for Linux kernel 2.6>=8 users and kernel developers only;-)


Why media reload is performed after every recording with growisofs? Well, it's performed only if you didn't patch the kernel:-) But no, I do not insist on patching the kernel! All I'm saying is that in the lack of kernel support, media reload is performed for the following reasons. In order to optimize file access kernel maintains so called block cache, so that repetitive requests for same data are met directly from memory and don't result in multiple physical I/O. Now the catch is that block cache layer remains totally unaware of growisofs activities, growisofs bypasses the block cache. This means that block cache inevitably becomes out of sync, which in turn might appear to you as corrupted data. Media reload is performed when flushing the block cache is not an option, e.g. only privileged user is allowed to perform it. Second reason is to force kernel to readjust last addressable block in case it was changed as result of recording. This is done to preclude spurious "attempts to access beyond end of device."


What does [kernel] "DVD+RW support" really mean? Even though DVD+RW has no notion of [multiple] sessions, to ensure compatibility with DVD-ROM it's essential to issue "CLOSE TRACK/SESSION (5Bh)" MMC command to terminate/suspend background formatting (if any in progress) whenever you intend to eject the media or simply stop writing and want better read performance (e.g. remount file system read-only). This is what the patch is basically about: noting when/if media was written to and "finalizing" at unlock door.

Secondly, whenever you employ fully fledged file system, I/O requests get inevitably fragmented. "Fragmented" means following. Even though you can address the data with 2KB granularity, it [data] is physically laid out in 32KB chunks. This in turn means that for example writing of 2KB block involves reading of 32KB chunk, replacing corresponding 2KB and writing down of modified 32KB chunk. "Fragmented requests" are those that are smaller than 32KB or/and cross the modulus 32KB boundaries. In order to optimize the process certain caching algorithm is implemented in unit's firmware. Obviously it can't adequately meet all possible situations. And so in such unfortunate situations the drive apparently stops processing I/O requests returning "COMMAND SEQUENCE ERROR (2Ch)" ASC. This is the second essential of "DVD+RW support," namely injecting of "SYNCHRONIZE CACHE (35h)" MMC command in reply to the error condition in question. The command flushes the cached buffers which makes it possible to resume the data flow.

Unfortunately the above paragraph doesn't seem to apply to the 1st generation drives, Ricoh MP5120A and derivatives:-( "SYNCHRONIZE CACHE (35h)" doesn't seem to be sufficient and the unit keeps replying with "COMMAND SEQUENCE ERROR (2Ch)" going into end-less loop. This makes it impossible to deploy arbitrary file system. I'm open for suggestions... Meanwhile the I've chosen to simply suspend I/O till the media is unmounted. Even 2nd gen unit were reported to exhibit similar [but not the same] behaviour under apparently extremely rare circumstances. At least I failed to reproduce the problem... The problem reportedly disappears with firmware upgrade...

Then some [most?] of post-2nd gen units, from most vendors, seem to not bother about complying with DVD+RW specification, "true random write with 2KB granularity" part in particular. Instead they apparently expect host to apply procedure pretty much equivalent to DVD-RW Restricted Overwrite. To be more specific host seems to be expected to coalesce 2KB requests and perform aligned writes at native DVD ECC blocksize, which is 32KB. Formally this should not be required, but it's the reality of marketplace:-(

This one really beats me. Sometimes the unit simply stops writing signaling a vendor specific positioning error, 03h/15h/82h to be specific. Especially if the media is newly formatted. Couple of work theories. One is that block buffer cache reorders requests so that they are not sequential anymore, "FLUSH CACHE" might do the trick. Another one is that under "underrun" condition background formatting kicks off and has to be explicitly stopped. "Underrun" is in quotes because the unit is supposed to handle temporary data stream outages gracefully. If you run into this (you most likely will), try to complement growisofs command line with [undocumented] -poor-man option (which has to be first in the command line). This option eliminates request reorders and minimizes possibility for "underrun" condition (by releasing the pressure off VM subsystem).


The original idea was to implement DVD+RW support in drivers/cdrom/cdrom.c. Unfortunately SCSI layer maintains private "writeable" flag controlling the ability to issue WRITE commands. The flag is impossible to reach for from the Unified CD-ROM driver. But why am I talking about SCSI when there are only IDE units out there (at least for the time being)? Well, as you most likely want to occasionally burn even CD-R[W] with cdrecord you want it to go through ide-scsi emulation layer anyway. So I figured that SCSI CD-ROM driver is the one to aim for even for DVD+RW.

Unfortunately it was not possible to implement it completely in sr_mod.o:-( Minor drivers/cdrom/cdrom.c modification was required to sense the media before decision about whether or not to permit write open. That's because DVD+RW drives are morphing their behaviour after currently mounted media and it's essential to identify newly inserted media.

Special comment about "what a dirty hack!!!" To my great surprise it turned out that time-out value you pass in cdrom_generic_command is simply ignored and time-out is set to pre-compiled value of 30 seconds. Unfortunately it's way too low for formatting purposes and I had to do something about it. Alternative to "the dirty hack" was to add another argument to sr_do_ioctl and modify all the calls to it... I've chosen to take over those 31 unused bits from the "quiet" argument instead of modifying all the calls (too boring).

But even if time-out value passed down to kernel (with either CDROM_SEND_PACKET or SG_IO ioctl) is taken into consideration, it's apparently not interpreted as user-land code expects it to. As I figured... There is no documentation on CDROM_SEND_PACKET, but following the common sense most programmers (including myself:-) expect it to be interpreted in at least platform-independent manner, such as milliseconds maybe? SG_IO timeout in turn is documented to be measured in milliseconds... Neither of this holds true! Kernel treats these values as "jiffies," which is a platform-dependent value representing time elapsed between timer interrupts. But if we attempt to send down "jiffies," it might turn out wrong too [at least for the moment of this writing]. The catch is that [IA-32] kernel developers figured it's cool to shorten "jiffy," but didn't care to provide user-land with actual value (well, not of actual interest, too much legacy code to deal with) nor scale timeouts accordingly in respect to the legacy value of 10ms.

There is another kernel "deficiency" I ran into while working on the (original version of) dvd+rw-format utility. The drive provides background formatting progress status, but unfortunately it's impossible to access it. That's because progress counter is returned [in reply to "TEST UNIT READY"] as "NO SENSE/LOGICAL UNIT NOT READY/FORMAT IN PROGRESS" sense bytes but with "GOOD" status. Apparently any sense data with "GOOD" status is discarded by the common SCSI layer.

It was pointed out to me that DVD+RW units work with Acard SCSI to IDE bridges.


What does plus in DVD+RW/+R stand for? Originally this paragraph started as following:

The key feature of DVD+RW/+R media is high [spatial] frequency wobbled [pre-]groove with addressing information modulated into it. This makes it possible to resume interrupted [or deliberately suspended] burning process with accuracy high enough for DVD[-ROM] player not to "notice" anything at playback time. Recovery from buffer underrun condition in DVD-RW/-R case in turn is way less accurate procedure, and the problem is that the provided accuracy is very much what average player can tolerate. Now given that both provided and tolerated inaccuracies are proportional to respectively writing and reading velocities there basically no guarantee that DVD-RW/-R recording that suffered from buffer underrun will be universally playable.

Well, it turned out that I was wrong about one thing. I failed to recognize that DVD-R[W] groove also provides for adequately accurate recovery from buffer underrun condition/lossless linking. Not as accurate as DVD+RW, but accurate enough for splices to be playable in virtually any DVD-ROM/-Video unit. Yet! When it comes to DVD-R[W] recording specificaton apparently insists that you choose between

The specification asserts that the latter is achieved only in Disc-at-once recording mode and only if data-stream was maintained uninterrupted throughout whole recording. Once again. Even though most vendors implement lossless linking in DAO mode(*), full DVD-ROM/-Video compatibility is guaranteed only if recording didn't suffer from buffer underruns. The problem is that "offended" sectors are denoted with certain linking chunk appearing as degraded user data, few bytes, which are supposed to be "corrected away" by ECC procedure(**). DVD+ splices are in turn only few bits large and are "accounted" to sync patterns, not to user data area. So that even if suffered from buffer underrun, DVD+ sector is logically indistiguishable from DVD-ROM. Which is why it's commonly referred to that DVD+RW/+R combine DVD-ROM/-Video compatibility with [unconditional] buffer underrun protection.

As already mentioned, DVD+ groove has "addressing information modulated into it," ADIP (ADress In Pre-groove). This gives you an advantage of writing DVD+RW in truly arbitrary order, even to virgin surface and practically instantly (after ~40 seconds long initial format procedure). In addition, DVD+RW can be conveniently written to with 2KB granularity(***). DVD-RW in turn can only be overwritten in arbitrary order. Meaning that it either has to be completely formatted first (it takes an hour to format 1x media), or initially written to in a sequential manner. And it should also be noted that block overwrite is never an option if DVD-RW media was recorded in [compatible] Disc-at-once or even Incremental mode, only whole disc blanking is.

Unlike DVD-R[W], DVD+R[W] recordings can be suspended at any time without any side effects. Consider following scenario. You have a lot of data coming in [at lower rate], which is to be recorded into one file. Meanwhile it turns out that you have to retrieve previously recorded data. This would naturally require suspention of recording. Most notably in DVD-R [and naturally DVD-RW Sequential] case it would result in a hole in the file being recorded. So called linking area, most commonly 32KB gap, has to be introduced. So that you either have to wait till the file is complete or figure out how to deal with holey files. Thanks to ADIP, DVD+R recording is resumed from the very point it was suspended at. In DVD-RW Restricted Overwrite case no gaps are introduced, but if the media was formatted only minimally, suspension/resuming procedure has to be applied and it takes ~40 seconds to perform one. In DVD+RW case, suspension/resuming is instant regardless media state.

What does all of the above mean in practice? Well, I was actually hoping that readers would [be able to] figure it out by themselves. Apparently a couple of "guiding" words are needed... It means that it's trivial to employ DVD+RW for housing of live and arbitrary file system, no special modifications to target file system driver are required... Real-time VBR (Variable Bit Rate) Video recordings are children's game...

Sometimes DVD+RW/+R recording strategy is referred to as packet writing. I myself am reluctant to call it so (or TAO/SAO/DAO) for the following reason. Despite the fact that DVD-R[W] provides for lossless linking (within a packet/extent only), packets/extents are still denoted with certain linking information which distinguishes it (recording mode in question) from e.g. Disc-at-once. Now the point is that written DVD+RW/+R media, rather its Data Zone, does not contain any linking information and is logically indistinguishable from one written in DVD-R[W] Disc-at-once mode (or DVD-ROM for that matter).

It's maintained that signal from DVD+ groove (the one essential for recording, not reading) is much stronger, which makes it quite resistant to dust, scratches, etc.

Now we can also discuss differences between Double/Dual Layer implementations. DVD+R Double Layer permits for arbitrary layer break positioning yet maintaining contiguous logical block addressing. In other words address of the block following the break is always address of the block preceding one plus 1, even for arbitrarily positioned break. DVD-R Dual Layer on the other hand implies unconditionally disjoint logical block addressing [for arbitrarily positioned layer break that is]. This is because block addresses as recorded by unit are pre-defined by DVD-dash groove structure. In practice it means that file system layout has to effectively have a hole, which "covers" twice the space between chosen layer break position and outermost edge of the recordable area. And in even more practical terms this means that mastering programs have to be explicitly adapted for DVD-R layer break positioning. Unlike DVD+plus that is.

(*) According to Mt. Fuji draft buffer underrun protection is not even an option in DVD-R DAO: "If a buffer under-run occurs, the logical unit shall stop writing immediately and the logical unit shall start writing of Lead-out." Protection is defined in Incremental Sequential mode and DVD-RW context. By the way, note that earlier versions of this draft also discuss DVD+RW. You should be aware that they refer to abandoned version which has very little to do with DVD+RW/+R implementation being discussed here.
(**) ECC redundancy does permit for more degradation, more that this linking chunk that is, so that it hadly affects the playability.
(***) DVD "native" block size is 32KB, and 2KB granularity is nothing but a trick, but you're excused from playing it, i.e. reading 32KB, replacing corresponding 2KB and writing 32KB back.