Printable Version of Topic
Click here to view this topic in its original format |
The CD Forum > General > Pebuilder Cd Layout Optimization |
Posted by: aec May 22 2004, 09:04 PM | ||
I've been always much concerned about the relatively slow start up of PE disks. This feeling gets stronger whenever I launce a Knoppix disk which boots quite fast. OK, Knoppix is one but PEBuilder disks are as many as we are, each with its own personality. To come now to the topic I want to discuss let me list some of the factors that affect boot time for a given PC system 1. The shell we use 2. The services we want to have available before our shell starts. 3. The CD layout for system files, driver files, plugin files, ... A few comments: 1. We now have several shells we can choose from (nu2menu, nu2menu+geoshell, bluebox, litestep, aston, and explorer shells (XPE with localization, this-is-me's optimized, TrueExplorer, ...). I think we can find fast shells and I believe that those that today are slow will become faster. 2. The services we want to have ready when our shell starts of course "eat" more boot time. However, most of the shells can be configured so that some services are started later on demand. 3. Thanks to the efforts of the people like Bart, Allen2, Paraglider, Frodo, Sherpya, ... we now we have better and faster plug and play support with HWPnp, PePnp, bartpe, and Frodo's plugins. Now we can support more and more devices but for this we need more and more files. And of course, we now have tons of plugins for nice programs we want to use in our LiveCD. 4. Suppose now we have decided for what we want to have on our LiveCD. Taking into account the relatively small access time and transfer rate of CD Players, the question now is: Is there any way to optimize our CD? - Does it help to store large files compressed (upx or zisofs)? - Does it matter where our files are located on the CD? Well, I haven't performed any compression tests. This is something that I find interesting and I would like to know if someone has some results regarding speed increase at least for fast systems (decompression itself needs a lot of time). However, I have done a first experiment with the CD layout and found out that it is certainly important. We can get a faster CD with a proper layout pattern. The First Experiment I used PEBuilder to build an XP-SP1 based PE system with XPE 0.97 shell and several plugins making a 680MB CD. I created three iso's by running the mkisofs. The first one with the standard options. The second one by adding the option -sort sort_pos_file and the third one with the option -sort sort_neg_file. The sort_pos_file and sort_neg_file are files with positive and negative weight definitions for certain files and folders of our CD. The idea is that files that are needed during boot should be close together and preferably in the order that they are accesed. I used the positive weight definition list to put such files near the center of the CD. Similarly, I used a negative weight definition list to put these files at the outer parts of the CD. I burned three CD's, the neutral, the positive and the negative. I tested them on my system and measured the time needed for XPE to boot. The results are:
My sorting list is just a first experiment. I share it in my next post, together with some instructions, hoping that it can be of use to anyone who wants to save some boot seconds and also wants to help to improve it. |
Posted by: aec May 22 2004, 09:07 PM | ||||
The mkisofs command line
Note: This command line is for the unpatched mkisofs. If you use bart's patched version then lowercase BOOTSECT.BIN. In our example The pebuilder @OutDir@ is C:\PEBLDR. Important!! It took me two days to find out that the sort option does not like backslash "\". I was thinking that -sort is not working. So use "/" in the command line and for the paths listed in the sorting list. Also the @OutDir@ path in sorting list should exactly match whatever you use in command line. e.g., if you use in command line "./PEBLDR" then you should replace "C:/PEBLDR/" with "./PEBLDR" everywhere in the sorting list. The sorting list pe_sort.txt. It consists of two columns separated by only one space or tab. The last column is a positive or negative number with nothing else after it. Filenames with higher positive weights are ordered first. Default weight is 0 and the order is alphabetical. Negative weight are ordered after 0 weighted files. The list was constructed using - info that is used by the defragmenter on a real windows system (windows\prefetch\layout.ini) to optimize disk layout and - information from some boot tracing with sysinternals regmon.
Try it and help to improve it. |
Posted by: mbarnes May 23 2004, 12:53 AM |
I use Nu2menu & Geoshell V2a (for the task bar), I only start up Explorer when or if needed. One thing which is slow is registering dll's, possibly 60 or 80 seconds out of 4 or 5 minutes boot up (for 36 dll's). I am going to have to find out if some dll's really need to be registered. (for USB , mmc & VBS plugins) Running infcache builder (by Paraglider or Sherpya) makes a significant reduction in boot up time. keep up the good work aec regards Mike Barnes |
Posted by: aec Jun 21 2004, 11:38 PM | ||
Follows a second version of the sorting list, slightly improved
|
Posted by: Triton Jun 23 2004, 04:57 AM |
Anymore results? |
Posted by: DigiWiz Jun 23 2004, 07:33 AM |
First I want to say I think your project is a great idea, and appreciate the work you've put into it... but I am unable to duplicate your results. I am usings Bart's patched version of mkisofs, and followed your instructions exactly. I created both a sorted and non-sorted version numerous times, but always had identical results. Boot time for both is 87 seconds (62 seconds until the autoRAMsizer pops up - another 15 seconds until it's done for a total of 87s)... Finally, in desperation to find out why sorting did not sem to help, I did a file compare, and unfortunately, except for the time stamps in the first few sectors, the ISO builds were identical. It would appear then, that perhaps mkisofs included with PE Builder does not support the sort function... any ideas or confirmations of this anomaly? DW |
Posted by: aec Jun 23 2004, 09:27 AM | ||
@DigiWiz
I believe its only a problem of the correct combination of Upper/Lower case, together with slashes/backslashes. At the begining I had a lot of trouble with all these things and to check if the sort option was working I wasted several cds and finally resorted to hex file comparison. I am using an unpached version of mkisofs on a directory tree where toplevel files and the "I386" folder/subfolders are uppercased. I checked that this procedure also works with the new win32 port of cdrtools that is not using cygwin1.dll and incorporates storage optimization (dfl is not needed any more) (link http://cdrtools.bootcd.ru/). So far I have not tested it with Bart's patched version. I already have an idea of what may be wrong with Bart's patched version, but give me some time to check it first. I think your CD is already loading very fast . I guess you are using Nu2Menu as shell and not doing much pnp detection while booting. |
Posted by: bobby100 Jun 23 2004, 06:35 PM |
@aec Wud you give me the permission to include your sorting list with final version of The_GUI v2 ? |
Posted by: aec Jun 23 2004, 07:41 PM | ||
@bobby100
Be my guest . @DigiWiz After some tests I have found how one can use the sort option with the patched mkisofs version. Both mkisofs versions, patched and unpatched, are actually case sensitive as far as the sort option is concerned. Bart's version, just stores filenames in uppercase (everything in the burned CD is uppercased). However, sorting takes place before burning and therefore the filename case is important. Practically this means that in the sorting list the filenames are written preserving their case (propably mixed), turning backslashes to slashes. E.g. the C:\pebldr\i386\system32 folder should be listed as C:/pebldr/i386/system32 and not as C:/PEBLDR/I386/SYSTEM32 The rule is: Respect case and use slashes. So if you want to use Bart's version you have to modify appropriately the filenames in the sorting list so that they match the case used in your directory tree. You may use @bobby100's program "The_GUI v2" to construct such lists. |
Posted by: d4vr0s Jun 24 2004, 01:49 AM |
Ok, after a little fight with the new mkisofs.exe and making the entire /I386 directory uppercase so it would boot again I made 2 ISOs one with sorting and one without. Testing with vmware, I found the one with sorting booted 20 to 25 seconds faster. |
Posted by: Triton Jun 24 2004, 06:24 PM |
Is there a link for The_GUI v2 and does this prog build lists? |
Posted by: Ixel Jun 24 2004, 11:02 PM | ||
I guess this is the case for me, too. This process only shaved off 2 seconds of the boot time for me (using bart's pathed mkisofs). Still, great investigative work aec. Thanks for sharing this info. Ixel |
Posted by: bobby100 Jun 25 2004, 01:33 AM |
@Triton Thread about The_GUI is here: http://www.911cd.net/forums/index.php?showtopic=7261 |
Posted by: aec Jun 25 2004, 06:55 AM | ||||||
Here is a version of the sorting list that can be used with the patched mkisofs version included with pebuilder. Actually the modifications affect only the filename case. I created it while beta-testing @bobby100's "The_GUI_v2". It is assumed that your pebuilder @OutDir directory is at C:\pebldr. You should make below the obvious modifications to match your @OutDir. The mkisofs command line (defined near the top of pebuilder.inf) should be modified from
to
Create a text file named "pe_sort.lst" and put it in the pebuilder folder, where also mkisofs.exe is. The contents of the pe_sort.lst are
I would like to remark that if you are using the patched version of mkisofs, it is very important to respect the filename cases. So if you rename your i386 folder to I386, then this list above will not work. You will have to make the appropriate modification in the list. This also means that you have to check yourself all the filename cases listed in the above list. My advise, is to use the new unpached version of mkisofs, uppercase the full i386 tree and a few top level files, and the previous version of sorting list which mainly contains uppercased entries. |
Posted by: daederatz Jun 30 2004, 02:05 PM | ||||
Hi there, I followed this topic with great interest, but i have to wonder why none of the posters have mentioned the obvious: use cdimage instead of mkisofs for ISO creation. cdimage.exe (v 2.47) is a tool made by MS for internal use that can be found on the net, just google for it. Legally it may be marginal to use it, OTAH it's not sold, so there's no financial loss for anybody. One of the programm options is -y5 which means "write \i386 directory files first, in reverse sort order" There are many more options, most notably the one to reduce the iso size by removing duplicate files. My complete build command is:
This works really great, resulting in visible faster boot time then with mkisofs (can't measure exactly now, being at work). One thing though, out of precaution I use "filecase.exe" to make certain files on the CD uppercase. There should be other topics on this, here is just the code for my layout:
I remember that my CD worked even without the uppercase thing, but I read elsewhere that there could be problems with certain programs, so it's really just to be safe... Note that inside my code snippets you must replace everything between < and > with your own variables / hardcoded paths respectively. You may have to fiddle a bit to get it all right, but thats what we are here for, isn't ist? |
Posted by: bobby100 Jun 30 2004, 03:28 PM |
@daederatz Mkisofs also can remove duplicated files |
Posted by: daederatz Jun 30 2004, 05:45 PM |
@bobby100 IIRC this needs another external program (dfl.exe?) I tried this once and at first it seemend to work, but then I received weird errors on further runs. Could be because my CD consists of more than Bart's PE. I actually build a compilation with the best stuff from several BootCD's (Hiren, UBCD, some Antivirus apps and a lot more little proggies). This other stuff changes less frequently then the PE part, so I usually run PEBuilder, then copy it's files & dirs over to the root of my BootCD and then build the ISO. If I do so and run "dfl.exe", effectively for the 2nd time on the same directory tree, with only the PE dirs/files changed, it doesn't work any more (complains of "broken directory structure" or something). So I really have given up on this method and switched to cdimage. Works great so far. Or is there is another, build-in mechanism for this task in mkisofs? --- Tim |
Posted by: aec Jun 30 2004, 06:59 PM |
@daederatz There is a very new win32 port of mkisofs with internal storage optimization (same files stored once). dfl is not needed any more (link http://cdrtools.bootcd.ru/). The other advantage of mkisofs sort option is that you can instruct the program to put this file first, the other next and so on. I will try the cdimage -y5 option, which however only affects storing \I386 files in a predefined way. In other words, you dont have fine control on sorting. Thanks for the cdimage info. regards |
Posted by: philon Jun 30 2004, 10:50 PM | ||||
@aec Thanks for your effort to decrease booting time! Gave it a test, here are my results: CD-R/DVD drive: unsorted 167 sec sorted 155 sec difference -12 sec Also tested CD-RW drive: unsorted 152 sec sorted 139 sec difference -13 sec Never used the CD-RW drive as boot drive before and I'm happy to see a 25 sec difference now between CD-R unsorted and CD-RW sorted. I use a mini CDs filled up to the limit. For comparison purpose burned the same sorted content also on a maxi CD to see how much the outer circles are faster, but gained only 4 sec. My CD is a multiboot CD (EasyBoot) so I had to modify the (patched) mkisofs accordingly (ezboot folder beeing in root):
In the sortlist I changed the first lines:
philon |
Posted by: daederatz Jul 1 2004, 08:26 AM |
@aec thanks for the info on the new mkisofs version, sounds very interesting and I'll run a few tests over the weekend. My actual BootCD really depends on storage optimization (would be 20% larger without it and > 800MB, according to cdimage). I'll post my findings & boot-up times... --- Tim |
Posted by: paraglider Jul 1 2004, 02:06 PM |
I use dfl and the -follow-links -cache-inodes options to mkisofs. It produces iso files about the same size as cdimage. I would not use special versions of mkisofs as you will get left behind by the frequent updates to the cdrecord+ mkisofs package. |
Posted by: daederatz Jul 1 2004, 04:06 PM |
@paraglider and you don't have any problems with dfl.exe? Any hints what I may have done wrong to receive the "broken directory structure" error on the 2nd run of dfl.exe (as described in my earlier post above)? I also used -follow-links -cache-inodes with mkisofs. Am I right to assume that it doesn't like to run more than once on the same directory tree and if so, do you know a solution? I'll surely test the various methods during the weekend. If aec's sorting method with mkisofs really speeds things up for me, then it might be worth to fix the trouble with dfl.exe. Otherwise I'll probably stick with cdimage, seems a bit more straightforward to me... Thanks for your help (and, BTW, for your great PE tools ) |
Posted by: paraglider Jul 1 2004, 05:25 PM |
I haven't tried running it twice or moving the the files after creating the directories. For multiboot with ezboot I create ezboot as a plugin to pebuilder. That seems to work fine. |
Posted by: daederatz Jul 1 2004, 06:23 PM |
Ahhh... yes . I must admit that the whole concept of creating a plugin of a Multiboot Programm for PE eludes me completely. Shouldn't this be the other way round, i.e. making PE work with the Multiboot Program? I use BootCDWizard to multiboot and stumbled over a plugin for PE - didn't understand this either . I just use the standard BCDW (bcdw150z_en.zip) to boot all my apps, DOS, PE, WIN or Linux, from my CD. Time to learn a few new tricks, maybe... --- TIm |
Posted by: Rickj Jul 1 2004, 10:42 PM |
I have been making these multiboot CD's for a while as well. Using Bart's patched mkisofs+dfl and mkisofs(mingw32) seem to work about the same to me. M$ cdimage also works well, but creates a cd image about 1-2 MB larger. Each program seems to have some minor downfalls. mkisofs+dfl modifies the base directory structure and makes it difficult to add to your build directory. mkisofs(mingw32) may not be updated as frequently but allows you to add to your build directory and use a sort list. cdimage is very stable and dependable but doesn't allow for a sort list. I'm testing different sorting methods and will post my results when I'm done. Keep up the good work. Thanks to All! |
Posted by: daederatz Jul 2 2004, 02:47 PM |
Here are my findings, comparing the various methods from this thread. Setup: - Iso's where written with NERO 5.5 (DAO/96) - BootCD consists of PE & other stuff, multiboot with BCDW - all files are uppercase - time measured: from hitting "Enter" on the PE entry in BCDW's boot selection screen to the point when the Network Dialog of PE pops up - PE Shell is nu2menu & Geoshell 2.0a, no plug&play detection - ISO apps: mkisofs.exe (Win32/Mingw32) v2.01a32-3-w32 / cdimage v2.47 - storage optimization was always used with both apps Result: mkisofs without file-sorting 141 sec mkisofs with file-sorting 136 sec cdimage 145 sec So, we have a winner Seems that mkisofs does a better Job than cdimage and file sorting makes a small difference too. Wasn't expecting this, I think I'll switch back to mkisofs... The only thing left to try would be the "standard" version of mkisofs in conjunction with "dfl.exe", maybe I'll do this later. --- Tim |
Posted by: nitro001 Jul 2 2004, 06:46 PM |
I also am trying to optimize my CD, however, I started using the mkisofs that comes with PEBuilder. I then tried the newer mkisofs from the web site, but when I build using that one, I get the message: CDBOOT: Couldn't find NTLDR. I have tried with/without sort, etc, although the new version on the site no longer supports the -follow-links -cache-inodes parameters, but now has a -duplicates-once switch. I have tried with and without this switch with no luck. Although it does create an ISO MUCH smaller than without it, almost 20MB. The version that comes with bart, builds the disk fine if it is SP1, but with no compression. I do see references on this page about dfl.exe, where do I get that from? and is there a good description of what it does and how to use it with PEBuilder somewhere? Also has anyone seen where you boot a PE disk on a pc with only 64MB ram, it will not create the Ram Drive? Thanks, Nathan |
Posted by: paraglider Jul 2 2004, 10:34 PM |
Just search on these forums for dfl. Make sure to search all dates not just the last 30 days. You also need to convert the i386 subtree to uppercase along with a few other files. Again its all been described in these forums. |
Posted by: daederatz Jul 3 2004, 12:15 PM |
Last update: I'm afraid "dfl.exe" just doesn't work on my current cd-layout. I suspect there are actually too many similar files, I keep getting "Can't create symbolic link: ..." & "Errors found, tree corrupted..." errors. I set all the files up cleanly, i.e. removed all symbolic links that might have been there from earlier runs, but still no luck. So for the moment I'll use the http://cdrtools.bootcd.ru/ together with aec's sort-list. I need the storage optimization of this special mkisofs version and sorting speeds the thing up a little bit. Thanks to all, keep up the good work! --- Tim |
Posted by: aec Jul 3 2004, 01:48 PM | ||
@daederatz Thanks for your feedback.
Just a question. What was the type and the speed of the CD that you performed this test? Have you tested your set of cd compilations in another machine with a slower or faster CD reader? regards |
Posted by: daederatz Jul 4 2004, 01:26 AM |
@aec The CD-Burner was a Lite-on 16x-speed, the reader (from which the CD's booted) was an LG drive 32x-speed. Unfortunately I have already dumped the CD's, so no further tests on other machines are possible . --- Tim |
Posted by: samplempg Jul 4 2004, 04:18 AM |
hello aec i have a silly question...how do you use regmon trace boot files and time into pe_sort.txt while machine is booting ? could you share the secret with us ? |