What’s A "Compaction Thread"?


I’ve received a few questions from users who have said that their WM5 device seems to slow down and speed up erratically.  (To be clear, we’ve only heard reports of this happening on two upgrade devices, and not on any of the devices designed specifically for WM5.) Some have dug in and found that “filesys.exe” is using a lot of CPU power and that it appears to be running something called a “Compaction Thread.”  There have also been questions about a registry key called CompactionPrio256. I can’t explain some of the bigger problems people are describing, but I can at least tell you what these things are and what they’re trying to do.


This entry assumes you know the difference between RAM and ROM and that you know what I mean when I say “NOR” and “NAND.”  If not, please read this first.  I’m also assuming you know what Persistent Storage is.  If not, please read this.



It’s easier to move on, than to forget


Flash ROM is divided up into “blocks.”  Each block is then subdivided into “sectors.”  Sectors tend to be 512 bytes each.  Blocks vary in size, but are often around 128K.  The most difficult thing to do with flash is to get it to forget what it knows.  In most storage systems (hard drives, RAM, pencil and paper, etc) you can erase as much or as little as you want.  That’s not the case with flash ROM.  If you want to erase one bit of information in flash, you need to erase the entire block that it’s in.  On NOR flash, erasing one block can take up to two seconds (NAND is much faster at erasing). 


As if the long erase time weren’t motivation enough to not do it very often, there’s an added problem that a flash block can only be erased so many times before it wears out and stops working. 


Here’s how we deal with these problems.  Whenever we need to change something that’s in a sector, we copy the sector into RAM, change whatever needs to be changed, and then write it back to a new spot in flash.  Then we mark the old sector as invalid. 


This does a few things.  First, it makes most writes much faster (we don’t need to wait two seconds for the block to erase before we write the new data).  Second it spreads the data around, which results in spreading the erases around.  Rather than erasing one block ten times, you erase ten blocks one time each.  (This is called “wear leveling.”) 



Trash compactor


So, life is peachy.  We just mark sectors as invalid and write the data to unused sectors.  Unfortunately, this largesse of spending eventually catches up with us.  At some point we’ll run out of unused sectors.  Now what?  If we were a government, we’d just print more sectors and keep running.  Alas, we’re not.  “Deficit sectoring” isn’t allowed here.


What we do is have some code that keeps an eye on the state of the flash and the state of the system.  If the flash starts getting full of invalid sectors, and if the system seems to be idle, this code will kick in and start “compacting” the sectors.  If there are any blocks in which all of the sectors are invalid, it’ll erase them.  If there are blocks that have a small number of valid sectors and a bunch of invalid sectors, it’ll move the valid sectors elsewhere, and then erase the block. 


We call the code that does this the “Compaction Thread.”  The compaction thread lives in filesys.exe, which is one of the primary components of Windows CE (it’s the file system).


The goal of the compaction thread is to keep free blocks around at all times so that all writes work, but to do it in a way that you, the user, don’t notice.  So, it runs only when the system is on but idle.  Unless…



Emergency!


What happens if we do a ton of writing to flash and fill up all the flash blocks, but the system is never idle enough for the compaction thread to run?  Eventually we’ll go to write something, discover that there are no free blocks, and declare a state of emergency.  The file system will then impose martial law, take over the CPU, and run the compaction thread long enough to erase a block and free up a sector to be written to.  We call this “Critical Compaction.” 


Due to the way the hardware works, flash memory can not be read from or written to while a block is being erased.  So, regardless of how we got into the compaction thread, the system isn’t going to be very responsive while we’re erasing a block.  (And, remember, NOR flash can take up to 2 seconds to erase a block.)



What can I do about it?


Compaction is a result of data being written.  Reading data doesn’t cause compaction.  If you’ve got an application that is writing data frequently, especially if it’s writing it in small chunks, that app is going to make you compact frequently.  (Here’s a great blog entry that explains why writing small amounts is bad.)  Also, the less free space the flash has, the harder the compaction thread is going to need to work to find blocks to compact.  So if you’ve got your storage space maxed out, you’re going to hit compaction more often than if you have spare space. 


Why does this happen on WM5 but not WM2003?  Compaction is only needed when data is stored in flash ROM.  WM2003 PocketPCs stored their data in RAM, which is much faster and doesn’t need to be compacted, but has the disadvantage of being erased if you lose power.  (Note that all Smartphones since the original 2002 release have had persistent storage and have done compaction.)


Why are Axim users seeing this so much more than everyone else?  That I don’t know.  The compaction thread is the same on the Axim as on every other WM5 device.  It would seem that the Axim is doing something different than other devices.  I don’t have visibility into their design and can only speculate. 


Why would a NOR user see this more often than a NAND user?  NOR takes much longer to erase a block than NAND does.  It’s possible that both are compacting, but the NAND device got done quickly enough that you don’t notice. 


Why would this possibly run for longer than 2 seconds?  I’m at a loss there.  Maybe something is continuing to write data and is forcing more compaction?



What about these registry keys?


Someone told me that changing the registry keys “CompactionPrio256” and “CompactionCritPrio256” fixes the problem.  Unfortunately, that can’t be correct.  There are two things you need to know.  First, in Windows CE, smaller numbers are higher priority.  Compaction defaults to 255, which is the lowest possible priority.  Making the number lower would increase the priority.  That’s the opposite of what you think you want to do.


More importantly, though, changing those registry keys doesn’t do anything. It is the file system that loads and creates the registry.  But the file system needs some information before the registry is loaded.  So there is a very small bit of data (a “boot strap” registry) that is loaded before the user changeable registry is loaded.  The real versions of these values are in that pre-registry.  The ones you see are kind of like reflections of those real values.  You can change them to your heart’s content, but you’re only changing the reflection.  You’re not changing the actual values the file system uses.  There’s no end user way to change the real values.



<braces for impact>


Hopefully I answered some of your questions.  I know that I didn’t answer the question you really care about though.  “When are you going to fix this?”  I’m expecting a lot of angry comments as a result.  I’ll apologize up front that I don’t have an answer to that one.


Mike Calligaro 


Comments (241)

  1. badbob001 says:

    Given the fact that nor takes 2 seconds to erase, would you still recommend the device rom be on nor or nand? From the axim x50v to the x51v, dell has made the move from nor to nand. First I thought it was for costs reasons since nand is cheaper, but maybe there is more to it…

    On my dell x50v, which has a nor rom, I’ve changed some registry entries to move the temp folder, ie cookies, ie history, and ie cache to my SD card (nand). Good idea?

    Someone at microsoft should take a good look at what temporary files are used by WM5 and write a powertoy to alter the system so those files are written to external NAND storage.

  2. MikeCal says:

    It’s hard to make a straight recommendation.  NAND has some advantages over NOR, but NOR has other advantages over NAND.  NOR is much better to execute code from, and NAND is much better to store data in.  The ideal is to have the system code stored in NOR and the user data stored in NAND.  That, by the way, is what the Treo 700w did.  

    All SD cards are NAND, and, though compaction is done on SD cards, the OS doesn’t have to do it.  So, yes, offloading your temporary files to an SD card can speed things up.  The downside is that, if you eject the card, you will lose your cache and need to regenerate it.  That won’t cause a crash or anything like that, but it will impact your performance.  If you always leave your SD card inserted, though, it’s an idea that should provide some benefit without much downside.

    Mike

  3. Patrick Y. says:

    Hi,

    I really hope that you can help to look into the issue with x50v. The behavior of filesys.exe on x50v is completely different from what said it should do. The compaction thread normally runs more than 15 minutes straight on and causing 100% CPU usage. A random file writing process such as syncing, caching, etc. immediately triggered the compaction thread again. Please help us with the problem.

  4. Lat says:

    Hi Mike. Thanks for the blog.

    Like Patrick Y., I’m yet another x50v with the WM5 upgrade user that suffers from the compaction problem. What he says is true, and we do suffer from the compaction thread seriously affecting performance. Our workaround thus far has been a little patch that somebody from the forums we frequent (aximsite.com) made that suspends the compaction thread while not on AC power.

    You can take a look at the following threads as to what our only option is:

    http://www.aximsite.com/boards/showthread.php?t=106009

    http://www.aximsite.com/boards/showthread.php?t=112194

    Anyways, if you had time to skim those threads, what do you think of the suspending the compaction thread? Would it cause excess wearing to our ROM?

    We at Aximsite are all hoping that Dell will release a new ROM update, but despite many promises by Dell customer services, no avail…and so we have to take the problem into our own hands and try to solve it.

    Of course, experts like you i’m sure will help us along a lot. Hint hint =)

    Thanks!

  5. Dan L says:

    Same for me, another Axim x50v sufferer here.  Oddly enough, I did notice a very tangible improvement in speed after deleting the compaction registry keys.  I know it shouldn’t have made a difference, since those keys are the "reflection" to the real ones.  I’ve also been using the patch_filesys that shsmith provided in the Aximsite forums, and again it seems to work.  Although from reading the blog, suspending the file system may not be a good idea anymore.

  6. Beer says:

    Can we set "compaction thread" to be manual likes "Defragment" does???

  7. Nino.Mobile says:

    TGIF!

    Software / Hardware

    Sling Media has released SlingPlayer for Windows Mobile (via msmobiles.com)&amp;nbsp;…

  8. BG says:

    Hx4700 also have this problem

  9. Avatar says:

    Confirm.

    hx4700 has this problem. After fiddling with the registry, increasing cache and stuff found that wm5.0 is a VERY FAST os, except when compaction happen.

  10. Neo says:

    I’m another one who suffered from upgrading PPC2003 to WM5. Following the steps on adding a custom program to constraint filesys.exe and modifying the registry entries did help to improve the performance.

    Everythings turn out to be faster, and error free

  11. Avatar says:

    I have few questions:

    Why on a hx4700 the registry setting for "CompactionCritPrio256" is 0xFFFFFFF ? Is this an undocumented value or error ?

    Is there some way to read more information from the CompactionThread – free blocks, blocks freed and so?

    It’s almost impossible to work with a device that does compaction 40% of it’s time. Compaction is not a cheap (cpu and battery) process. Almost any program, even the wm5.0 designed ones are using the / partition more than the storage, because under wm2003 it was times faster – for example ie cache, opera cache and many more.

    Is it possible to decrease the strata flash block size? Now it’s about 255kb, wich is realy huge, for example if 20 files (small writes) force the compaction process to run, it’ll need (at worse) to move 2x255kb?

  12. Avatar says:

    I’m sorry to comment again, but I have found few other issues with my hx4700:

    The block size reported by the HP Assets program for the strataflash is 128kb, the registry says 0x40000 (HKLM/Drivers/BuiltIn/StrataFMD/Blocksize). Is this correct?

    On a nearly empty storage I install two apps (it takes about 2 minutes), after I stop, I leave the filesys.exe to finish it’s job, it takes 10 or more minutes to complete (I don’t suspend it, lower it’s priority and etc.)

    I think that there is a problem in a range of pre wm5 devices (axim x50 and now hx4700 series) that oems (dell and hp) do not acknowledge.

    Thank you.

  13. hx4700 user says:

    I am also an Ipaq 4700 who upgrade to WM5 and have this problem…any way to solved this problem yet?

  14. tbmorris says:

    I have also "upgraded" my 4705 to WM 5 and found it completely unacceptable, mostly due to the speed issues which did not resolve with the tweaks.

    I will simply stay with my previous OS until MS and HP can get their act together.  I am quite surprised that they would even release such an upgrade.  It makes me think that they never even tried their own products before releasing.  

    It is a wasted $39.95 for me.

  15. Sunny says:

    I disagree with the threads should be set at a priority of 255. It’s too low that any activity can stop the compaction thread from running. This simply decreases the efficiency of erasing those deleted blocks and increases the probability of triggering the critical compaction. Having a appropriate priority level may improve the overall efficiency. OR just add a tool to do it manually. Of course, critical compaction is always essential.

  16. Chung-Ho Liu says:

    M-systems’ DiskOnChip is actually Nand flash with NOR-like interface. Almost all of HTC’s MS mobile phones use DiskOnChip for code storage now. I think DiskOnChip can also be used for data storage in the future.

  17. leek says:

    Aren’t we simply talking about a form of garbage collection here?

    "Garbage collection" is the right term — not "trash compaction", although I understand this article may be using it simply because that’s what WM5 calls it.

    Whether the cleanup of blocks should be deferred (lazy) or immediate (aggressive) is not always clear-cut. Sometimes it’s better to take 2 seconds and clean everything up, then proceed. Other times it’s better to do it in the background when nothing else is happening.

    Also, the user MIGHT be able to change settings, but only if they are into hacking ROMs, like our friends at http://www.xda-developers.com .

  18. MikeCal says:

    Leek, although I said "Trash Compactor" as the silly heading to one of the sections, we don’t call this "trash compaction."  We do call it the "compaction thread" though.  It’s similar to "garbage collection" but not precisely the same.  A garbage collector usually goes through, finds memory that’s no longer refrenced, and frees it.  The compaction thread does both less and more.  On the "less" side, it doesn’t need to do any work to find out if a sector has no refrences.  It’s simpler in that the sector has been marked as dirty/unused.  On the "more" side, the compaction thread is doing wear levelling work that a typical garbage collector doesn’t need to do.  Garbage collectors usually work on RAM, where wear levelling isn’t needed.

    Still, what a garbage collector does and what a compaction thread does are pretty similar.  In simple terms, they both free up unused chunks of memory so they can be used again.  So, if you want to call this a "garbage collector," I won’t complain.

    Mike

  19. Solnyshok says:

    What I found on my HX4700 with WM5 is that keeping more PS free (i.e. installing programs on SD/CF instead) results in much faster response, that I assume is a result of OS having less need to start compaction thread.

  20. Galt says:

    Over at CoreCodec.org forums (for TCPMP aka Betaplayer) many WM5 users have noted reoccuing pauses (every few minutes) in video/audio streaming.  Could this be due to this issue?  Is there a programming strategy the devs can use to avoid this?

  21. solnyshok says:

    Galt, this issue occured even before WM5 on tcpmp (betaplayer) doing video over wifi. It should be related to slow wifi (b) and inability of current pocket pcs to handle wifi data receiving AND high bitrate video at the same time. Hence, delayes while buffering next 2 minutes of movie from the air. I think compaction has nothing to do with it.

  22. stub says:

    what’s a key "NeverDorkMemory"? anybody knew about it?

  23. Avatar says:

    NeverDorkMemory is only for 2k3 devices, basicly it’s "do_not_move_the_storage_ram_slider_automaticaly".

  24. jugagagah says:

    pls release a patch to move PS to SD card? is it a good solution(for user has relative large free space always inserted SD card) on x50v with wm5? thanks

    by the way, x50v, x51v, wm5 bt stack is lacking of profiles too

  25. Avatar says:

    I was tweaking with the registry and probably trying to mount as root a cf or sd card will be interesting solution to the problem. However I’m not sure that this could be done via the registry, without ‘rebuilding’ the image.

  26. JiH says:

    Why if I start work with system the  Compaction Thread does not finish its work only with current block which is processed…

    :-( Aaargh :-( This process (after its start) works for a long time. It works independently of the fact, if a user starts to work with the system, or not.

  27. Ahmed says:

    "Galt, this issue occured even before WM5 on tcpmp (betaplayer) doing video over wifi. It should be related to slow wifi (b) and inability of current pocket pcs to handle wifi data receiving AND high bitrate video at the same time. Hence, delayes while buffering next 2 minutes of movie from the air. I think compaction has nothing to do with it."

    I found personally that the same bitrate media that was working fine under wm2k3se will not work properly under wm5 on my iPaq 4700 using the same version of TCPMP.  I should note that while TCPMP is buffering, the WinCE color wheel indicating the system is busy would show, whereas the intermittent pause that occurs under wm5 does not show the same busy indicator.

    Since TCPMP utilizes a buffer while streaming, is there a chance that this triggers WM5’s compaction?  The operating system saves the memory state to ROM, so why would a media buffer stream be treated any differently?  If that’s the case, then surely WM5 is compacting the ROM every so often to keep up with the constant rewrites to the media buffer.

  28. JS says:

    Hi Mike,

    Is there a way to store data in RAM in WM5 just like wm2003 did? Sometimes we need to write some temp files or log files. If we can store these files in RAM, it will reduce the use of Flash ROM and the "Compaction" time will also be reduced.

  29. badbob001 says:

    If compaction on storage cards is done by hardware, why can’t hardware compaction be used for the rom? Perhaps the same reason I’ve never heard of a self-defragging harddrive.

  30. MikeCal says:

    Jugagagah, it’s possible to mount the internal storage on an SD card.  We did this during WM5 development to let us do work on a device that didn’t have enough flash available.  However, the system can’t handle it’s persistent storage being removed on the fly.  If you were to have your storage mounted on an SD card and you ejected the card, the system would crash.  That was acceptable for development, but we’d never make end users put up with it.  Even if you personally found it acceptable, it’s not a change you can make yourself.  The image itself needs to be changed, and only your OEM can do that (not even Microsoft can).  You might think you could make a registry change or something like that, but remember that we’re taking about the storage system here.  The registry is IN the storage system.  So it’s loaded after the storage system is mounted.  The change needs to be more fundamental than that to take effect in time.

    JS, all new WM5 devices are required to use persistent storage, but we gave OEMs the right to do RAM based storage on upgrade devices (devices that they upgraded from WM2003 to WM5).  However, that’s an OEM decision and not something an end user can change.

    If you just want a space for temp files, and that space can be cleared every reboot, then you want a RamDisk.  I did some investigation into RamDisks a while ago:

    http://blogs.msdn.com/windowsmobile/archive/2005/07/20/441038.aspx

    but I’m not aware of any changes since then.

    Mike

  31. oxynorom says:

    Thanks for the post. Most users does not aware or really care what cooking "behind the scene", until problem surfaced.

    The hardware had its limitation, but DMF needs to take responsibity to tell their customers that their devices may be not up to the task at hand, and leave the decision to the users.

    Like : "You have been warned" then I think there will not be as much complains except for the ignorance.

  32. Jeff Kelly says:

    WM5 killed my hx4705. As the chief engineer of a major broadcasting company, I keep everything on my PDA. With WM2003SE, I had issues like GWES.exe growing until I ran out of RAM. So, I purchased WM5, hoping the repurposed strata flash and new OS would resolve the issues. At first glance, the OS is pretty slick and as others have said, it is fairly fast after tweaking a few cache parameters. Enter compaction….the process would run when I turned on the PDA or about every 10-20 minutes. It was so debilitating, I couldn’t use the PDA because 80+% of my CPU suddenly became committed to filesys.exe. I lasted two weeks and had to switch back to WM2003SE. The PDA is much more stable (and faster) now as I installed everything on my SD card, leaving the strata flash virtually empty. I’ve got 10 gigs of storage on my PDA, so retiring 80 MB of strata flash is no big deal.

    I’m debating a refund. It seems there is no real solution here because the issue is really about the type of flash in my ROM chip. That being said, can’t Microsoft inhibit compaction and allow owners to use a 3rd party defrag utility when it is convenient?

  33. JS says:

    Hi Mike,

    Thanks for the information about the RAM disk. I haven’t tried buzz’s stuff yet but I think it’s a good start. As a developer I would like to have such a RAM disk built-in in the system with the %Temp% folder pointing to it, maybe in WM5.0 SP1?

  34. Richard Aarnink says:

    Well, I am confused. I read a few articles now and hopefully somebody can help me understand.

    I read that in WM5 you can no longer install programs in to RAM. My HX4700 has 128MB ROM, but this is a NOR type  ROM which has the problem mentionted with Compaction (slow writing performance).

    But than again, When I have a fixed set of programms in use, there would be no need to write? When installing additional or replace programs it is required to write the ROM memory.

    WM5 and all other programs would need to write temp data on another location, i.e. a SD card.

    Would this solve any problem, or am I missing a crusial thing here?

    —> A other issue is lack of memory in my HX4700. I am thinking of expanding it to 128 MB. I am not sure If this would affect the ROM or just the RAM.

    When ROM is affected. Should I ask for NAND memory instead of NOR? Or should this issue be resolved in the software?

    Furthermore, is it wise to expand the RAM memory anyway for WM5, or is there no need because RAM is onl y used for running programs, not installing into RAM?

    Many questions, hopefully someone is able to shine a lite on the issues I mention.

    Many thanks.

    Richard Aarnink

  35. ILikeWM5 says:

    The SD card mounted as internal storage worked great on my MPx200 w/ WM5. You know, the one they said couldn’t be upgraded?

  36. MikeCal says:

    Jeff, the problem with inhibiting compaction is that the thread isn’t just defragmenting.  It’s also wear levelling.  If we didn’t wear level, then a single block of flash could get written repeatedly and wear out.  Wear levelling is designed spread the writes out, thus keeping the flash alive longer.  Though, clearly, the unusable condition people are reporting isn’t good either.  An unusable device isn’t much better than a worn out one.

    JS, I’d also like to see us mount a RAM filesystem under %TEMP%.  On some devices, at least, since people complaining about having too little RAM wouldn’t want more taken up by a RAM disk.  

    Richard, you’re right that all storage is in Flash ROM now.  So applications don’t get installed to RAM anymore.  The main issue with compaction threads, though, isn’t about programs being stored but data.  As you said, once you’ve installed your programs, they don’t change much.  But your data (email, etc) changes frequently.  

    Regarding lack of memory, I’d need to know what you’re running out of.  Do you mean storage space or program RAM.  WM2003 used to store data in RAM and, typically used half the RAM to do this.  Since WM5 stores data in ROM, you’ve effectively doubled the amount of RAM in the device.  And, as you said, you’re using the RAM for fewer reasons now than you used to.  I’d be surprised if there’s a need to upgrade your device with more RAM.

    ILikeWM5, please see http://blogs.msdn.com/windowsmobile/archive/2005/10/19/482683.aspx for a discussion on where the WM5 build for your device came from.  

    Mike

  37. ILikeWM5 says:

    Thanks for the link.  While I’m only a "normal" user, I really like this blog.

  38. Gary Byron says:

    Mike, thank you so much for this insight into the problems regarding WM5. My OEM has never pubically admited the rpoblems, and I’ve spent several months complaining about the infamous lack of support. This problem you have finally accepted blame for, so I will adress my complents to you.

       Much of this could of been solved several months ago if more of WM5 had been open sourced. Because of the closed nature of many Microsoft products a flawed product was released. Open sourcing much of this product would have resulted in the problems being solved or a least minimized.

       Also releasing a beta test of the product could have saved much aggravation we end users have had to endured with it so far.

       Because of tech advancing so qucikly today, and the relaese of better and more advanced products, I hold little hope that these problems will EVER be solved. Another more compelling reason for open source.

       The world is changing. Time that Microsoft recognized it, and became more open sourced. The future is there.

  39. David says:

    This is all very well but the OS doesn’t tell you as a user what it is doing. It just appears to lock up and even worse, starts queuing taps. This is going to be sheer joy for any applications vendors as they, being the front line, will have to field hundreds of questions as to why their application is "locking up". How hard would it have been to pop up a spinning "compaction" cursor? It doesn’t even pop up the hourglass.. criminal!Users are much more forgiving when they know why something is happening even if they don’t actually understand it.

    I wouldn’t say I hate WM5 but I dislike it intensly.

  40. Jeff Lewis says:

    You know – there was a perfectly simple solution to this problem: let the user choose which way the wanted to run the OS: in ROM (new style/mobile phone style) or in RAM (old way).

    Running in ROM offers just one advantage – surviving total loss of power – but it has HUGE disadvantages including speed, the speeding up of ROM degradation (most ROMs have a cell ‘lifetime’ in terms of max # writes before failure) and as we now see – issues with WM5 just *working* well.

    I’ve had Windows Mobile devices for years now without this feature and it’s never been a problem because I do this rare and apparently unheard of thing: I regularly back my PDA up to CF or SD card and can restore it in minutes if there’s a major battery failure.

    Wouldn’t it have been a hell of a lot simpler to just design in an autobackup system? Or a hibernate mode similar to WinXP, which on detecting that the power has dropped too low, writes the memory to ‘disk’ (in this case CF or SD)?

    Worse, because WM5 requires (well, essentially requires) that you have to have more ROM than RAM, and up to now everyone’s been busily adding more RAM than ROM – most WM2003 PDA cannot be upgraded to WM5.

    That’s good for the PDA makers, since it means you have to buy a new PDA to get WM5 – but it’s bad for Microsoft because it means many of us will just hold on to our WM2003s as long as we can.

    Sorry guys, given the newly universal dislike I’ve seen for WM5, I think Microsoft really blew it with this one. I’ll wait for WM6 – maybe by then Microsoft will have worked out how to make upgrades possible directly from Microsoft…

  41. Chris Oh says:

    Just to add to Jess Lewis’s comments – here, here!  

    I am also longtime Windows Mobile device user (since ver 1.0 – remember the Casio E10?), and the current state of WM5 is terrible!

    I hope Microsoft learns from this and hopefully will address some of these issues – but like Jeff, I’m not "gonna hold my breath" on it!  Maybe these issues will be addressed in WM6…

    It’s a pity really – maybe we need Palm be around a bit longer, it feels like there’s no more incentive anymore now that Microsoft to work harder!

  42. Rich says:

    Does WM5 do compaction on the Extended_ROM2?  I use mine for some extra storage.

  43. [CpD]bob says:

    Mike, could you please clarify somу issues:

    1. My Axim X51v has some 49,47 MB RAM available – even with persistent storage. Where 14,53 MB are situated?

    2. After start I have only 27-28 MB of free RAM. It meens, that 14,53 MB of RAM are NOT allocated in running programs.

    3. I have a problem with memory disappearing: after running newsland+pRSSreader+IE with PocketPlus+TCPMP for 3-12 hours I get about 3 MB of free RAM. Then system starts to close applications, but only small amount of RAM is freeing. MemMaid shows filesys.exe to become the largest memory consumer. Can this issue be related to compaction process?

    4. Data to be often accessed like PIM databases should cause lots of compaction process. Isn’t it temporarily stored in RAM and written in storage memory only from time to time (i.e every power off or once a minute)? Maybe this causes amount of RAM to be always used?

    5. Why I never met this issues on MS Smartphone 2002 and WM2003 for Smartphones? For example, Motorola MPX200 uses NOR flash.

    6. What about competing platforms? For example Palm OS Garnet uses "persistent storage" in its latest releases.

  44. MikeCal says:

    I’ll comment on a few things here.  The first is that there are a large number of WM5 devices that don’t have compaction thread issues.  I’m not trying to deflect blame, but if two WM5 devices have a problem, and ten don’t, it’s something of a stretch to say that WM5 is completely broken.

    Secondly, I’d like to remind people about the "Why Can’t I Upgrade?" blog entry (http://blogs.msdn.com/windowsmobile/archive/2005/10/06/477999.aspx).  Remember, there are a ton of reasons why OEMs don’t want to do upgrades.  These problems you’re seeing are one of those reasons.  HP and Dell are losing money on this WM5 upgrade.  They did it in an attempt to support their customers and give them an inexpensive way to get the new OS.  This does NOT make it acceptable that the upgrade devices perform badly.  And, even if they’re losing money on it, you paid money for it as well.  But you see that it’s a lose/lose situation.  Remember this if you’re dissapointed the next time an OEM doesn’t offer an upgrade.  When the hardware and software is changing as quickly and dramatically as it is in this space, upgrades are extremely difficult to pull off.  

    Regarding letting the user choose whether to use RAM storage or ROM storage on an upgrade, that would require that the OEM do two completely different images.  They’re already losing money on this.  Doing that would require that they lose twice as much.  However, we did give them the option of keeping RAM based storage on their upgrade devices.  They choose not to use it because they believed that the value of persistent storage was one of the reasons to upgrade to WM5 in the first place.  

    As for holding on to your WM2003 devices as long as you can, we actually do expect that.  As much as we’d love to have everyone buy a new device every year, we know that’s not realistic.  The devices are too expensive for typical people to purchase them that frequently.  If your WM2003 device is meeting your needs, and WM5 doesn’t offer enough of an advantage over it to justify the price of a new device, then I won’t be surprised if you wait until the next version.  I completely get it, because that’s what I do with desktop video cards.  

    As for needing Palm around to keep us working hard, let me tell you, Palm still keeps us working hard.  I’ve never met a group of people more dedicated to shipping good products.  It’s just that, where before they pushed us indirectly, now they push us directly.  

    But, that aside, since 2000 we’ve shipped six major releases, one semi-major release, and about twenty smaller releases.  In that time, we’ve moved from devices with no phone capabilities running on MIPS and SH3 processors and unchangable mask ROMs to full fledged phones running on completely different technology.  From the outside, that may seem really easy, something we could just muddle through without any hard work.  But, I assure you, it’s not.  And we’re working harder today than we ever have.

    Mike

  45. Victor Weber says:

    If these problems are related to the physical properties of the flash memory, why is it then that my Symbian phone (Nokia 9300) has no such problems? You can install apps in normal memory, in flash memory, on the external memorycard, whereever you like and and they run perfectly smooth. I think Microsoft made a big design error in their OS and is now blaming the hardware.

  46. MikeCal says:

    Victor, the vast majority of WM5 devices also have no such problems.  I’m not sure what we can conclude from the existence of a phone without compaction problems.

    Mike

  47. jugagagah says:

    Thanks Mike. I got your point. Thanks. Unless the user really stuck the SD card forever on the slot.

  48. Chillywinter says:

    =====

    As if the long erase time weren’t motivation enough to not do it very often, there’s an added problem that a flash block can only be erased so many times before it wears out and stops working…

    …Rather than erasing one block ten times, you erase ten blocks one time each.  (This is called "wear leveling.")  

    =====

    So given nominal usage, what’s the expected life of a PDA before a flash block wears out and stops working?

  49. MikeCal says:

    Current flash can typically be erased about a million times.  A typical block is 256K, so there are 256 blocks in a 64M device.  So, assuming good wear levelling, you’ve got to do 256 million block erases before the device will wear out.  How quickly you do that will depend on how often you change the data on your device.

    With nominal usage, I’d expect the blocks to never wear out.  With heavy usage, I expect it to last some number of years.  Of course, I may have a different definition of "heavy" than someone else would.

    Mike

  50. badbob001 says:

    So which ones of all these perfectly running WM5 devices using NOR flash? Can the problem be narrowed down to the type of flash used or is it simply OEM-error?

  51. Tremblay Claude says:

    The Dell X51V is a clone of Toshiba E-740.

    A beautiful machine with video card, sd and

    cf extension, Wi-fi and more but, the OS is

    not compatible with the hardware. There is no

    solution for this. An upgrade to a best OS

    is impossible. Toshiba makes everybody unhappy

    in 2002 with it an never gives an upgrade

    for a 850.00 Can$ machines. HP, Dell an the others ones use this new memory for Wm5 and

    this is a crash. The next OS and new machines

    with fast memory is the way.

  52. MikeCal says:

    After a pretty eye-opening misquote in eweek (is my writing really that unclear?) I added a sentence to the top of this.  

    Mike

  53. zenchih says:

    How about stealing time from reading/writing on NOR flash? If writing some bytes to NOR takes 100ms, you can use 10~20 ms to copy some valid sectors to another block. Users should not feel the difference between 100ms and 120ms. Thus compaction thread will only need to erase blocks in an emergency.

  54. yukon_28 says:

    Hi Mike!So,can you clearly confirm that using as litle flash ROM as possible will help to avoid a compaction problem?I mean,if i’ll store all my programs,data and same on the fast SD card,also redirect sys cash,temp files,browser cash,mail data and saving path for all programs to the same place i’ll notice a smaller amount of bugs that someone uses his device ROM at the full strength?Also i wanted to ask why battery life of upgraded hx4700 reduced (about 15-30%),what can be the reason?Generaly i like new OS,i love the fact that i can keep a twice running programs with the same RAM without any speed problem (realy,how its possible?With the same free RAM,30-35mb on 2003 i can’t run half of the programs that i can on WM5,i mean runing programs at the same time),i like new PIM and Office,i like better working wireless modules,and many others…

  55. Lat says:

    MikeCal: "Why are Axim users seeing this so much more than everyone else?  That I don’t know.  The compaction thread is the same on the Axim as on every other WM5 device.  It would seem that the Axim is doing something different than other devices.  I don’t have visibility into their design and can only speculate."

    Dell: "03/27/2006 12:34:10PM Agent (EA Chris S.): ‘For further information, the upgrade is actually Microsoft’s. You could call them and ask if they have a patch on the way.’"

    Users: lose-lose situation

  56. Genival Carvalho says:

    Hello Mike.

    Can you explain us "How Memory Swap Works in WM5"

    Regard’s from BR

  57. gufoe says:

    Mike, if e.g. 40% of my flash memory is used for stored programs and the other 60% is free for user/temp data, what does the wear levelling make? Is all the 100% flash memory used for writing (e.g. with swapping the stored programs around the sectors) or does the wear levelling only affect the 60% free space? In the last would be correct, it will be better to store all programs external in the sd-card and give the system 100% free flash to increase the lifetime of the flash. Thanks, gufoe!

  58. MikeCal says:

    Genival, I think you’re looking for this entry:

    http://blogs.msdn.com/windowsmobile/archive/2006/03/31/566187.aspx

    Yukon, yes, the less you write to flash, the less you’ll need to compact.  It’s really about data though.  Assuming that the programs don’t write a lot of temp files in the directory they’re installed to, it probably doesn’t matter where you install them.  

    Badbob and Lat, until I know what the actual problem is, I really don’t want to speculate on whose "fault" it is.  And, even after knowing what’s wrong, I’m much more interested in getting the problem fixed than assigning blame.  There’s Microsoft code in the update and there’s OEM code in the update.  It could be a problem with either or both of us.  So far it’s only NOR devices, but we’ve seen NOR persistent storage devices in the past that didn’t have this problem. I suspect that something about NOR triggers the problem but that it’s not NOR’s "fault."  

    Gufoe, the flash part is partitioned into multiple sections.  One part holds the system image and another holds all user data.  Because the system image is rarely changed, we only wear level the user data partition.  However, any programs you install after you get your device are installed into the user part.  We wear level the entire user data partition, not just the free space.

    Mike

  59. Alexander Dovgaluk says:

    [HKLMSystemObjectStore]:CompactionPrio256 _does_ change thread priority

  60. meneame.net says:

    Mike Calligaro dixit: &amp;quot;A typical block is 256K, so there are 256 blocks in a 64M device. So, assuming good wear levelling, you’ve got to do 256 million block erases before the device will wear out. […] With nominal usage, I’d expect the blocks

  61. ron3kl says:

    "HP and Dell are losing money on this WM5 upgrade.  They did it in an attempt to support their customers and give them an inexpensive way to get the new OS."

    Hmmm. I’m not sure about HP, but Dell did it to make sales.  Shortly after the X50 was released there was a Dell press release about the availability of an upgrade path to WM5 being a reason to buy an X50.

    So rather than wait until Dell released a true WM5 PDA, customers like myself bought the X50 with the advance expectation of being able to upgrade to WM5. I can’t even get Dell Asia Pacific to talk about or respond to emails about purchasing an upgrade to WM5, let alone providing technical support.

  62. Roy says:

    Still in the HP FAQ!

    Therefore, HP has decided to provide the Microsoft Windows 5.0 upgrade on the following product series: hx2000 and hx4700 series. These models will best be able to take advantage of the new features in Microsoft Windows Mobile 5.0, while not compromising product performance and functionality.

    not?

  63. Karl Brennan says:

    Mike, First and formost thank you and Microsoft for your attention to these issues for consumers. Dell has totaly washed there hands of the Wm5.0 issue with the Axim x50v. I like so many others in the Axim community are greatfull for you stepping up to the plate. I am on my 3rd Axim x50v 2 for various hardware issues and 1 for the Upgrade "Brick" issue. I like WM 5 and I am sticking with it although it has some slowness I have full faith in MS that eventualy we will have an answer to our issues. Thanks Again to you and Mircrosoft for your dedictaion and hard work.

  64. .JLBader says:

    I have no idea of what it os, bot the fx47000 with WM5 is clearly not ready for prime time. I am astonished that compaq would release it for upgrade. It’s jost not ready.

    lb

  65. Simon says:

    Finally my Dell Axim X50V with WM5 upgrade died.

    The ROM became corrupted. The replacement came with 2003SE and I won’t be considering upgrading. Total cost of my WM5 experiment.

    AUD 79 for upgrade + 309 for replacement unit + a lot of wasted time.

  66. Aldo says:

    I have recently purchased a HP hx2750 which comes from factory with WM2003, but HP offers the possibility to upgrade the device to WM2005 at 39.95. After reading all of these issues, I’m of course reluctant to upgrade my PDA, which is working fine. However, I’ve seen some comments around saying that there are devices which don’t seem to be broken by the WM2005 issues. I would like to ask a couple of questions: Does anybody have a word about his/her experience in upgrading a hx2750 to WM2005? And second, does that person find features offered by WM2005 worth the inconveniences (slowness, cost, etc.)? Thanks

  67. black says:

    if you make some flash yourself, you must need some flash button, and you’d like to have the beautiful ones, yes, click here, to carry out your dream!

    http://www.yaodownload.com/video-design/animationdesigntools/lightfuzz-buttons_animationdesigntools.htm

  68. I wrote a pocket PC application using .net that uses merge replication and SQL Server Mobile 3.0.

    My customer has issues with the softwares performance WM 5.0 systems.

    As a .net programmer, is there a way to improve performance as it is related to WM5.0.  The performance problem does not exist in previous versions of PPC OSs.

    Thanks

    Pierre

  69. Ris Misner says:

    This is very interesting, and I want to thank you for providing this blog.  This is my first time discovering the WM blog site here, so I look forward to browsing it for more info on a variety of subjects.

    I wonder if this ROM Compaction issue could be related in any way to the technology of "Memory Mapped Files"? (CreateFileForMapping, and related APIs)

    As an application programmer for the WM OS, I often struggle with limitations of using memory mapped files, because I want to load more data into RAM than there is RAM hardware available (typically for read-only purposes) and I don’t want to create my own complex memory management system.

    Mapping files stored in the file system to virtual pointers allows me to do this, but I still run into resource limits if I "load" a file that is too large, or attempt to load too many files at the same time.  Considering that my files are typically stored on NAND flash (SD/CF cards), could overusing the memory-mapped file technology, either with too-large or too-many files, be triggering the compaction thread?  Or that is, does the underlying implementation of the memory-mapped file system cause small writes (or even large writes) such that it would "fragment" the flash ROM and require more compaction, even when memory-mapping files for read-only purposes?

    Is there anything I should do as an application programmer to avoid performance issues with the compaction thread?

    I have struggled with these problems on systems dating back to WM 2002, and in fact the performance has improved with subsequent versions of the OS, so I don’t think that changes that are new to WM5 are having any negative impact on the memory-mapped file system, but I don’t want to start introducing problems as we move forward.

    If it makes a difference, I use unmanaged, non-.NET, plain old C++.

  70. techydude - anthony says:

    Hello Mike,

    Wow, you really stirred the hornets nest with this one… :)

    The issue with the Dell AXIM X50(v) (and a couple others it seems) is very curious.  Dell provided the first and thus far only ROM image for WM5 in October05 (compared to 6 for the X51(v) so far), and upgraders have been suffering ever since – that’s 6 months for Dell, working with MS where ever appropriate, to get it right.  They have not, and are remaining tight-lipped as to why/when/etc.

    They even seem to be starting to cave in to more persistent customer’s demands for a replacement to the X51 series (i believe the argument goes "Dell have declared X50 fit-for-purpose of running WM5 by selling (and continuing to sell!) the X50 WM5 upgrade, but it doesn’t work, isn’t fit-for-purpose as advertised, so replacement with X51 is justified"), having previously been adamant that X51 replacements would be out of the question.

    After 6 months, with the impressive list of hacks/work-arounds/fixes/patches/etc that Axim fan site users have come up with (for many other glaring bugs in WM5 _on the X50_, not just flash compaction), you’d think they’d fix at least the problems they could just to keep the natives happy.  They have not.

    But I have no inkling as to why, as the growing consumer dissent and – i suspect, snow-balling free X51 replacements – it only seems it’s costing them even more by taking this head-in-the-sand stance than you suggest it already has to provide the WM5 upgrade in the first instance.  Wierd…

    Secondly, a very big thankyou – for two things:

    1- for the content of this blog.  i was unaware of the MSDN blogs before and look forward to browsing others at a later date too.  just a few of your posts have given me far more insight into the workings of Windows PDAs (my areas of interest lay elsewhere til recently), and appreciation of the difficulty involved in meeting such varying needs of mobile device OEMs and users.

    2- for the clarity and eloquence of your posts.  i’ve spent the last few DAYS browsing forums/etc (which i rarely do, now i remember why) to extract and then implement/test the salient hacks/work-arounds/etc to get my Axim X50/WM5 working barely adequately (and be better informed about the issues) amidst an exhaustingly low signal-to-noise ratio (one respondent to another of your posts compared it to "root canal" ;).  your blog has been a breath of fresh air!

    I’d like to say "don’t worry" about the eweek misquote and that it was typical of sloppy jounalists half-reading then filling in the rest with their own vitriole/spin, but i guess everything you and other MSDN bloggers say here is regularly reported into the IT media – not a burden I’d like!  I tend to call a spade a fracking shovel too often… :)

    thanks & regards,

    Anthony.

  71. BTLewis says:

    OK after reading though this I am still at a loss as to whether I should or should not upgrade to Mobile5. I do not use my Ipaq for anything other than e-mail retrival. Contacts list and Advantgo. I want to be able to open office and PDF files (from e-mail) which I can not do now (even after numerous hard resets).   Or should I wait and hope a SP1 fixes all the Memory problems???

  72. MikeCal says:

    Ris, here’s a nice blog entry by the Filesystem people about how to maximize your throughput when writing to flash filesystems.  

    http://blogs.msdn.com/ce_base/archive/2006/03/15/IncreaseFSThroughput.aspx

    It doesn’t talk specifically about memory mapped files, though.  I’ll do some digging on your specific questions and get back with you.

    Mike

  73. MikeCal says:

    Anthony, I appreciate all the kind words.  Thanks!

    BTLewis, I’m not going to tell you which way to go, because doing so is just too much a lose/lose proposition for me.  

    What I will say, though, is this.  A large number of users of these two upgraded devices feel that their performance is unacceptably bad.  If what you’ve got now works, you’ll want to weigh any potential benefits against their experience.

    Mike

  74. DJ LV says:

    I’ve read through all the threads and just want to know if a "fix" is in the works, or even possible?

    Thanks,

    A devoted hx4700 user

  75. PPCusr says:

    I have an hx4700 and I have read through most of the thread here for guidance.  All of great help, as previously I had been at a loss with people’s speculations as to the cause of the problem and how to fix it.

    From just the standpoint of operating my PPC without changing registry or what have you, what can I do to minimize the slowdowns?  On WM2003SE I always was closing programs to run faster using the memory tool or 3rd party apps.  Does the same logic still apply, or does the new memory handling mean I should/can leave more programs running in the background to improve performance of the programs I use most often?

    Sometimes this works quite well and quickly if I am rapidly switching from one to another once I have done so a few times in succession, but when I let the PPC sit for a few minutes it is slow once again.

  76. Robert Stern says:

    I have a 4705 ipac. It seems WM5 problems with Dell? Bottom line, do you recommend the upgrade? Thanks.

  77. HGE says:

    Straight to the point. With all these pitfalls when will I be able to purchase bugs-free full blooded wm5 pda phone with the following specifications:-

    1) OMAP3 or equivalent multicore processor

    2) 3.7" VGA 256K colors

    3) Dopod 838 type side slide out keyboard

    4) Nand type flash memories

    5) Responsive persistence memories

    6) Able to receive, view and send fax

    7) 2 megapixels camera

    8) Wifi and bluetooth

    9) 3G capable would be fine but not must have

    10) Builtin GSP receiver would be fine but not must have

  78. Sean Howarth says:

    Hi Mike,

    Thank you for the interesting article.

    I was hoping you could shed some light on some of the internals of the "Compaction Thread", specifically in regards to the thresholds that trigger the thread to start/stop.

    1. How long must the system be idle before the thread is run?

    2. During an idle compaction, will the thread continue as long as the system stays idle? Or will it be stopped after erasing some number of blocks?

    3. What is measured to determine idle-ness? CPU activity or I/O activity?

    4. When is the flash considered "full" of invalid sectors? Is it based on a ratio of invalid to unused sectors?

    5. Can more than one block be erased at a time?

    6. How long will the thread run during critical compaction? Does it always just erase a single block?

    Thanks, Sean

  79. John Curry says:

    Does anyone know if it is possible to obtain a copy of Windows Mobile 2003 to install over 5.0? I had to replace my 2415, which I loved, and was only able to find a 2495…. with 5.0. I find it more than annoying – slow, buggy, lots of my old programs won’t run. I would be more than glad to buy 2003 from a vendor. Microsoft doesn’t sell it, nor does HP. Any help???

  80. MikeK says:

    What are FLAGS and there values in the registry in Mobile 5, i’m seem to be able to minize to effect of the Compaction by modifing the values on FLAGS…

  81. Moses3d says:

    Mike,

    On May 20 the following tweak was published [URL=http://www.pocketpcmag.com/blogs/index.php?blog=3&p=825&more=1&c=1&tb=1&pb=1#more825]here[/URL]. It is said there that the compaction process may be "slowed down" so-to-say. Do you think that using the provided patch (the link to the file is there) can really affect the intensiveness of compaction?

    Thanks!

  82. MikeCal says:

    Hi Sean, here are some answers to your questions:

    The compaction thread is just a thread running at lowest priority.  It is eligible to run whenever there are no higher priority threads running.  Conceptually it could start running a millisecond after all normal priority threads stop.  

    It will start compacting when the number of dirty sectors is greater than the number of free sectors.  Flash blocks are subdivided into sectors.  When you write into a sector it is marked as having data.  When you need to change the data in that sector, the data is copied into RAM, the change is made, and the data is then written to a new sector.  When this happens, the old sector is marked at "dirty."  When enough sectors are dirty the compaction thread goes through and starts erasing blocks to free them up.

    During an idle compaction, the thread will continue until the compaction is done, or a higher priority thread (basically any thread in the system) starts running.  However, the system can’t stop a block erase in progress.  So, if one is happening, that has to complete before the idle thread can be halted.

    Idle is determined by CPU threads running.  Theoretically this is just CPU cycles, not IO.  But, in reality, most IO requires the CPU to run eventually.  There can be short bursts of DMAed data, but usually the CPU wakes up to do something with that data.  So, typically, both CPU and IO will keep the system from being idle.  

    To my knowledge, only one block can be erased at a time.  It’s possible that someone could invent flash memory that erases multiple flash blocks at a time, but that would be handled by their flash driver.  

    The system always keeps one block free.  When we get down to having just one block free, critical compaction kicks in and erases another.  I’m pretty sure that critical only erases one block.

    Mike

  83. MikeCal says:

    Ris Misner, I found some more answers to your questions about Memory Mapped I/O.  These come from Sue Loh over on the ce_base blog.  (http://blogs.msd.com/ce_base/).  If you have more paging questions, that’s a great place to ask them.

    Mike

    From Sue:

    Sorry for the slow reply.  R/O memory-mapped files will never write back to flash, so they should never increase the flash compaction problem.  R/O mapfiles use the page pool, so if you read a lot of mapfile data you might end up paging out some code or other data you already paged in – causing extra flash reads.  R/W mapfiles don’t use the page pool, and as a result the can grow to consume a large amount of RAM.  What happens is the OS doesn’t write back or page out pages until you either call FlushViewOfFile or run low on RAM.  In general, R/W mapfiles are going to be about as good as you can get in terms of delaying write-back until the last possible moment (thus minimizing flash compaction problems).  The mapfiles don’t have any kind of smart algorithm about LRU paging or write-back.  They write back the whole file at once, and they page out the whole file at once pretty much.  I don’t have much advice except for stuff that’s relatively obvious: Touching a whole lot of data leads to thrashing, and forcing flushes all the time will cause perf hits since we’d be writing so much.

  84. Sergio says:

    Hi, I am not a PDA specialist, and did not understand fully all the technical issues described in this blog, what I can say, is that the HP IPAQ HX4700 works great with windows mobile 2003se, stabel, fast, simply great! windows mobile 5.0 issimply crap! the 64 mb ram in the former file store are not used at all and the program memory fills up incredibly fast, allowing to have very few programs open contemporary, when I listen to mp3 with windows media player I have 10% of the memory free, and you don’t do much with it. I exerience a lot of conncection and syncronization problems, that never happen when the machine is running windows mobile 2003se, and last, but not least, there is a big problem recognizing the CF card, I have to eject and reinsert it every time I want to access a file or program stored there, otherwise simply it doesn’t work. I downgraded to windows 2003se and I will stick to it unless HP and microsoft come up with some real solution. I asked for refund, let’s see what they tell me.

    Goodluck

  85. ASKu says:

    Fellow hx4700 users:

    Check out device specific advice for the WM5 upgrade here:

    http://www.pocketpcmag.com/blogs/index.php?blog=3&cat=96

    The columns and the author are quite helpful, and updated on an almost daily basis.  Use the information in conjunction with what you learn here from Mike about the cause of the problem.  The changes listed are simple to execute and VASTLY improved the performance of my hx4700 and made the WM5 upgrade finally play nice with my device.  Keep in mind these fixes are not the product of MSDN or HP development so they cannot provide you help if something goes wrong nor would they be at fault.

  86. mikekuk says:

    Compaction solved…

    try this edit and maybe someone here could explain…

    hklmdriversbuilt-instratafmd

    blocksize=262144

    the value 262144=128×2048 so try changing the value 128×512=65536

    maybe someone here with the tools can see if the filesys is still using all cpu… for me it seems much better after using PIE with large web pages and all other settings as standard WM5 build.

    I’m a HX4700 owner..

  87. Frankmts says:

    I have recently installed WM5 on my HX4700, and it runs like a dream (apart from ocassional incompatability issues with some older software).

    Why should I have no trouble, yet so mnay users are complaining of major slowdowns?

    Suits me to stay with WM5.  Really slick and smooth!

  88. gekko says:

    Just wait until you have had WM5 on your PDA for a while. It will come to a halt after some use…  :-(

  89. cristianomc says:

    Just switched back my hx4700 to WM2003… wow! …I had forgotten how fast it is compared to WM5.. No way to stay with WM5.

  90. Recently one of the filesystem guys and I spent some time investigating the Compaction Thread issue on…

  91. jevans says:

    Does compaction run on Compact Flash cards as well?  Is it Pocket PC only, CE 5, or ?

  92. Ted Metzing says:

    I’ve been resisting the urge to have "the latest-and-greatest" and leave WM2003SE for WM5…. Thank goodness for my resistance!

    I’m using HP’s rx3115, so maybe I’d come out unscathed, but I’m still reminded of the old axiom: "If it ain’t broke, don’t fix it!"

    Thanks for your heads-up and saving me from joining THE CLUB.

    Ted Metzing

  93. MikeCal says:

    jevans, every type of flash memory that is written to repeatedly needs to do wear levelling (compaction).  But most CF and SD cards do it internally to the card.  The OS doesn’t know it’s happening and doesn’t do anything to configure it.

    Mike

  94. A Houston says:

    Thank you for a every informative discussion regarding flash ROM wear & wear leveling.  I am impressed with Microsoft’s openness and clarity of the root cause.

    I completely agree with the assessment that this is a hardware OEM issue.  They had two very understandable choices,

    – Offer an upgrade on RAM or ROM.  OEM’s engineering was to choose a stable path based on their hardware.  Dell & HP failed to assess the hardware compatibility correctly or,

    – Do not offer an upgrade.  However, Toshiba decided to not offer a similar WinCE upgrade and lost nearly all market share.  This knowledge may have exerted enough pressure to force Dell & HP into offering a hardware incompatible OS upgrade.  

    If Toshiba’s PDA market loss was their rationale, they should have considered two additional influences.

    – Toshiba’s PDA market reputation was already extremely fragile due to issues with E740 series’ incorrect battery level problems.  Systems were turning themselves off, and loosing all RAM data, due to false low battery indications from the hardware.  Toshiba’s initial response was denial.  Their last response was recall and replace with a different reconditioned unit.  Unfortunately, even the reconditioned units redeveloped the problem after a few months.

    – Toshiba’s had released E755 almost immediately before MS announced the new OS release.  No wonder the users, who spent 600+ USD a month ago on the latest technology, expected a free upgrade.  Toshiba simply refused.  Probably due to cost saving considerations.

    Dell is a RELATIVELY new player in the PDA market.  Their brand recognition could be forgiven for their first engineering blunder.  HP should have been smarter.  They ended up in litigation and recall with similar engineering errors related to a 65k (16-bit) color display claim while the driver chip supported only 16 (4-bit) color encoding!  I am talking about HP Journada 545, i.e. first color WinCE PDA on the market.

    I hope HP will refund HX4705 Mobile 5.0 upgraders their money.  I am sticking to my HX4705 with WinCE 2003 SE & a Pocket PC Tech RAM upgrade to 128MB.  Its expensive, but I have not had any memory shortage problems.  It is the best performing PDA I have ever owned. I will not be buying the Mobile 5.0 upgarde.

  95. MikeCal says:

    Ahh, the Journada 545.  There’s a trip down memory lane.  My memory of that event is a bit different than yours, though.  First, the 545 was a PocketPC 2000 device.  We had color PDAs for a number of years prior to that release, and HP shipped a number of them.  

    The 545 didn’t ship with a 4 bit (16 color) screen.  It’s screen was 12 bit (4096 colors).  I worked with them on the issue, and it was a surprise to all of us.  Their engineers believed they had a 16 bit screen, and all of their code, including their frame buffer, was 16 bit.  However, without realizing it, they had bought a 12 bit screen that had a 16 bit interface.  The display hardware hooked up to a 16 bit frame buffer and threw away 4 of the bits when generating the colors.  

    You need color gradients or certain kinds of pictures (especially images of the sky) to tell the difference between 12 and 16 bits, so none of us, HP included, realized the issue was there.  In fact, they didn’t learn that they had a 12 bit screen until an end user complained and they rechecked with their hardware supplier.  

    It was definitely a black eye for them, but I can assure you that they weren’t purposely being underhanded about it.

    As for the compaction issue, see my "Some Compaction Answers" post that describes what we’ve found to be some of the issues with these devices.  

    http://blogs.msdn.com/windowsmobile/archive/2006/06/07/621132.aspx

    Mike

  96. A Houston says:

    I agree,

    – J545 was not the first color PDA.  It was sold as the first color PDA with 65k colors.

    – I incorrectly recalled 4 bit.  It was 12 bit 4096 vs. 16 bit 65k colors … it was a long time ago.  Don’t you hate excuses :-)

    New disclosure,

    – I never read a disclosure that their was a within-screen 12 bit vs. 16 bit conversion issue.  The news articles reported a display chip to screen bit compatibility issue. HP’s PDA engineering gained some points back in my book.  Display manufacturer, whoever it was, lost some.

    HX47xx is an excellent and stable device.  It had the potential for being the best PDA product platform to date.  This was the kind of stable product a corporation can specify in its computer purchase standards.  Users would be happy with features & IT would be happy with standardization.  Unfortunately, a poorly executed OS Upgrade has destroyed the potential.

    Prior to the OS upgrade, the user’s biggest Kano dissatisfier with this product appeared to be RAM capacity.  HX47xx already had a kind of "persistent storage" in the form of the iPaq file store.  The only technology update this product needed was phone capability & latest OS based application support.

    The best HP can do for market recovery now is refund the OS Upgrade and release a new HX48xx with more RAM & RAM based Mobile 5.0 storage implementation.

    Users in this forum are right.  A computing device that goes into a high-CPU-usage maintenance mode every 20 minutes is not a useful gadget to pull out of your pocket in an important business meeting.  In fact, it could be embarrassing if someone else finds the same information in a paper book faster than your PDA.  HP has a choice to make.

  97. MikeCal says:

    Regarding what HP should do: As an engineer, I focus on the technical issues.  I’ll leave concepts like refunds, etc to marketers and lawyers.  I’m not in either of those fields by choice.  (-:

    As an engineer, I think the best thing to do is to fix the issues we found with the device and see how it performs then.  Hopefully that will give people the experience they want and make the upgrade worthwhile to them.

    Mike

  98. Werner says:

    Hi Mike,

    interresting architektur. NAND as RAM and Code in RAM.

    As i understand the advantage is the boot time, which is close to zero, because the latest CPU status before switch off is stored in NVRAM (also in NAND?) and all application data are valid in NAND.

    My question is, how is the CPU status syncronised to the last modified sektor?

    Suppose there is a DiskOnSilicon NAND like m-systems which has its owen cache RAM and controller where you never know which part of cache is written back to NAND before power down.

    Werner.

    ps. here in germany we have already WM2006 😉

  99. MikeCal says:

    Werner, I’m not sure I understand your question.  I’ll answer what I think you’re asking.  If I’m misunderstanding though, please ask again.

    Every flash sector has a small amount of overhead stored in it.  Part of this overhead is a "dirty bit."  The way flash works is that you can change a 1 to a 0, but if you want to change a 0 back to a 1, you need to erase the whole block.  When you erase a block, it becomes all 1s.  

    So, when we erase a block, all the sectors get their "dirty bits" set to 1.  When the CPU needs to mark a sector as invalid, it sets the dirty bit to 0.  The code that looks for dirty sectors looks for dirty bits that are set to 0.  

    As for how it knows which one it wrote last, that is just stored in the filesystem somewhere.

    Mike

  100. MikeCal says:

    Regarding WM2006 in Germany, can you elaborate?  We haven’t released a product called "WM2006" anywhere in the world.  While we’re obviously working on new products, we haven’t even decided on the name of the next one yet, much less released it.

    Mike

  101. Werner says:

    Hi Mike,

    WM2006 is the Soccer Worldcup 2006 😉

    Back to engineering. Maybe i got it completly wrong. As i understand, not only the file system is stored in NAND, also all data like variable or stacks and heap is stored from time to time in NAND to guarantee a fast warm boot time.

    Werner.

  102. MikeCal says:

    I’m from a country that doesn’t even call the game by the right NAME (it’s "soccer" here even though the entire world calls it an equivalent of "football").  So I’ll claim cultural ignorance and hope you don’t think too little of me, but what do the letters "WM" stand for in the world cup?  I’m assuming it’s a non-english language acronym, but I’m at a loss.  I know there are a couple of languages where "World" starts with an "M" but that’s as much as I can figure out.  Please, be gentle on the american.  He wants to learn.  (-:

    Back to a world I understand, in Windows Mobile, stacks and heaps are not stored in NAND.  They’re kept in RAM.  Only code is kept in NAND.  An actual boot (as in, pull out the batteries and stick them back in) doesn’t do any real tricks to speed it up.  However, on a PocketPC when you press the power button, the system doesn’t reboot.  It just goes to sleep.  When asleep, the RAM is still powered, so none of the stacks, etc, are lost.  

    Mike

  103. Kevin says:

    Mike,

    Whilst it is somewhat entertaining to understand the ins and outs of Compaction Threadding, is there a simple answer?  

    Should I upgrade my HX4700 or leave as is with 2003.  It’s the old story ‘if I wanted to be a mechanic, I’d buy a landrover….’.  I don’t want top be a PC/PPC technician, I just want it to work… I’m happy to treat all of this techo stuff as a black box.

    Thanks,

    Kevin

  104. MikeCal says:

    Kevin, there’s no simple answer.  Some people like the upgrade, many despise it.  But, if I were making the choice myself, and I saw this many people having this much trouble, I’d wait to upgrade.  At the very least, I’d wait until we see if there are any results from the investigation Andrew and I did.

    http://blogs.msdn.com/windowsmobile/archive/2006/06/07/621132.aspx

    Mike  

  105. David Jenkins says:

    I was keen to gain some of the advantages for my 4700 when WM5 first appeared and have just recently given up on it. It seems to me that even what I first thought were improvements over WM2003 didn’t work out that way in practice:

    Clearly the compaction problem is a killer; when I turn on my 4700 to look something up I don’t want to have to wait longer than it takes my laptop to boot for it to appear.

    Persistent storage at first glance seems like such a good idea. It should result in no data loss when the battery runs out (it does), better battery life (not true for the 4700) and better use of the iPaq file store – also not true since if you actually write to storage compaction runs longer and makes the device virtually useless.

    Going back to WM2003, I’ve discovered that it is faster than WM5 even when compaction is not running….. perhaps because writes are to RAM, not flash?

    I actually have more free RAM on WM2003 than WM5, negating what I thought would be more efficient use of RAM and flash.

    Battery life is significantly better on WM2003 than WM5; I presume this is because the device is writing to RAM rather than flash. Yes, I did turn down the standby setting to its minimum value on WM5.

    As for persistent storage, I have Sprite backup (iPaq backup) scheduled for a system backup at 3:00 am every morning; not perfect since I could lose some data entered during the day, but a far saner approach than WM5’s version of persistent storage.

    So for me, going back to WM2003 was like returning to an old reliable friend. I really hope that WM6 or whatever it’s called is less ‘persistent’ in its use of storage.

    Oh, and I forgot to mention how relieved I was to get rid of that wretched menu bar at the bottom of the screen.

  106. studiodaz says:

    I have been reading this web tutorial (The concise tutorial to making your WM5 HP iPAQ hx4700 working great)and was wondering if it will fix anything and is it safe to use?

    http://www.pocketpcmag.com/blogs/index.php?blog=3&title=the_concise_tutorial_to_making_your_wm5&more=1&c=1&tb=1&pb=1

  107. cristianomc says:

    Anybody knows if using a 4GB Hitachi CF Microdrive will avoid the slow performance of WM5? Is WM5 (on iPAQ hx4700) compatible with this Microdrive? Using it I won’t have any memory limitations on WM2003 and WM5, right?

    Thank you!

  108. Tony Smith says:

    I have an hx4700 and have upgraded to WM5. The performance of this OS is really slow. Please tell me this issue is being worked on and that a solution will be coming out soon. Otherwise, I will be going back to WM2003, today! Thank you…

  109. Michael says:

    Any reports on how the WM 5 does on the HP 2215? I was going to upgrade but now, I’m really hesitant.

  110. Tony Smith says:

    I reverted back to WM2003. What a difference. If you’re thinking about upgrading to WM5, think again. There are definately issues with this OS with regard to performance that need to be resolved. WM2003 is by far the best OS for the hx4700.

  111. Sam says:

    I have just wasted $39……

  112. Ying Li says:

    is it good for hp 1900

  113. Solnyshok says:

    Mike, do you know if HP is working on fixing filesys algorithm and msflash sector size or is it a lost case? Any hope?

  114. MikeCal says:

    HP was definitely interested in the information we sent them, so there’s reason to be hopeful.  But they haven’t reported their plans to me, so I can’t tell you more than that.  (Note that they really don’t have any reason to report their plans to me, so don’t take this as a negative.)

    Mike

  115. Gekko says:

    Can you please forward the information to DELL as well? Their implementation of WM5 on the Axim X50v is pathetic…

  116. Stratocaster says:

    Can I upgrade my iPAQ 6365 to WM5?

    How come?

  117. MikeCal says:

    Stratocaster, please see the following blog entries.

    "Why Can’t I Upgrade?"

    http://blogs.msdn.com/windowsmobile/archive/2005/10/06/477999.aspx

    "Then Why Do You Get To Upgrade?"

    http://blogs.msdn.com/windowsmobile/archive/2005/10/19/482683.aspx

    Mike

  118. em says:

    how to upgrade a hp ipag hx2200?

  119. zappers says:

    I upgraded my hx4700 to WM5 about 2 months ago.  I have not realy had any problems with the upgrade.  Have never seen this notorius compaction thread run and think that system performance is quite good except for the occational reset if I left my ppc on and it went into standby mode by itself.  I run all my cash files on sd/cf card.  Have GPS device via CF slot that works great.  Had problems before upgrade that my ppc would randomly lose all info as if battery ran out with battery power on 100%.  Upgrade has solved this prob as now my programs gets stored on rom and not ram.  Happy customer here and cant believe I only had to pay 39 USD…

  120. hx4700user says:

    there seems to be a new release for the hx4700(english):

    http://www.ipaqchoice.com/generic3.aspx?page=WM2005

    Anyone tested it yet?

  121. ks_chai says:

    My company uses a Cingular 2125 phone. I connected the Cingular 2125 phone to the USB Host, and the program in the Cingular phone calls the following code. However, the Cingular phone does not seem to send out any USB data. I check that in the registry HKLMDriversUSBFunctionsDriversDefaultClientDriver = "RNDIS".

    Can I use RNDIS as a COM to send data out from the USB port?

    #define FILENAME _T("COM5:")

    hNewDevice = CreateFile(FILENAME, // pointer to name of the file

    GENERIC_READ | GENERIC_WRITE, // access (read-write) mode

    0, // share mode

    NULL, // pointer to security attributes

    OPEN_EXISTING, // how to create

    0, // file attributes

    NULL); // handle to file with attributes to copy

    if ((hNewDevice == NULL) || (hNewDevice == INVALID_HANDLE_VALUE)) {

    hNewDevice = NULL;

    }

    else

    {

    success = WriteFile(hNewDevice, // handle of file to read

    data, // pointer to buffer that receives data

    size, // number of bytes to read

    &num_bytes, // pointer to number of bytes read

    NULL); // pointer to structure for data

    }

    if (!success)

    {

    CloseHandle(hNewDevice);

    return FALSE;

    }

    CloseHandle(hNewDevice);

    Hope you may help me send my questions to a relevant person, or post this question on a web site.

    Thanks,

    KS

  122. werner says:

    Hi Mike,

    but amerikan football is definitely not soccer or Fussball 😉 And WM2006 is an acronym for WeltMeiterschaft 2006. Again, thanx for clarification,

    Werner.

  123. MikeCal says:

    Thanks Werner.

    Mike

  124. Wyatt says:

    I spoke with HP extensively today, and even though they are aware of the problem with the HX4700/4705, they cannot confirm when a fix would be available.  This does not make a ton of sense, but they are saying that a BIOS update would be available in the future for that model, though they could not tell me approximately when that day may come.  I’m apprehensive about thinking that HP will come through with an update because of how they dumped support for PPC2003SE on the 3955 inside of a year after releasing PPC2003 for the 3955.  I am considering the 4705 at this time, but I am reluctant to based on HP’s shady support history.

    I guess the number one question here is:

    Will a BIOS update fix this problem or is this just lip service by HP to get us off of their backs?

  125. MikeCal says:

    Wyatt, check out "Some Compaction Answers."  http://blogs.msdn.com/windowsmobile/archive/2006/06/07/621132.aspx

    Two of the three things Andrew and I found can be fixed by an HP update.  The third can not.  We won’t know what effect those changes have until (if) HP gives us a build with them made.

    Mike

  126. Ankur says:

    Hi guys,

    HP has released a BIOS update on their website:  HP iPAQ hx4700 Pocket PC Series ROM Update

    version 2.01   (27 Jul 06)

    http://h18007.www1.hp.com/support/files/HandheldiPAQ/us/download/23729.html

  127. Cristiano says:

    Lets try it! Does anyone else did it?

  128. Cristiano says:

    Well, it seems that this new patch has partly solved the performance issue.

    Now when you start using the device the compaction seems to stop. The device is much more prompt. But, of course, not as fast as the WM2003.

    I’m having some sync problems (can be caused by a third party extension USB hub because when connceted directly to the USB port of the PC the problem doesn’t occur) and Opera had hanging the device twice, forcing a reset (Opera is much more processor dependant than IE, but I don’t know the cause of hanging).

    I thing HP and Microsoft did the possible to make WM5 more reliable. It seems that it was the final "sprint" to make WM5 running on "adapted" devices a good option.

    By the way, at this time I agree that it’s a good option to hx4700! Thanks to Mike, all the people that has worked hard on trying to solve the problems on this and other blogs and forums, and to all the Microsoft and HP staff!

    😉

  129. MikeCal says:

    Thanks for the update Cristiano.  I’m glad to hear it’s working better now.  How is it going for everyone else?  Did the updated rom make things better for you too?

    Mike

  130. Atenimo says:

    Hi Mike,

    I was looking to see if the WM 5.0 upgrade was a possibility, and I just read through the whole text of this Blog.  

    I am impressed.  I am surprised.  I had a ton of questions, but I can actually say that most of them are answered for now.  

    I will be returning to see how the ROM updates turn out.  I already ran one for HP, but not the most current release.  I am pretty happy with the information in this Blog, and I appreciate the information you and the rest of the Blog community have presented.

    sign me,

    Waiting on Obviously New Data Enlightening Readers, Instead of Negative Guesses….

  131. Ashley says:

    Try to run Slingmobile player in WM5.0, but after running it for awhile, it hit wince501bexception in wmvdecoder.dll and halt.   Anyone has any clue to this and whether there is a fix for it?  

    By the way, I run this on my HP iPAQ hx4700 Pocket PC Series with ROM Update version 2.01 (27 Jul 06)

    http://h18007.www1.hp.com/support/files/HandheldiPAQ/us/download/23729.html

  132. Ashley says:

    Try to run Slingmobile player in WM5.0, but after running it for awhile, it hit wince501bexception in wmvdecoder.dll and halt.   Anyone has any clue to this and whether there is a fix for it?  

    By the way, I run this on my HP iPAQ hx4700 Pocket PC Series with ROM Update version 2.01 (27 Jul 06)

    http://h18007.www1.hp.com/support/files/HandheldiPAQ/us/download/23729.html

  133. Paul says:

    3-rd day after update and works great

  134. Aaron Marshall says:

    Well i upgraded my hx 4700 and well i see the problem mentioned as demonstrated by the device, so i got my team together we wrote a few programs together and pretty much had to write a boot loading app which basically tells WM5.0 and installed apps to write any logs temp and other information to a ram temp file it took us a long time to develop but we cracked it finlally and got a fix for the problem in the mean time we ran a few bench marks and tests we found out that compaction was critcally reduded by upto 94% which was outstading overall perfomance betterage. and only time it occured was when we installed the apps which we had  written to the device and setup once all squared off and apps finished and tweaked to a full operational standard the device worked perfectly. the only minor draw back was that are program consumed 12% of CPU power but in conparison to previoulsy metioned problems was outstadningly better. The Program we released is like a un-official service pack but its for upgraded devices only. will keep posted in the mean time of updates we soon hope to go public and get microsoft approval.

  135. Cristiano says:

    Did you tested with the new HP patch?

    How does it perform comparing a 4700 with the new HP patch and one without the HP patch but using your app?

  136. Bill S. says:

    To Mike and all the other posters:

    WOW!  I am impressed!  First, with Microsoft for hosting this blog, and second for the knowledge shown by all involved.

    I just bought a HP HX2795 – 624mhz cpu, 64mb ram, 192mb flash rom (should arrive tomorrow) and discovered this blog more or less by accident.  The hx2795 ships with WM5.  Can anyone confirm that it doesn’t have the same performance problems that the upgraded 4700 people are seeing?

    Thanks

    Bill

  137. Bert says:

    The HP update for the hx 4700 does not solve the problem. The device responds still a lot slower than it did with WM2003. So I strongly advise against the upgrade. I did not come across one feature that is worth the upgrade. The degraded performance on the other hand …

  138. lockupchick says:

    I’m experiencing the same problem with my 4705. I’m very disappointed that this was released with the problems it has. I enjoy some of the new benefits, but certainly not worth the loss of performance and frequent crashes.  I’m afraid to install the new bios update after this fiasco. Is MS & HP planning to put out a patch or, better yet, a complete new download for those of us who already spent the money for WM5?

  139. hx4700 says:

    "un-official service pack but its for upgraded devices"

    Can we have a link for this please, would like to give it a go. I’ve  been trying to have all temp files created on ramdisk but failed.

  140. Solnyshok says:

    Aaron Marshall, where we can find this service pack?

  141. visitor says:

    filesys is only for upgraded devices….so why was it not designed for nor rom’s…ms not testing the code…just design on paper and hope for the best.

  142. Jon McNamara says:

    I have the hx4700 and I’m quite impressed with the knowledge of some of the people here. Keep it up guys and we may get a final resolution.

    Could someone please confirm if PocketInformant is included in the WM5 upgrade? If so, what version is it? Is the Sprite backup utility also included?

  143. visitor says:

    PI is installed but not the latest version (PI Rev 4 Build 491)

  144. visitor says:

    Yes why is compaction such an issue when it’s only for upgraded devices.  MS did not do any real testing of the upgraded devices or never had a device with half full internal ROM as it becomes unusable. What a waste of time and money. I hope MS come up with a new Compaction algorythem to fix our devices.

  145. MikeCal says:

    Compaction is used on every device, not just upgrade devices.  The algorithm was heavilly tested on a large number of devices, NAND and NOR both. It’s been in use for a number of releases now and has worked correctly for literally millions of units.

    The main reason for the trouble on the two upgraded devices is that they were misconfigured by their OEMs.  They started with XIP devices and a configuration that was appropriate for that purpose, and moved to non-XIP without updating the configuration appropriately.  In general, we don’t test OEM devices for them.  We have 40 OEMs, some of whom release ten devices a year.  

    But if you’d like to blame us for this, you’re welcome to.  It DOES say "Windows Mobile" on the box, so ultimately people are going to point at us if something doesn’t work.  We probably could have provided our OEMs with a tool that made sure they configured this part of the world correctly.  Or maybe we could have made the needed change more explicit in the documentation.  Etc.  I’m not trying to pass off blame on the OEMs here.  

    But it’s just not right to claim that we just code in a vaccum and release stuff without testing it.  We have almost as many testers as developers, and we are consistently test constrained.  When we want to do a feature and can’t align enough testing for it, we cut the feature.  

    Mike

  146. hx4700 says:

    "The algorithm was heavilly tested on a large number of devices, NAND and NOR both. It’s been in use for a number of releases now and has worked correctly for literally millions of units."

    So why does the HX4700 still have a filesys issue? any thread that consumes 80/100% cpu for more than a few seconds is very bad and

    even more so on a PDA. This is a premium device that has been around since WM2003SE.  Are you saying NOR devices can use your algorithm and not have any of the issues we see!! Are there WM2000 or WM2002 NOR devices that can run better than a WM5 HX4700??!?!  So what else is going on with the HX4700.. There are not many NOR memory manufactures out there and as the HX4700 is a premium device and not been out that long i.e. it doesn’t run any OS before WM2003SE…I’m confused why we still can’t rely on our devices.

    Were MS commissioned to make a WM5 specific version for HP and Dell or not, if yes you failed.  I’m sure most of us are not interested in Blame only that we have a real solution.

    Thanks for your comments.

  147. mrpda says:

    I don’t have any compaction registry key’s in my native WM5 device but my friends upgraded device does have them.  If they both do compaction why don’t they both have the key’s in the registry? what happens if I change the Sector size in the compaction registry, it seems like it takes longer before compaction kicks in with a sector size of 256? and even longer when set to 128, can this damage the device?

  148. MikeCal says:

    First, it would be great if you would read the "Some Compaction Answers" entry I did in this blog.  http://blogs.msdn.com/windowsmobile/archive/2006/06/07/621132.aspx  I answer some of these questions there.  

    Second, no we were not commissioned to do a WM5 specific version for HP or Dell.  Most OEMs take our OS software, write their own drivers, add their own applications, do their own testing, and release their own product.  In many devices, over half of the software in the ROM comes from the OEM, not Microsoft.  

    mrpda, the sector size is indeed the biggest problem here.  People have reported much better experiences with the newly released 4700 ROM that has the sector size set correctly.  Unfortunately, you can’t change the sector size by changing the registry keys.  The sector size needs to be known before the registry can be loaded, so that information is stored in a deeper configuration file that can only be changed by the OEM at image creation time.

    Mike

  149. Mike,

    First, I want to thank you for all of the excellent information that you’ve provided on the compaction issue. I’ve just finished several hours of reading this and related threads, and I think I’ve finally gotten a good handle on what’s going on.

    I’m an iPAQ 4700 owner. I’ve been holding off on the WM5 upgrade, hoping to see if the problems got resolved. It sounds as if the latest update from HP improves the situation quite a bit, but it’s still not perfect. I wish that I could stay with WM2003SE, but unfortunately I’m bumping up against the problem of having too many startup/sip/today programs that I want to run.

    Here are some questions for you: A few months ago on this thread, you talked a bit about how flash memory wears out. What is the granularity of wear-out? Is it at the 512B sector level or the 256kB block level? What does WM5/CE do when a sector or block wears out? Does it mark the worn-out area and continue using the rest of the device, or is the whole thing kaput? Can you anticipate when a block or sector is about to wear out? Is there some kind of counter being maintained, or do you just keep writing/erasing until one day it just fails? Does a block or sector gradually become unreliable, or does it behave in a more binary fashion were it works perfectly up until that final erase/write cycle?

    You may have guessed where I’m going with all this. If the granularity of wear-out is small and the device can work-around the wornout spots and continue functioning with an ever-increasing number of them, then I personally would like the reclaimation policy to be: Don’t do wear leveling and just let spots wear out. I’d augment that policy with a manual wear leveling utility that exchanged static file storage areas with those that have been written and erased frequently.

    Is such a policy feasible? If so, could you run it by HP to see if they are open to it?

    My rational is this: I bought my 4700 about 2 years ago when it first came out. It was at that time, the best portable device available, and even now, 2 years later, I think it’s still the best available. The current crop of phone devices such as the various HTC Wizards can’t match it’s capabilities, and probably won’t for the next year or two. UMPCs–which everyone expects to take over for the high-end PDAs–are still too new and way too expensive. And neither HP nor Dell nor anyone else seems interested in doing one more generation of non-phone, high-end PDAs. Given all that, I’m going to continue using my 4700 for the next 2 or 3 years. And since a 4 or 5-year-old PDA isn’t likely to have much resale value, I don’t much care if the flash ROM wears out in spots. As long as it performs well and lasts me another few years, I’ll be a happy camper!

    –scott–

  150. MikeCal says:

    Hi Scott, I’m glad you’ve found these articles useful.

    Flash wears out at the 256K block level.  In general, wear comes from erases and erases are done on blocks.

    If we try to erase a block and the erase fails, we’ll mark the block as bad and stop using it.  I believe we’ll also mark a block as bad if we try to write and that fails.  However, it’s possible for an erase and a write to seem to succeed but for the data to be corrupted.  If we try to read that data, we’ll get an ECC error and throw the data out, but we won’t mark the block bad.  Theoretically the next erase will fail and then mark the block bad, but I’m not enough of an expert on flash hardware failures to know for sure that that will happen.

    We can’t really anticipate when blocks will fail.  Numbers like, “a million erases” are statistical and not fixed. So even if we knew that a block had been erased a million times, we’d only know that it has a higher probablity of failing soon, not when it would go.

    There are two main reasons for wear levelling.  One, we don’t want the available amount of flash to decrease over time.  We’d rather you have a fairly consistent amount of storage and then have it all go at once.  And two, if there’s a chance of your data being corrupted as a block wears out, we want that to not happen until the end of your whole device’s life.  We don’t want it to happen throughout the life of the device.  

    As for the feasability of not wear levelling, the way that would need to work is that HP would need to ask us for a special non-wear leveling filesys.  We’d then need to make and test one and release it to them.  Doing that work would have to be at the expense of some other work we’re doing, so it would have to be a very high priority request to happen.  

    Mike

  151. Trevor says:

    Does anybody know if the HP update dramatically improves performance on the 4705? Please resond.

    Thanks.

  152. cristianomc says:

    On my hx4700 the performance was dramatically improved as the CT isn’t happening so many times as before…probably the same should happen with the 4705.

  153. Melvin says:

    Nothing has dramatically improved. I installed the HP patch on several of our hx4700s. I notice a "bit" of improvement, but overall our users are disgusted with the sluggishness of Mobile 5. The GUI just drags, it feels like a .NET app. What gives, did Microsoft write the OS using .NET VM or something?? We are ordering a Dell x51v just to try a comparison, but if we find no solution, we either have to revert to 2003, or move to Palm.

  154. visitor says:

    would an even smaller sector size help? i wish we had the tools todo this.

  155. valerie says:

    I have an HP iPAQ h6365, does anybody know if I can upgrade to Windows mobile 5.0 now or in the near future ?

  156. Bert says:

    The HP patch for HX4700 does not solve the problem at all. Fortunately, the WM5 install disk has the option to downgrade the OS back to Win2003 SE. I feel like I’ve lost 40 € and a lot of time. WM5 is probably a fine OS, but it is not suitable for the HX4700.

  157. Solnyshok says:

    new WM5 ROM 2.01 for hx4700 shows a good speed now. You need to remember to kill NavPointService and do fake server for activesync and install prety much all programms to CF card. But then it ROCKS.

  158. zebra says:

    after I upograde 2.01 for my hx4700, it seem WM5 is "acceptable" for speed. its better then old WM5 ROM.

  159. Big H says:

    Please excuse me if I have limited knowledge of how PDA memory works. I have an X50v and have had all the slowdown problems resulting in me going back to 2003SE. I understand the need for compaction due to the type of memory usedin the Axim, but I just want to focus on an earlier post regarding the use of the SD card to store temp memory to avoid the need for compaction to take place. The answer was that users would eventually need to remove the card and thus creating a problem. My question is simply – the Axim has both Compact Flash and SD capabilities – Would it be possible to have a Compact Flash card perminantly inserted to use to write all systems files to – and SD for in and out day-to-day use. Could this work?  

  160. Neriglissar says:

    Since WM5 runs so fine on so many PDA’s and only craps out just on the Axim50 and the HP hx4700, there’s gotta be a hardware issue. Can someone describe this specific hardware issue/incompatibility on the hx4700? Why is this only happening to this devices and not to others??

    I think thats the key question and the one which needs to be worked on the most.

  161. MikeCal says:

    Neriglissar, please see "Some Compaction Answers" http://blogs.msdn.com/windowsmobile/archive/2006/06/07/621132.aspx

    Mike

  162. hx4700user says:

    but why is the cpu at full when compaction is going on? only the upgraded devices have this issue. if a special bit of software was developed to allow the upgrade it’s not correct for the type of rom used. I bought a hx4700 because it was supposed to be designed for wm5 in mind. i just hope ms can develop a more efficient compaction thread. Are you working on something? no dates just a hint.

  163. best says:

    I have an HP iPAQ hw6515 mobile messenger with WM2003  S.E. , does anybody know if I can upgrade to Windows mobile 5.0 now or in the near future ? and how cant I do it?

  164. Nils Bersweden says:

    How does one revert to the previous WM2003 after having installed WM5?

    Thanks for this discussion. I have just updated my HP iPaq hx2400 with WM5 and have a had a frustrating 24hrs! Interestingly I decided to update because i wanted to update another piece of useful software which required WM5. But the problems – slowness, need to update other software which aren’t supported by WM5 and – I can’t believe it – no Access Database synchronisation! – plus other sync problems – have finally defeated me and I want to restore my old iPaq backup! Any advice please? Many thanks.

  165. I am just writing an applicatin on WM 5.0.

    My application is crashing by giving the error

    wince501bException while using new operator.

    Can u pls let me know, why was this happening?

  166. Martin Hofmann says:

    We have developed a WinCE 5.0 Device. Free flash size is about 10MB. 1 Block contains 252 sectors each containing 512Bytes.

    After writing 252 files (each 100Bytes) the compaction thread starts! It ends after 4 minutes!

    When i write 252 files more the compaction thread starts again, still taking 4 minutes until it ends!

    Q1: Why does the compaction thread start when just one block is full?

    Q2: Why is my application slowing down like hell? Isn’t this the idle compaction thread? Even setting the application to very high priority does not help!

    Q3: Why does it take 4 minutes after just writing 252 files? What exactly is he doing?

    Q4: Is there a possiblity to recognize the compaction thread?

    Q5: Is there yet a solution for this problem?

    Thank you very much for any hints and pointers!!

  167. MikeCal says:

    Martin, do I understand correctly that you’re an OEM making a general embedded device (not Windows Mobile)?  

    Mike

  168. Martin Hofmann says:

    Mike you are right! We use WinCE 5.0 (not Windows Mobile) and had built our own OS using Platform Builder!

    But I think the compaction issue is very similar (This was the best info about the compaction thread I found!)

    Do you have any pointers for my problem?

  169. ce_base says:

    Martin:

    Let me see if I can answer some of your compaction questions. First of all, the assumption that writing 252 100 byte files will equate to 252 * 100 bytes of total data being written to flash. There are a couple of things that make this untrue:

    1) Flash, as with most storage devices, is sector addressable. This means that I can only read and write data in terms of sectors. If I change 1 byte in a file, I have to re-write the entire sector. This limitation can be mitigated if you use some sort of lazy-write or write-back scheme, but there will still be times when more data is written than is optimal.

    2) Every file on the file system will occupy physical space equivalent to a multiple of the file system’s cluster size. Effectively, the cluster size is the file system’s storage resolution. At a minimum, the cluster size is equal to the sector size of the media (could be a multiple of the sector size for larger disks). Given a typical 512 byte sector size, every file will take up at least 512 bytes. So, when you write a 100 byte file, it actually occupies 512 bytes. That’s the way the FAT file system (and most others) work.

    2) When you write to a file, there is meta-data being updated in addition to file data. The directory entire for the file needs to be updated with the new file size, file name, etc. Similarly, the File Allocation Table (FAT) needs to be updated to describe the allocation for the new file. All of these changes need to be written to disk, and recall from #1 that the minimum write size is 1 sector.

    So, while you appear to be writing only 252 * 100 total bytes to the file system, there’s really a lot more than that going on. How do you minimize the impact? There are two things I would recommend to you:

    1) Disable FAT’s write-through mode. This setting is on by default for robustness, but it will negatively impact your performance. From common.reg:

    ; Un-set FATFS_FORCE_WRITETHROUGH (0x00000020)

    [HKEY_LOCAL_MACHINESystemStorageManagerFATFS]

       "Flags"=dword:00000044

    2) Enable write-back mode. This setting will force FAT to spin-up an idle thread to flush the cache only when necessary or when the system goes idle. This can help coalesce multiple changes and reduce the total number of writes if the same sector is modified multiple times simultaneously.

    [HKEY_LOCAL_MACHINESystemStorageManagerFATFS]

       "EnableWriteBack"=dword:1

    You might also want to play with the cache size, though I expect the defaults will be ok for your relatively small storage space.

    Also, I’d like to clarify that flash sectors only become "dirty" when they are re-written. If you’re always writing new data and not changing data (remember there’s meta-data, too), you won’t be generating dirty sectors and won’t trigger compaction.

    (Andrew Rogers, Windows Devices Core OS)

  170. Takashi Murakami says:

    I’m an unhappy Dell Axim x51v due to its incridible poor performance. All of the PIM functions I performe much faster in my old PalmOS 3OMhz than my new WM5 624Mhz!!!! It’s true, trust me.

    Anyone if it’s possible to downgrade WM5 to WM2003  in x51v?

    Thanks in advance.

  171. Martin Hofmann says:

    Andrew:

    Thank you for your answer! You where absolutely right that there was not just one block full! I built the debug version of the OS and turned on all the debug messages in the fmd.dll.

    Creating a 88Byte file from our program wrote to 57 different Sectors, some of them twice! Writing 252 files therefore needs 252*57*512Bytes = 7.3MB! Because there was only 10MB free flash, the compaction thread started to defrag.

    So far so good!

    After this I tried to track why there are so many sectors writen!

    We have installed TFAT (Transaction safe FAT) installed for flash so it is normal to have some "overhead". But why that much? I found one/the reason: We make a file stream, writing the file in 2 Bytes portions. Every time we do that, it is directly written to the file system, every write access using one new sector (I think because of the TFAT, is it possible to cache TFAT?).

    Now I changed the code and write to the file with only one write! Result: only 10 Sectors where written instead of 57 per file! Means the compaction thread starts 5.7 times later! :-)

    Maybe it can be more optimized by switching from TFAT to FAT, but I’m not sure if this is a good option for our project..

    One question resists: Why is the performance of our application that bad when the compaction thread is active? Is that not idle compacting?

    Martin

  172. palm says:

    Buy a Palm (not the one with WM!)…

  173. Riaan says:

    Having read this blog for the last 6 months,

    I delayed upgrading my hx4700 untill the beginning of November.

    The results : Cost 39Euro+10Euro shipping: Total 49 euro.

    Results :

    I have installed all my programs to CF memory, but compaction still kicks in every two days or say, but only for about 3 to 5 seconds.

    Having moved all my data also to CF memory, means that Activesync can’t find any documents, so I have to move them manuallly.

    The biggest disappointment however, were Excel 5. What a load of rubbish!. Whoever wrote this code needs his head examined. Not everyone lives in America, where the Date format is arse about face. The rest of world like to work with DD/MM/YY  or even YY/MM/DD. But that is the one format that is missing in Excel 5.

    (It will display it correctly, but you cannot enter it correctly in the formula bar).

    Then, reading the persistant store blog, it is suppose to make the battery life longer, but it dosn’t. Just the opposite.

    And the memory hungry WMP10. I still don’t know how to create my own playlist in WMP10. I could do it in WMP9, but not in WMP10.

    End result: Is WM5 worth the 49 euro upgrade ?

    Answer : NO, an emphatic NO

    Riaan

    (From South Africa, if you wanted to know)

  174. phendric says:

    Having read through this article and all of these comments, I ask the following:

    If the system detects an idle state and kicks in the compaction thread (so it’s a non-critical compaction), what happens if the system suddenly becomes busy with something else (ie, the user starts a program)?  Does the user-initiated process have to compete with the compaction thread, or can the compaction be stopped, since it’s non-critical?

    Phillip

  175. MikeCal says:

    Phillip, the compaction will stop, but if a block is being erased, that needs to complete.  NOR ram can take up to two seconds to do a block erase.  NAND is considerably faster than this.

    Mike

  176. Eti says:

    I just upgraded an HP Hx2410 to wm5 and boy what a disappointment so far. The performance is the first noticeable problem but more worrying is the unreliability. numerous freezes, and missed alarms.

    After being powered down for prolonged periods I find the blue led blinking and can’t power on the device until I reset it. I have to reset more than 10 times a day! One whole night it persistently restarted until the battery died, and I only resolved it by removing the SD card (which was corrupted by the os- the inbox folder showed up as 4GB when the card is a 512MB)

    I didn’t have these problems before paying $46 for the upgrade.

  177. Dd says:

    I can echo and agree with the comments from Eti above. My 4700 is slower and less reliable on Wm5 than 2003. I’ve paid wuite a lot of dosh to work that one out!

    My about page shows mine as version 5.0 OS 5.1.70 (build 14406.1.1.1)

    Am I up to date? if not where can I find updates.

    Thanks

  178. cristianomc says:

    Mike, using a defrag tool can help the compaction to stop?

  179. CK says:

    Anyone who has upgraded their IPAQ HX4700 to wm5 experience unexpected power loss? Since upgrading I have noticed that my 4700 battery dies even when I make sure the power is off. Any answers here???

  180. MikeCal says:

    A hard disk defrag tool won’t work on flash, and I wouldn’t expect anyone to write a flash defrag tool.  File fragments are an issue on a hard disk because they cause the drive heads to seek more often, which slows the read down.  Flash doesn’t need to seek, and fragmented files really don’t take any more time to read than contiguous ones.  This is a good thing, because the lower level wear levelling would refragment the files anyway.  

    So, no, I wouldn’t expect any sort of defrag tool to help improve compaction.

    Mike

  181. Ben says:

    CK,

    I don’t have a solution for you but I can confirm your power issue.

    My wife has an Ipaq hx2000 series and has experienced both slowed performance and power loss. The device appears to turn off but will still drain the battery over time. Being the lazy type … I’m just going to go back to PPC2003.

  182. checko says:

    Does the "wear leveling" feature exist in every filesystem  ?

    So I don’t have to worry about my nand flash failed soon when I keep modifying the file ?

    Thanks

  183. Stephen says:

    I guess we should treat the OS UG version as sort of favor by the manufacturer since they design the hardware with reference to the OS they using at that point of time. Especially the one with WM2003 to WM5, reason behind is the whole structure of data storage being change from RAM to ROM, and ROM are flash memories that need time to perform I/O and give the convinence of keeping the data valid even power fails, unlike WM2003 and before. I am quite sure futures UG will not be as problematic, unless MS decided to change the data storage method once again, which is unlikely. Now WM6 is going to roll out, and I am hoping for the UG being offer for my device and give it a try, with or without a cost.  

    As we notice WM2003 device typically have more RAM then ROM or equal, and for WM5 devices have more ROM then RAM. This makes the Upgrade harder to be perfect.

    As for the flash memory wear out and sector become dead, I believe that issue you will not need to worry about it too much, unless you are using your device for more then 5 years, else chances are you not going to use more then 60% of the quota before it wears out.

  184. MikeCal says:

    checko, it should be the case that every device that allows you to write user data to flash will do wear levelling.  Certainly every Windows Mobile device does it.  What matters is how often you write to the flash, not where in the flash you write.  So writing to one file 10 times is roughly the same as writing 10 different files one time each.

    Mike

  185. cristianomc says:

    Does anyone knows a place or a person that can replace my HP ipaq hx4700’s NOR type ROM to a NAND type ROM?

    Thank you!

  186. checko says:

    這一篇msdn的blog 有講到有關nand flash的wear leveling..

    在需要修改某個sector的資料時,並不會將整個block erase掉,filesystem會將原來的那個sector設為invalid sector,然後將修改後的資料寫到未使用的sector上。這樣,漸漸的,flash上invaliid sector就會越來越多,導致flash 的空漸漸不夠。filesystem (filesys.exe)中有一個thread (Compaction Thread)負責..

  187. New says:

    I don’t know hp6515 can upgrad wm5  

  188. cristianomc says:

    I defenitely found a way to make WM5 as fast as WM2003se. Take a look at:

    http://forum.brighthand.com/showthread.php?t=230392

    Of course it has downsides:

    – increased amount of MEMORY used by the system;

    – considerably increased amount of POWER used by the system.

    Besides theese downsides, I’m happy!

    Regards!

  189. HotBoi21MD says:

    I was going to upgrade until now. I’m sure when you start doing some work with it over at MS, you’ll fix it. Until then 03 is all me.

  190. Mazor says:

    There is a fix now!!! I havae thrashed my ipaq, put, everything in the non volitile NOR, and it performs flawlessly.

    Here is the link for instructions for patching your firmware before yuo upload it to your device from Solnyshok

    http://forum.brighthand.com/showthread.php?t=230392

    After having a near unususable ipaq 4700 for over half a yr, now it is fully operational, ready to take on symbian again.

    Mazor

  191. Mazor says:

    Sadboy, it is very useful, that thread. I used the hex editor and used the latest patch firmware to find the matching HEX code I then copied the firmware into my older first install folder for converting wm2003 to wm5. from there I followed the instructions, wherby when it asks to flash now, I change the file name of the original 130mb firmware and replace it with my HEX edited modified, more recent firmware, vola, sinple as that. Have not noticed the additional power use, but man, it is fast, the way WM5 was meant to be :)

    Mazor

  192. cristianomc says:

    You’re right Mazor. The "fix" is useful. I was wrong about the power consumption, it’s the same. But remember also to create a Ramdisk and put at least the IE cache there. Also the registry modification for the Nav Point Service and Active Sync.

    I did it all and the compaction still take time and processor (less than before) but this is not so frequent like before.

    Best regards!

  193. Mazor says:

    cristianomc, I used to do the navpoint modification, and IE cache, but there is no need for it now!! The Pagepool mod, allows for full operation. I have filled my NOR memory to the brim witrh apps, photos, cache etc, and the performance is absolutely fantastic! I have comped my ipaq 4700 against the Dell X51 and it matches it perfectly. The modification should be applied by HP, not by us, users of the device having to do dangerous Hex editing of firmware, etc.

    Mazor

  194. cristianomc says:

    How long have you used the iPaq with the firmware mod? I used about 3 days and then I noted the compaction again (the unit becomes slow to respond to any screen tap), but repeating, it takes much less processing and occours rarely. Have you noted the compaction again?

    Cristiano

  195. mazor says:

    I have had mine for over 3 days now, still going strong! No compaction problems what so ever, simply amazing. Previously loading even divx movies took ages. Now everything works fast!

    Mazor

  196. cristianomc says:

    Try using several times, for several days, the email (read and send), Skype or MSN, IE, Opera, any Street Navigation Software…

    Then let us know!

    Regards!

  197. Bosco says:

    Doesn’t anyone have an answer to the Dell Axim x50v delay? Can anyone suggest anything other than what is on the Aximsite? I realize the compaction is a problem..But what is the answer? Thank You!

  198. Andrew Foraward says:

    SHAME to HP and Microsoft….because:

    1. HP

    HP service told us (IT company), that their hx4700 was bult for the future WM5 system and suggested us to upgrade all our machines.

    A few months ago (around 6 months), HP informed us, that WM 5 Upgrade German is available. So we ordered it an upgraded our units.

    Result, they are usless.

    HP Service told us after many CHAT meetings that this power and speed problems are not known and have to be a Microsoft Problem.

    2. Microsoft

    Well, they realy cant handle each kind of hardware unit.

    BUT as engineer of their OS the should be able to generate a patch or something else to solve the problems.

    Bouth, HP and Microsoft should do all their best to take care of their customers!

    It ist not possible that the customers have to find some work arounds and to do’s to fix this.

    Maybe we have to think about other Hardware and OS.

    Regards

    A. Forward

    PS. How great was the time with our Tungsten T3 and Palm OS 5.* 😉

  199. cristianomc says:

    I think the UMPCs are becoming a good option by this year. The "second generation" promisses. By the way,  the new HTC smartphone to be launched really rocks.

    Regards!

  200. mazor says:

    Many days has past now, played video, opera, IE, multi window IE, etc, playing lots of MP$ videos, played games, still no slowdown!!

    I did set the pagepool to 16Mb, could this be the reason why my ipaq is going soooo good!!

  201. cristianomc says:

    Well, I did mine to 4.8Mb and sometimes it becomes slow. MAZOR, is your 4700 upgraded to 128MB or have the default 64MB of RAM?

  202. mazor says:

    default 64mb ram, 16mb loss, should not be a problem, as in the past I had less than 32Mb in WM2003SE where I installed core software into.

    I believe 4.8Mb is the default setting? Should try 8mb if you are scared of taking too much memory away.

  203. cristianomc says:

    Sorry, it was 8MB. I reflashed yesterday with 16MB and no other registry modifications with exception of the "fake server" for the Active Sync. Lets see what happens!

  204. ipaq says:

    how much ram is available after you flash with a 16mb pp?

  205. cristianomc says:

    There was 25MB of RAM free. But after using the device for some time it dropped to 19MB. Soft reseting brings back the 25MB. I don’t know why this happens but there aren’t workarounds for everything!!

    Regards!

  206. mazor says:

    my ipaq is still going strong!!! no slowdowns or lockups! Pagepool to 16mb is a miracle I tell you all. HP are you listening to this! Total solution. 16mb pagepool!!

    How is yours going cristianomc ?

  207. cristianomc says:

    Until now mine is going great! Much better than with 8MB of PP. Really there is no need to do any registry modifications (althout I did the fake server for Active Sync). But I’m considering a RAM upgrade to 128MB from PPC Techs. Is there a 256MB available? Do you know another company that can  do this service with a better price?

  208. MAzor says:

    cristianomc, Good to hear now your ipaq 4700 is like mine!!! fully operational and fast. Why do you need more RAM for your ipaq?

    What do you use that would need more than 25Mb free RAM? All the current photo editors now know how to use swap to SD card or persistant storage. I know in the past RAM used to be an issue with large photos, but no more now.

    From what I head, based on the type of RAM that the iPAQ 4700 uses, it cannot be stacked and is a total replacement, hence maximum RAM 4700 has address is 128Mb.

    The only reason I would consider upgrading the 4700 to 128Mb is if I had WM2003SE, where half my RAM was used for inbuilt applications and my documents or if I wanted to run a full blown linux GUI OS on my ipaq.

  209. cristianomc says:

    Because I have a Navigation Software called "Apontador Duo" (it’s from Brazil, where  live!) that eat my RAM! When programing a route, I must connect to the brazilian Webraska servers to download the route. Webraska says that it’s the best way the users have to have the maps always updated. If I choose to download a route with 20Km of radius and displaing the streets distant 450m of the entire route it wil take about 3MB. But this 3MB are unzipped 5 times on the RAM for visualization of the route. So it goes to 15MB. Adding more 2.5MB of the running software with these 15MB I have 17.5MB overloading my RAM. There are stil those running services of the WM5 including the SD card GPS connectec to the SD port. Resuming: the device locks sometimes.

    This is why I need more RAM!

    But PPC Techs RAM upgrade is too expensive for me because I have to pay an expensive shipp to USA and and back to Brazil.

    Best regards!

  210. Gordon James says:

    I have an HP iPAQ rx 1950 which has Windows 5 for PC. This should support Thai characters. Unfortunately they enter as Thai characters and are then reduced to squares. HP Thailand have admitted they are unable to help in this matter. Mr Hurd the CEO in the US has sent three e-mails to HP Thailand requesting they offer assistance. HP customer support in Thailand have failed on all occasions to obey the instructions of their CEO.

    Can anyone recomend a reliable pocket PC with reliable customer service. HP couldn’t care less about their customers.

    Please e-mail me if you can offer advice or help.

  211. ipaq says:

    Now running a 16mb pagepool and confirm that it’s much better than 8mb.  all my games run with the exception of thps2, no great loss.  Cod2 still plays great on my hx4700, super smooth.

    i’ve managed to exract the 2.01.05 rom using the xda  imgf tools and add/remove components but what do hp use to generate rom’s? i think if i have an extracted rom i could use what the pros use and maybe update the aku or add .net cf sp2 to the image and ajust other memory pools dma buffers etc..

  212. mazor says:

    Good work Ipaq, glad people are finally enjoying their WM5 ipaq 4700 after such a long time.

  213. cristianomc says:

    I’m having some troubles but I don’t know if they are related to the change made on the PP (16MB). My unit is hanging sometimes (at least once a day), always when using the wireless (Opera, Skype, MSN or Messaging), and the remmaining battery shown on the Today Panel is wrong sometimes (ex: it shows 65% but when I try to turn the device on it doesn’t turn on. So I plug into the wall and it turns on showing the battery with 0%). I did set up the backlight to turn off after 1 minute idle and it’s not working. It turn off and imediatelly turn on again. When I flashed everything was ok but using the device these problems start occour.

    Any suggestion?

    Thank you!

  214. Mazor says:

    I am not sure what could be wrong with ur unit. I have been pushing my unit real hard since I have gotten it usable in WM5. Watching videos, Divx, etc. It is simply brilliant! My unit hangs only after about a week of no resetting, and that is due to loading too many 3rd party applications. My battery seems ok. Did you even remove your you battery during your upgrade? If so, check the battery contacts are clean, that could be what is causing your fake battery levels.

    Mazor

  215. ipaq-mk says:

    best of both worlds, all memory and no compaction issues.

    set the pagepool to 0, replace 48 with 00 when you hexedit the rom.

    try it! HP should have not missed this !

  216. IPAQ2410 says:

    I have decided to try WM5 on my 2410. I have read every thing I can. I would like to change my PagePool. When I try to mofify my nbf I get an error saying that the verify is not correct and the process stops. Can anyone help me get past this part.

    I have used the BIOS modify program that came with the new BIOS and the one that runs with the upgrade. They are both ROM Update utility 2.02.014. I get error 608 and 5xx ( 5 something – I forgot to copy)

    I just bought the upgrade last week and I have the BIOS that gives:

    BootStrap Ver 1.00.04

    Bootstraploader 1.00.00.L

    I also used SP32878 (BIOS) and SP33653( Drivers)

    I am using the tool XVI32U to edit the nbf. I also note that I have two nbf files. One in folder C:IPAQ WM5H and one in C:IPAQWM5L. I thnk the 5L is the one to modify but I did try changing both.

    When I put the nbf files that were unpacked I can download to the unit.

    HOpe I have all of the detail -anyone still reading this?

  217. MrHX4700 says:

    I own a hx4700. I wish I had read this useful blog before upgrading to WM5, but now I’m in the mud hole. So I have read the details of the fix, but it is still something highly technical people might do. For the technically challenged people, does anybody know if the the fix (or surrogate) will be or has been incorporated into an official HP patch, or if there a more user friendly version of the instructions (for example, if I need a hex editor, then where to find it..and a clearer explanation of the pre, port 2.01 steps…etc…) Thanks for any pointers

  218. MrHX4700 says:

    Never mind everybody. I was able to do the fix Mazor’s link (earlier in this blog) documents.

    (http://forum.brighthand.com/showthread.php?t=230392)

    It is working like a champ on my HX4705.

    This is what I did–Note: this is what worked for me and by no means I suggest you will be equally successful…but you will find the steps I followed more straight forward that what the link describes…good for those of us who have less technicall prowess that some of the people in this blog and link….so proceed at your own risk.

    I did it somewhat differently and simpler.

    1) I made a backup of the 2003-to-WM5 upgrade installation software I purchased from HP.

    2) Ran the 2003-to-WM5 installation, which extracts and installs the upgrade files and application on my PC.

    2) Searched for the Hex editor recommended by the link using Google and installed it on my PC.

    3) I did a file search using Window’s file on files having "nbf" as part of the name and found only 1 file called CEOS_UPG.nbf in the WM5 update folder. Not knowing whether this was the "nbf" file the link was alluding to (need to say none of the names mentioned in the link matched anything I saw in the upgrade folder), I searched by the HEX pattern documented in the link and found it exactly. So I deducted this must be the file that needed updating–I had nothing to lose as my iPAQ was really in very bad shape…plus I figured I could re-install the whole upgrade as delivered or go back to 2003 (yes, the HP upgrade software does come w/ software to return to 2003).

    4) Made a copy of nbf file, opened the new copy using the Hex editor, searched for the text instructed by the link, and made the update in some other windows folder. I updated the size to 8.0M as suggested in the link.

    5) Ran the update application to the point where the application prompts you to confirm proceeding w/ the update (iPAQ needs to be already in the craddle by  now).

    6) At that point, I replaced the delivered nbf file extracted earlier w/ my new version and proceeded w/ the installation and confirmed to proceed w/ the update.

    Works great so far. I have tried different things that were giving me awful performance problems and so far it is much better.

    I need to admit though there is a possibility that my success could have been due to the second installation (something not updated properly when I first applied the update)…so if you don’t want to have any doubts like me, try a second installation without doing the PagePool code update to see if anything improves…then proceed at your own risk w/ the steps I followed above.

  219. Eaglesteve says:

    Mike,

    It seems to be that compaction thread is a real problem to WM5 users, as it kicks in at the moment when we don’t expect and hangs on to the resources for too long.

    Would it not be possible to do compaction thread in a way similar to defragmentation? Have a separate utility program that could be scheduled by users.  We can run it say at 1 a.m in the morning, and if it does not finish by say 6 am, stop and continue the next day?  Also, make it possible to interrup the compaction thread process anytime, and continue later when we wish to.

  220. Eaglesteve says:

    Mike,

    My device is a Dopod U1000, and the compaction thread like behavious is being experienced by many fellow users, causing too much pain.

    Since there is a microdrive on the unit, do you think that performance would improve if we keep all our data there, and use the SD card just for programs?

  221. Alexander says:

    Is there a way to start compaction manually? Its a good solution, so, for example, i could manually start compaction to prepare PDA for that moment, when i extremely needed of quick response from device.

  222. Hugo Leonardo - Brasil says:

    Hi Friends

    I am a Brazilian user of a HP IPAq model HX2750 and would like to know their opinion on the update of this equipment for the use of the Windows Mobile 5.0

    I do not have any problem with the Windows in version 2003 but it would like to explore a greater potential of IPaqHX 2750. I believe that for its power of processing I can obtain excellent results with the upgrade of the system.

    Thanks!!

  223. sean r. says:

    after installing 5.0 on my hp 4700 it does not sync. done all the firewall stuff.

    is there an answer to this issue?

  224. Arenkay says:

    Well, I was looking to change my hx4700 O/S from Windows CE to WM5 because I want to run CoPilot Pocket PC6.  But after eading all the obove, I think I’ll forget the idea.

    Anyone know a good GPS application that’ll run on WCE?

  225. mochacat says:

    I have an iPAQ model 2490 purchased with WM5 already installed – no upgrade.  Is there any way to get Windows 2003 on it?  We made the switch thinking latest and greatest and it has turned into a huge mistake….   Please help.  Thanks,

    Lisa

  226. C F Hyder says:

    I lost my CD pack that came with my HP IPAQ rw6828. I was having two problems which is why I had to clean boot the device (Not soft reset). Also I had some problem with my notebook which is why I needed to install Active Synchronise 4.1. But I’m having the following problems:

    1) Each time I am connecting the device and activating Modem link, "windows usb modem" is found and thereby looking for the right driver. Whereas, it should have been covered under the active synchronise software (which I downloaded from Internet) from Windows website.

    2) I am setting all features of dialup connection through bluetooth modem. But I cant get connected although everything is ok.

    3) I need to know if there is any PAN netwark connection facility available in the rw6828 through which I can connect using my Nokia 6300 mobile adhoc (device to device) blue tooth network card.

    Will sure appreciate your response to my email: cfhyder@gmail.com

  227. max says:

    Quite an old thread, but fascinating just the same. One thing I was curious about after reading this. Mike mentions several times about how the end user can’t do alot of things because of the inability of changing the ROM. Ofcourse, now with the various tools available on XDA and other places (like mamaich’s rom tools, bepe’s tools, rommaster etc.) we CAN make many changes. So, anyway, I was wondering…he mentions that during testing they used the SD Card as internal ROM…Quote from MikeCal:

    "Jugagagah, it’s possible to mount the internal storage on an SD card.  We did this during WM5 development to let us do work on a device that didn’t have enough flash available.  However, the system can’t handle it’s persistent storage being removed on the fly.  If you were to have your storage mounted on an SD card and you ejected the card, the system would crash.  That was acceptable for development, but we’d never make end users put up with it.  Even if you personally found it acceptable, it’s not a change you can make yourself.  The image itself needs to be changed, and only your OEM can do that (not even Microsoft can).  You might think you could make a registry change or something like that, but remember that we’re taking about the storage system here.  The registry is IN the storage system.  So it’s loaded after the storage system is mounted.  The change needs to be more fundamental than that to take effect in time."

     So,now with the ability of cooking our own ROMs, or atleast tweaking them a bit, now available to us, wouldn’t it be possible to set up the ROM to put the internal storage on the card and just be sure to never remove it. With two slots available on a hx4700, that wouldn’t be so bad. Then you could you have:

    NAND on the flash,

    NOR on  the internal,

    and your RAM. ???

    A combination witch he mentions as being the ideal route taken with the Treo 800.

    Is this possible to do? Could we cook a ROM that takes advantage of the NAND of a flash card (never to be removed) and the NOR of internal (using XIP too?)  

    The question was asked once in this thread earlier but never answered. Mike isn’t going to follow up on this kind of question obviously because it involves the cooking of ROMs and all that which MS certainly can’t sanction. But just for the sake of my understanding, after reading this fascinating and enlightening thread. Could you  do this?  Please set me straight. Thanks.

  228. Jeff says:

    If the device is soft-reset in the middle of a compaction is it possible to corrupt the file system?