What Happened To NOR?


I've been asked why OEMs would use NOR flash in a persistent storage world.  (If words like "NOR flash" or "XIP" are confusing, read this.)  I'm going to give you some history about where NOR flash came from, explain how we got from there to here, and prognosticate a bit on the future.



Who was that masked chip?


The first PocketPC to have any sort of embedded flash memory was the original iPaq (it was released in 2000).  Before that, all our devices shipped on what was known as "Mask ROMs."  A Mask ROM is created at the factory and can't ever be erased or changed.  Obviously, we never stored user data in these (there was no way to write them), and the only way to ever "upgrade" a device with a Mask ROM was to physically remove the chip and put a new one in.  (Yes, I'm ignoring EEPROMs.  They were using during development, but few, if any, devices were shipped to customers with them.)


It's important to note that code in Mask ROMs could be executed in place.  That is to say, they XIPed. 



Enter the Flash


Now, NAND flash had been around for a long time.  Those old Mask ROM PDAs had CF slots, and the CF cards were made of NAND-based Flash memory (frequently called Flash ROM even though that's not technically correct).  NAND, however, can't XIP, so it couldn't be used as a direct replacement for Mask ROMs. 


NOR flash, however, had the very handy aspect that it could XIP.  This allowed hardware designers to easily replace their Mask ROMs with NOR flash.  And that meant that devices could conceivably be upgraded without physically opening the device. 


This was such a compelling feature that the sole supplier of NOR flash was able to charge a hefty premium for NOR over NAND.  In the first iteration of PocketPC 2000 devices, only the iPaq had NOR flash.  Everyone else had Mask ROMs.  The iPaq, however, was so popular that every one of its competitors quickly followed suit.  Shortly thereafter, all PocketPCs were using NOR flash.



Rebirth of a NAND


Remember that sole supplier and the hefty premium?  Such things never last.  In the 2000 and 2002 releases, our OS completely relied on XIPing, which largely locked people into NOR flash.  As a result, NOR became so expensive that is started to cost more than NAND and RAM combined.  (16M of NOR cost more than 16M of NAND + 16M of RAM.)  When that happened, some clever OEMs started putting in NAND and an extra chunk of RAM.  Then, at boot time, they'd copy all of the code out of the NAND into the RAM and run from the RAM (you can XIP from RAM). 


We got a lot of OEM requests to enable NAND in a more reasonable way.  It was overkill to copy the entire image into RAM.  So, in our 2003 release, we enabled a scheme that allowed NAND to be a first class citizen in PocketPCs and Smartphones.  NAND devices would need more RAM than equivalent NOR devices, but only a few megs more, not the entire image's worth.  Not needing that couple of megs of RAM was a definite advantage for NOR.  But the price differential was so large between NAND and NOR that most OEMs switched to NAND. 



Persistent Storage and NOR


To make things worse for NOR, Persistent Storage also entered the picture.  The first MS Smartphone was released in 2002.  It was NOR based, XIPed, and had persistent storage.  It's extremely difficult to do persistent storage and XIP in the same chip at the same time.  There was technology to do it on those 2002 smartphones, but their performance, especially when storage got tight, wasn't very good. 


While we were developing the 2003 smartphone release, we found that if we stopped XIPing and treated the NOR as if it were NAND we got a substantial speedup in persistent storage.  When OEMs learned that they could get a performance benefit by treating their expensive NOR like inexpensive NAND, most of them switched to NAND flash. 


To make things even worse for NOR, it's much faster to erase and write NAND than it is NOR.  So, even when you treat NOR as NAND, NAND outperforms it. 



Why use NOR then?


There are a number of reasons.  For one, as NOR's lock on the market eroded, so did its price differential.  These days NAND and NOR are in the same ballpark in terms of cost.  For another, if persistent storage isn't involved, then XIPing is better than not XIPing.  XIPing saves memory and launches apps faster.  For instance, we saw a significant improvement in performance when we first enabled XIP on the Treo 700w. 


In the final days of non-Persistent Storage PocketPC devices (pre-WM5), the choice between NAND and NOR was pretty difficult.  The prices were in line with each other, the XIP benefits were tangible, and the erase/write time differences didn't matter. 


Also, some processors come with a sizeable chunk of NOR flash built in.  That reduces the size of the circuit board and makes NOR extremely competitive, price wise.


Devices like the 700w are poster children for great uses of NOR.  The 700w has 64M of NOR and 64M of NAND.  It stores the OS in the NOR and XIPs it, and it stores the user data in NAND.  That gives it the best of both worlds in terms of performance.



The future is … murky


So, what does the future hold?  As always, that's hard to predict.  Hybrid devices with some NOR and some NAND show promise, especially when the NOR bit comes "for free" in the processor.  On the other hand, some NAND providers are building a bit of RAM into their chips and making them act like they XIP.  The hope there is to get the read benefits of XIP with the write benefits of NAND.  There are also upcoming technologies that are promising the persistence of Flash with the speed of RAM.  These devices could conceivably replace both NAND and NOR.  Your guess is as good as mine which of possibilities will prevail. 


Mike Calligaro


Comments (14)

  1. Braden says:

    So how much RAM does a 700w have then?  I see NOR and NAND, but you can execute stuff that’s loaded on a CF card or dumped to the NAND, so where does it put that executing data?

    Is it true that the 700w only has a paltry 32mb?  Many "power users" have shunned the 700w because they claim it runs out of RAM too quickly when you get any real 3rd party apps on it…

  2. MikeCal says:

    Only programs built into the base image can XIP.  Code installed later (whether to SD or the internal flash) is loaded into RAM first and executed from there.  For this reason, the Treo has a page pool.  It’s smaller than a NAND-based device’s page pool, but it exists.

    The Treo does have 32M of RAM.  I disagree that the amount is "paltry."  There are a huge number of users, myself included, who think it’s a great device and use it daily without memory problems.  

    One of the largest sticking points for the Treo was an email app that tried to allocate 7 megs of RAM at launch.  You can blame the Treo for not handling that app well, but it’s ironic that people complain about Microsoft "code bloat" and then try to run 3rd party apps like this one.  Media player doesn’t even use that much RAM while it’s playing video.  What is the email app doing with so much RAM?  There are some 3rd party apps that have a legitimate use for extravagent amounts of RAM.  But most of them really make you wonder if the ISV understands that this is an embedded system without the unlimited resources of the desktop.

    Mike

  3. Akac says:

    Mike, I’m guessing you’re talking about FlexMail where it was/is a huge memory hog. If so, yes we knew about that as an issue but there was not much we could do about it originally because by the time we realized it we had too many people using the data store for us to convert. Originally FM was designed with the idea that it would only load what it needed. However, when the storage tech we chose was given to us, it required ALL storage to be loaded into RAM. We found out the hard way. Unfortunately its not easy to change storage technologies midway through development. And that’s why it would take that much RAM. 75% of Mail’s memory use was the SolFS storage engine we used.

    The interesting part is that the same app ported to a custom WCE phone uses 1MB of RAM for the same because of changes in the data store. Those same changes we’ve now put into the next rev of the FlexMail product (using CEDB/EDB) and now the product is dramatically faster AND uses dramatically less RAM than before. We’ve got beta builds that we’re running on the Treo 700w and its working beautifully.

    So to say the ISV doesn’t understand is not always right – sometimes a design decision years ago affects us now and we don’t have the benefit of having a fresh OS install to work with. Reading this blog, I’d think MS would know that quite well.

    BTW – why can’t I run Messenger and IE simultaneously on the Treo? I run IE and go to a few web pages (with images turned off to save cache RAM) and then go to Messenger to send an SMS. I send it and go back to IE and it has to start up again because the OS turned it off. This is a fresh reboot with nothing extra running. I think that shows that the Treo does have paltry memory.

  4. MikeCal says:

    Akac, I’m not going to start naming ISV apps that use what I consider to be too much memory.  What I think is “too much” might be perfectly acceptable to someone else.  

    That said, for every person who thinks the Treo has too little memory, there are at least 10 people who have sent us feedback that they want our devices to cost less and last longer on their batteries.  Reducing the amount of RAM in the device contributes directly to both of these very popular customer concerns.

    And, when just starting a single ISV app takes 50% as much memory as the entire operating system, that app puts a severe cramp on a OEM’s ability to solve its customers biggest concerns.  That’s a problem, regardless of what the app is or what it’s doing with the RAM.

    I have this amazing job where they give me a lot of different devices to use.  When I leave my office at the end of the day and need to decide what to bring home, I frequently have to choose between four different devices.  And, whenever possible, I choose the 700w, 32M of memory and all.  

    Here’s a common scenario for me.  I’m reading SciFi Wire in pIE.  I bring up pMSN and check to see if I have any new Hotmail.  Then I switch over to my exchange email.  Because the exchange email is always up to date, I see that I have a new meeting request from a coworker.  So I check my schedule, see that it’s fine, and accept it.  The acceptance gets synced up.  (To belabor the point, here I made a phone call by tying in the first name of the repient, which used both contacts and the dialer.)  I go back back to pIE and it’s still on the same page.  I don’t use SMS very often, but I literally just did the above.  Since you talked about IM, I then signed in to pMSN’s IM, sent a message, and signed out.  pIE still has the page I was reading.

    Your mileage my vary, but this device WORKS for me.  And judging by how its selling, it works for a lot of people.  

    Maybe it’s because I’m old school when it comes to embedded development (I started writing code for these devices when they only had 1M of RAM).  But I just don’t consider 32M to be paltry.  

    Mike

  5. oxynorom says:

    Thanks for sharing the view.  "Old embedded school" programmers valued their ROM and RAM bit by bit, because they do not have extravagant amount of memory to start with – as in desktop PC. Yes, 32MB is a huge amount of RAM to play with but not everyone is graduating from "old school". I’ve seen quite a few apps for embedded system by third parties which take 50% ROM space just for drivers and in those "special" occassions the processors are often needed to upgrade .

    Back to the future, implement NOR + NAND + RAM in one device is a complex issues in hardware and software design. Traditionally, NOR is Fast read Slow write but realiable, NAND Slow read Fast write (need errror correction) and RAM is Fast read and Fast wirte but consump power, compares to these others two. Most of all, we are power hungy (don’t get me wrong) : we want our device last longer and run faster !!!

    Having more RAM does not neccessary means more power needed if the RAM’s speed is faster, much faster . The faster RAM will require less time to tranfer the same amount of data. Hence, we may get even or better for more RAM in the same design, guess who will pay for the speed and effiency ?  

  6. MikeCal says:

    Oxynorom, the problem with RAM power burn isn’t the couple microseconds while you’re writing to and reading from it.  The problem with RAM is the 24 hours a day that it burns power refreshing itself.  

    Flash memory only needs power when it is being read from, written to, or erased.  RAM, however, needs a constant source of power to remember its data.  And the amount of power it needs depends on how much total RAM exists, not on how much of it is being used at a given moment.  So, if you have two devices that are otherwise identical, the one with more RAM will burn more power and last a shorter time on its batteries.

    As technology improves, RAM burns less power than it did in the past.  A device designed today will use RAM that burns less power than a device designed last year.  We’ll definitely get to a point where 64M of next generation RAM burns less total power than 32M of some previous generation’s RAM.  Maybe we’ll get to a point where the amount of power RAM consumes won’t matter compared to other things in the system.  But we’re not there yet.

    To give you an idea where we are with power, consider the little green LED that blinks whenever you’ve got a data connection.  On a device I was working on recently, I increased the total device standby time by 12% by reducing the amount of time the green LED was on from 200 milliseconds to 50.  We’re talking about almost a day of standby time here.  

    LEDs and RAM are different things. But what I’m trying to show is that we’ve already gotten rid of all the big power consumers and have reached the point where small things seriously matter.  

    If ISVs treat these devices as having desktop-like unlimited resources, then we’re going to have trouble making our batteries last longer.  We’re already seeing that.  With reviewers and end users calling 32M "paltry" and ISVs complaining that their unecessarily enormous apps don’t work, do you think we’ll ever see another 32M device from anyone?  

    What a shame for the 90% of consumers who don’t use these giant apps that they’ve got to suffer the battery life required by a small but vocal minority.  

    Mike Calligaro

  7. akac says:

    Mike, I use a Treo 700w daily and its my preferred device. I am not knocking the Treo. Now what I saw with Messenger and IE is something that happens daily to me, but not every single time. My point was simple.

    I understand the reasons for having 32MB of RAM. I DO think its paltry because of the issue I mentioned above and if I use a GPS app it barely runs.

    The problem you’re having is you’re not thinking of data. Code, yeah. Code needs to run in that much RAM or less easily. But when you’re having to deal with performance – you’re going to need RAM. I could page a bit of data and keep 4k in RAM at all times, but man its going to take forever to do anything if my data set is 50MB as I keep reading in 4k at a time and only deal with that amount in RAM. Sure my RAM use is tiny but the app is unusable.

    A perfect example is Pocket Informant’s Contacts. We have a need to sort on the contacts data in ways that the built in database system’s just cannot do. So to do so we create as small as possible in-memory cache of the specific fields we have to sort on. But this requires using an extra bit of RAM depending on the size of the user’s contacts database. If we did a sort that kept only a certain amount of RAM in use the contacts list would take dramatically longer to come up and nobody would use it.

    So while I am personally comfortable and have plenty of experience writing apps working in a memory space of a TOTAL of 2MB on current devices (feature phones), I also know that there is a balance between performance and memory use. Every time I update PI I go through it and try to find every byte I can squeeze. I cringe at adding a 2-byte member to a struct that’s going to be used a few thousand times. But at the same time customers demand more and while I use many tricks to keep my usage down, sometimes its just necessary.

    Maybe its just me, but I’m gathering an attitude towards my post and I don’t know why. I almost feel as you are personally hurt by me saying that the Treo’s RAM is paltry. I don’t know why. Its just a piece of hardware. A piece of hardware I personally depend on daily. Its the best PPC I’ve ever had. But I’m not blind to its defects. Just like I love my software that I right. But I’m not blind to their defects and I will gladly take criticism that’s specific and usable. BTW – I’ve seen IE take up 10MB of RAM easy. You realize that an entire 1.5MB of FlexMail’s memory footprint can be an HTML email using the PIE engine?

    I hope I’ve made my points. I’m not criticizing – I am trying to clear things up on our end.

  8. MikeCal says:

    There are definitely applications that need a large data set.  I’ve written a few myself.  I still remember being excited when the first 16 meg device was released because it allowed me to write a Japanese study program that needed 4M to opperate.  

    Akac, I want to make it very clear that I don’t think you folks did the wrong thing with FlexMail.  You were hit by an unexpected circumstance that you realized late in development.  It was too big to change then, so you shipped it, and set to work rearchitecting the next version to solve the problem.  That’s good software development in my book.  

    The important difference between what you’re doing and others aren’t doing is that you’re FIXING the app.  You recognize that 7M is too much for an email app, and you’re changing it.  

    You’re not the problem I’m fighting against.  The problem I’m fighting against is the large number of ISV apps that DON’T understand that 7M is an enormous amount of RAM for an embedded system.  

    I need ISVs to understand that phones are constrained systems, where RAM is precious and not to be consumed without good reason.  You and your developers clearly understand this.  There are many who don’t.

    Mike

  9. Matthew says:

    Part of the problem is the platform. I try to be very concious of wasted space, but sometimes I can’t help it. For example, to create an MFC application I must staticly link in MFC to have any hope of it working. That adds at least 800K to the file. The only other alternative would be to create my own MFC build and load that DLL, which would provide a savings for mulitple programs should I provide a set.

    Now, why must I do this wasteful thing? The reason is DLLs are handled improperly. The MFC in the Windows directory is useless. I tried just that, but half my test devices would claim my .EXE was not a valid program. Bring allong the proper DLL provided with VS2005 and now it works. I’m not just talking about devices without the MFC8.0 DLL, but rather devices that have some other version of it. If I overwrite that file, then I may break something else (bad) and its just waiting for someone else to come do the same to me. So, I could just put it in my own program directory, except if another MFC program is already running then my program would attempt to use that already loaded MFC DLL because the system will just ignore the DLL in my own directory. To avoid this, I would have to have my own build of it, but now I’m loading yet another DLL. As I understand it, that causes memory pressure due to the way DLLs are loaded to allow them to be shared. By static linking, at least all the MFC cruft is in my own process slot and not eating up room in the shared DLL slot.

    This is but just one example of a system not really architected for 3rd party development. It seems CE was ok for embedding in simple stuff, but not really designed to handle a myriad of 3rd party applications being loaded.

  10. badbob says:

    With the Treo 700w, you mention that it has a split NOR / NAND storage configuration, where NOR is for XIP and NAND is for user data. So does this mean that the NOR portion of the Treo is readonly and that any data that needs to persist (such as the registry) is stored to NAND? If so, is this configuration just one of the possible configurations of WM5 provided to OEMs? Was the 700w designed with this storage configuration with WM5 in mind or did it inherit its design from its Palm lineage?

    On my axim x50v, I’ve been trying to keep writes off my NOR as much as possible by installing all my programs to a SD card and moving all cache/temp files to the card as well. I suppose any further efforts would require a new oem image. 🙁

  11. is that LED blink setting a registry change? I’d like to make it to my iPAQ if it is, so posting it would be a public service 😉

  12. MikeCal says:

    badbob, effectively yes.  The NOR part isn’t completely readonly, because it can be reflashed for system updates (like the one that updated the Treo to AKU2).  But user data isn’t stored there, so it’s effectively readonly most of the time.  As you said, this is just one of many possible configurations WM5 provided to OEMs.  I’m not sure what the base reason for them doing it that way was, but it sure is a great way to do things.  All the benefits of XIP NOR with none of the negatives of trying to store data in it.  

    Mary, the LED driver is written by the OEM, so different devices will act differently.  I’m not aware of any OEMs who have put the blink rate in the registry.  I’d personally like to disable the gsm blink on my devices.  I don’t find the power drain worth it.  Others may disagree with me, though.

    Mike

  13. Glyn Davies says:

    Q1. If it possible to use some ram up as an EWF with Windows Mobile, in a similar way to WinXP Embedded?

    All this hacking around with ramdisks and the like is a bit of a.. well, hack.

    Q2. I was also just looking on the HP website for this magical update for the hx4700 – can’t see anything there except an update dated July 26 (and signed July 27th 2006) which claims to be version 2.01.05

    My iPaq reports 2.01 on startup – does this mean it is not the latest?!

    Ta,

    Glyn

  14. Glyn Davies says:

    Scratch Q2 – running the update tool says I have 2.01.05 installed, its just the startup screen doesn’t display the minor numbers

    G

Skip to main content