Page 1 of 6 123456 LastLast
Results 1 to 20 of 118
  1.    #1  
    I'm currently working on adding the compcache modules to UberKernel, and as a result of that I'm getting deep into understanding the infamous Too Many Cards (TMC) error.

    The goal here is to retain the TMC error handling, but to make it take into account compressed RAM swap space provided by the compcache modules, so that TMC errors don't happen until both normal RAM and compressed RAM swap space are almost full.

    Note that the current experimental compcache kernels from Marco (F102, F104, etc) completely disable the TMC error handling altogether, meaning that when you finally do get to a memory full condition, your kernel will simply crash and cause your device to reboot. Obviously, that's not suitable behaviour for the mainstream UberKernel.

    So I'll use this thread to document my findings for posterity.

    The TMC errors come as a result of Palm's memnotify kernel module, which is implemented in drivers/misc/lowmemnotify.c in the kernel source tree.

    There are 4 thresholds: Normal, Low, Critical and Reboot:

    Code:
    static unsigned long memnotify_messages[6] = {
            MEMNOTIFY_NORMAL,      /* The happy state */
            MEMNOTIFY_LOW,         /* Userspace drops uneeded caches. */
            MEMNOTIFY_CRITICAL,    /* Userspace should attempt to stop new allocs */
            MEMNOTIFY_REBOOT,      /* Super critical point...we should reboot */
    };
    These thresholds have enter and exit conditions, based on the percentage of memory in use (more of the definition of that term later):

    Code:
    static size_t memnotify_enter_thresholds[6] = {
            0,
            95,
            100,
            120,
    };
    static size_t memnotify_leave_thresholds[6] = {
            0,
            90,
            90,
            0,   /* You can't leave the reboot threshold. */
    };
    The percentage of memory in use is based on the total real memory available (which is fixed at about 244MB for a Pre, and probably about 500MB for a PrePlus), and the amount of free memory, which is calculated as follows:

    Code:
    unsigned long memnotify_get_free(void)
    {
            unsigned long free;
            unsigned long other_free;
    
            free = global_page_state(NR_FILE_PAGES)
                    + global_page_state(NR_SLAB_RECLAIMABLE);
    
            other_free = nr_free_pages();
    
            if (other_free > totalreserve_pages)
                    free += other_free - totalreserve_pages;
    
            return free;
    }
    The values resulting from these calculations can be seen using the following command:

    Code:
    root@palm-webos-device:/# cat /sys/class/memnotify/meminfo 
    Used (Mem+Swap): 144MB
    Used (Mem): 143MB
    Used (Swap): 0MB
    Used Ratio: 58%
    Free: 100MB
    Current Threshold: normal
    Last Threshold: normal
    Enter Thresholds:
    normal: 0, 0MB, Rem: -144MB:
    low: 100, 244MB, Rem: 100MB:
    critical: 114, 279MB, Rem: 134MB:
    reboot: 120, 293MB, Rem: 149MB:
    Leave Thresholds:
    normal: 0, 0MB, Rem: -144MB:
    low: 94, 230MB, Rem: 85MB:
    critical: 108, 264MB, Rem: 120MB:
    reboot: 112, 274MB, Rem: 129MB:
    root@palm-webos-device:/#
    The important value (free memory space) can also be seen in /proc/meminfo (it's the MemchuteFree line):

    Code:
    root@palm-webos-device:/# cat /proc/meminfo 
    MemTotal:       244772 kB
    MemFree:         66240 kB
    MemchuteFree:    96524 kB
    Tmpfs:             348 kB
    Buffers:          1788 kB
    Cached:          30096 kB
    SwapCached:         44 kB
    Active:         133728 kB
    Inactive:         7148 kB
    SwapTotal:      131064 kB
    SwapFree:       130836 kB
    Dirty:              24 kB
    Writeback:           0 kB
    AnonPages:      108960 kB
    Mapped:          26428 kB
    Slab:             5712 kB
    SReclaimable:     1348 kB
    SUnreclaim:       4364 kB
    PageTables:       1640 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    CommitLimit:    253448 kB
    Committed_AS:   697740 kB
    VmallocTotal:   647168 kB
    VmallocUsed:     11200 kB
    VmallocChunk:   632828 kB
    root@palm-webos-device:/#
    However, experimentation has shown that the MemchuteFree value does not always match the value which directly influences the TMC error (the Free line from /sys/class/memnotify/meminfo):

    Code:
    root@palm-webos-device:/# cat /sys/class/memnotify/meminfo
    Used (Mem+Swap): 221MB
    Used (Mem): 153MB
    Used (Swap): 68MB
    Used Ratio: 90%
    Free: 22MB
    ...
    root@palm-webos-device:/# cat /proc/meminfo 
    MemTotal:       244772 kB
    MemFree:         32276 kB
    MemchuteFree:    89488 kB
    ...
    root@palm-webos-device:/#
    This discrepancy will be the subject of a subsequent post.

    There is a script which configures the lowmemnotify/memchute thresholds on boot (it's in /etc/event.d/memchute):

    Code:
    root@palm-webos-device:/# cat /etc/event.d/memchute 
    # -*- mode: conf; -*-
    
    start on stopped finish
    
    pre-start script
        echo "0,100,114,120" > /sys/module/lowmemnotify/parameters/thresholds_enter
        echo "0,94,108,112" > /sys/module/lowmemnotify/parameters/thresholds_leave
        echo "/usr/sbin/oom_late_helper" > /proc/sys/vm/oom_late_helper
    end script
    
    exec /usr/sbin/memchute
    respawn
    root@palm-webos-device:/# 
    root@palm-webos-device:/# cat /usr/sbin/oom_late_helper 
    #!/bin/sh
    
    touch /var/state/OOM_reboot
    /sbin/tellbootie oom
    root@palm-webos-device:/#
    Now, you might think that we could just bump all those thresholds up to 200% and be done with TMC errors forever. Let's try it:

    Code:
    root@palm-webos-device:/# cat /sys/class/memnotify/meminfo 
    Used (Mem+Swap): 175MB
    Used (Mem): 175MB
    Used (Swap): 0MB
    Used Ratio: 71%
    Free: 68MB
    Current Threshold: normal
    Last Threshold: normal
    Enter Thresholds:
    normal: 0, 0MB, Rem: -175MB:
    low: 200, 489MB, Rem: 313MB:
    critical: 200, 489MB, Rem: 313MB:
    reboot: 200, 489MB, Rem: 313MB:
    Leave Thresholds:
    normal: 0, 0MB, Rem: -175MB:
    low: 94, 230MB, Rem: 54MB:
    critical: 108, 264MB, Rem: 88MB:
    reboot: 112, 274MB, Rem: 98MB:
    root@palm-webos-device:/#
    With the device in this state, I can't open a 3D game which has an appinfo.json file that requires 70MB of free space (as there is only 68MB free).

    I can keep opening other cards though:

    Code:
    root@palm-webos-device:/# cat /sys/class/memnotify/meminfo       
    Used (Mem+Swap): 282MB
    Used (Mem): 190MB
    Used (Swap): 92MB
    Used Ratio: 115%
    Free: -37MB
    Current Threshold: normal
    Last Threshold: normal
    Enter Thresholds:
    normal: 0, 0MB, Rem: -282MB:
    low: 200, 489MB, Rem: 207MB:
    critical: 200, 489MB, Rem: 207MB:
    reboot: 200, 489MB, Rem: 207MB:
    Leave Thresholds:
    normal: 0, 0MB, Rem: -282MB:
    low: 94, 230MB, Rem: -52MB:
    critical: 108, 264MB, Rem: -18MB:
    reboot: 112, 274MB, Rem: -8MB:
    root@palm-webos-device:/#
    The browser stops opening new windows at about 16MB free it seems, but Preware happily keeps reading in large feeds way past that point.

    -- Rod
    Last edited by rwhitby; 06/14/2010 at 01:33 AM.
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  2. #2  
    Interesting read. Have you noticed any relationship between this and the browser gray-out/refresh?
  3.    #3  
    Quote Originally Posted by DrHeathenScum View Post
    Interesting read. Have you noticed any relationship between this and the browser gray-out/refresh?
    Yep, that seems to happen when the Low threshold is hit. Any non-active browser windows seem to have their caches cleared to free up memory.

    -- Rod
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  4. #4  
    I'm no programmer, but is this something that Palm team can fix in future updates or we're stuck with it?
    Cingular 3125 --> Samsung Instink! --> Palm Pre --> ???
  5. #5  
    I may be getting ahead of things, but does there seem to be an issue with the memory freeing back up after going too low?

    Maybe I'm imagining it, but it seems once that threshold is hit, and the browser window(s) go gray/refresh, it continues to happen more frequently until restarting the phone.
  6.    #6  
    The percentage memory used value seems to be calculated in an inconsistent way.

    The memnotify_get_used() function returns the sum of the used real memory and the used swap space.

    The memnotify_threshold() function then calculates a percentage using only the total real memory (and not the additional swap space) as the denominator.

    Code:
    unsigned long memnotify_get_used(void)
    {
    ...
            free_mem = memnotify_get_free();
            used_swap = total_swap_pages - nr_swap_pages;
            used_mem = totalram_pages - free_mem;
    
            return used_mem + used_swap;
    }
    
    int memnotify_threshold(void)
    {
    ...
            used = memnotify_get_used();
            used_ratio = used * 100 / totalram_pages;
    The comment on the threshold percentages says:

    Code:
    /**                                                                                                                     
    * @brief Thresholds are % used of (TOTAL_MEM + TOTAL_SWAP)                                                              
    */
    Based on this, I'm considering changing the UberKernel to say the following:

    Code:
        used_ratio = used * 100 / (totalram_pages + total_swap_pages);
    -- Rod
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  7. #7  
    Lately it mostly happens to me when I'm web browsing. Glad you're looking into this. It gets frustrating.
    "Patience, use the force, think." Obi-Wan


    Ready to try Preware? Get this first: Preware Homebrew Documentation
  8. avnera's Avatar
    Posts
    174 Posts
    Global Posts
    185 Global Posts
    #8  
    perhaps it should be unsigned long long and not unsigned long


    Avner.
  9.    #9  
    Quote Originally Posted by avnera View Post
    perhaps it should be unsigned long long and not unsigned long
    Why?

    -- Rod
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  10. avnera's Avatar
    Posts
    174 Posts
    Global Posts
    185 Global Posts
    #10  
    assume
    Code:
    used = memnotify_get_used();
    returns 300 MB

    then you multiple by 100

    Code:
    used_ratio = used * 100 / totalram_pages;
    and you overflow 32bit variable.
  11.    #11  
    Quote Originally Posted by avnera View Post
    assume
    Code:
    used = memnotify_get_used();
    returns 300 MB

    then you multiple by 100

    Code:
    used_ratio = used * 100 / totalram_pages;
    and you overflow 32bit variable.
    The units are number of 4K pages.

    -- Rod
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  12. #12  
    That would be awesome rod. I knew you would figure something out for uber-kernel. :-)
  13. #13  
    And why hasn't Palm hired you already?
  14. #14  
    Quote Originally Posted by donnydonny View Post
    And why hasn't Palm hired you already?
    prob because of big timezone diff betw Cali & Australia... :-)
  15. #15  
    Rod,

    I understand almost NOTHING in any of your posts....and you STILL manage to impress the hell out of me.


    Keep up the good work and THANK YOU!
  16.    #16  
    Quote Originally Posted by rwhitby View Post
    The percentage memory used value seems to be calculated in an inconsistent way.

    The memnotify_get_used() function returns the sum of the used real memory and the used swap space.

    The memnotify_threshold() function then calculates a percentage using only the total real memory (and not the additional swap space) as the denominator.

    Code:
    unsigned long memnotify_get_used(void)
    {
    ...
            free_mem = memnotify_get_free();
            used_swap = total_swap_pages - nr_swap_pages;
            used_mem = totalram_pages - free_mem;
    
            return used_mem + used_swap;
    }
    
    int memnotify_threshold(void)
    {
    ...
            used = memnotify_get_used();
            used_ratio = used * 100 / totalram_pages;
    The comment on the threshold percentages says:

    Code:
    /**                                                                                                                     
    * @brief Thresholds are % used of (TOTAL_MEM + TOTAL_SWAP)                                                              
    */
    Based on this, I'm considering changing the UberKernel to say the following:

    Code:
        used_ratio = used * 100 / (totalram_pages + total_swap_pages);
    -- Rod
    UberKernel 1.4.x-37 in the kernel testing feed now has this change, and the compcache kernel modules available for expert alpha testing.

    To use the compcache kernel modules, you need to do the following:

    Code:
            swapoff -a
            insmod xvmalloc.ko
            insmod ramzswap.ko backing_swap=/dev/mapper/store-swap
            swapon /dev/ramzswap0 -p 1
    However, I still get TMC errors when I open the third 3D game at the same time, so there is some other limit in play which I'm yet to find ...

    -- Rod
    WebOS Internals and Preware Founder and Developer
    You may wish to donate by Paypal to donations @ webos-internals.org if you find our work useful.
    All donations go back into development.
    www.webos-internals.org twitter.com/webosinternals facebook.com/webosinternals
  17. #17  
    will this be a feature in future Govnah where you can enable compcach without a command script?
  18.    #18  
    Quote Originally Posted by graffix31 View Post
    will this be a feature in future Govnah where you can enable compcach without a command script?
    That is the direction in which we are heading, yes.

    As always, new features progress from experimental kernels like F102 and F104 to Uber testing feed to Uber mainline with Govnah support.

    -- Rod
  19. #19  
    Keep up the great work!
  20. #20  
    Quote Originally Posted by rwhitby View Post
    However, I still get TMC errors when I open the third 3D game at the same time, so there is some other limit in play which I'm yet to find ...
    Could you be exhausting the JVM's memory? http://forums.precentral.net/web-os-...rds-error.html
    Quote Originally Posted by Brain_ReCall
    I'm an Embedded Software Engineer. My idea of a Good User Interface is printf().
Page 1 of 6 123456 LastLast

Posting Permissions