Page 1 of 2 12 LastLast
Results 1 to 20 of 30
Like Tree25Likes
  1.    #1  
    Download current version here: webOS App Museum II

    Version History
    • Current Stable: version 2.7.0 - Layout changes, restore star ratings.
    • Previous Beta: version 2.6.7 - A minor update to correct some icons, especially in the new categories.
    • Previous Beta: version 2.6.6 - Support back-end change for two new categories, and alternate IPK sources... to be populated soon!
    • Previous Beta: version 2.6.5 - Loads "extant" catalog only, from unified web service source, with working images and IPKs. Depends on Preware to do the actual install.
    • Previous Beta: version 2.6.4 - Loads full masterAppData, missing app filter behind feature switch, uses single source for images and IPKs without configuraiton. Depends on Preware.

    This set of problems has always seemed accomplishable individually, but combined they appeared overwhelming. I decided to take it one step at a time.

    - The AppMuseum is a good listing of the apps that used to be in the webOS AppStore, but doesn't actually have the apps themselves, and many images are offline
    - The webosftp.qualitis.hu FTP archive has thousands of apps, but split across multiple folders, and completely lacks meta data
    - Internet Archive's Wayback Machine did index the Palm CDN Public directory, and has a few thousand images in the archive

    The idea is to scrape these resources into one location, and refurbish the App Museum in some form, so images and download links work where ever possible. So far I've:
    - Pulled down the catalog JSON from the URL the App Museum is using: banneret.nl/_webos
    - Downloaded the Qualitis FTP archive to my local hard drive
    - Reverse engineered the Wayback Machine API so I can theoretically scrape out all the images (actual code still needs to be written)
    - Written an indexer that walks the catalog JSON and checks it against the folders on my hard drive to see what apps are present or missing
    - Modified the indexer to also search for versions of an app later than what the App Museum knew about

    The results are in this Excel file, which shows that 6123 of the App Museum apps are missing, 2848 exist as archived, and 714 exist as archived and have updates.

    AppIndexer.PNG

    Another source of images is the app package itself, which is some kind of archive (7Zip calls it a ".ar" archive). That in turn contains gzipped tarballs, one of which is that actual app deployment, which has the appinfo.json which points to the App's icon. It should be possible to iterate through the catalog, temporarily explode each package, read its appinfo, find the icon, and extract it with an appropriately computed filename.

    If anyone knows of (or has) any other file archives, please let me know so I can index those files as well -- there's a lot missing.

    For now, the goal is just to get a working App Museum again. Once accomplished, it appears it would be pretty easy to use these learnings to create a "new" AppStore.
    Last edited by jonwise80; 10/27/2020 at 07:54 PM.
  2.    #2  
    The metadata and scraping/indexing toolchain is here:
    https://github.com/codepoet80/webos-catalog-utils

    I screwed up moving files around, so I'm pulling the archive again overnight to make sure I didn't lose anything.
    Image scraping up next, when I have the free time.
    Last edited by jonwise80; 10/24/2020 at 05:50 PM.
  3. #3  
    Quote Originally Posted by jonwise80 View Post
    The results are in this Excel file, which shows that 6123 of the App Museum apps are missing, 2848 exist as archived, and 714 exist as archived and have updates.
    I was under the impression that 'most' of the apps had been archived one way or another.
    JVLebbink1 indexed the FTP for the app museum. The museum was built by Misj'.
    The next user group meeting is on the 18th.
    Last edited by Preemptive; 10/10/2020 at 12:16 AM.
  4.    #4  
    I'll re-run my scanner utility when my fresh pull is done, but it does appear there's a *lot* missing. The archive has many duplicates, or apps with multiple versions, that might be inflating the numbers -- which my scanner identifies.

    IconScraper.PNG

    I added a utility to scrape the Palm CDN archive, which I'll modify to fall back to the Wayback Machine if needed. I'm making progress to having everything that still exists in a single archive -- I'm thinking of publishing as a Raspberry Pi image, that would allow anyone to host an AppStore themselves

    I managed to sniff the web service calls from the 'Museum, but if Misj or JVLebbeink1 are on here and want to share the source for the PHP, that would be really helpful! The banneret.nl website is functioning, but I can't find any contact info for the owner, so its looking like I'll have to reverse engineer the API.
    Last edited by jonwise80; 10/10/2020 at 05:51 PM.
  5. #5  
    There really shouldn't be a whole lot missing. I scraped whatever I could from the original app catalog before it went down, including paid apps and geo restricted apps from people who "donated" their Palm profile. This should all be on the FTP. I also have a local copy I can give you access to if needed.

    Sent from my Titan using Tapatalk
    HP Veer (daily driver), HP Pre 3, HP Touchpad Proper 4G/LTE (Sierra MC7710), HP Touchpad 32GB WiFi, Palm Pre 2
  6.    #6  
    Quote Originally Posted by Herrie View Post
    There really shouldn't be a whole lot missing. I scraped whatever I could from the original app catalog before it went down, including paid apps and geo restricted apps from people who "donated" their Palm profile. This should all be on the FTP. I also have a local copy I can give you access to if needed.
    IF we're talking about the same FTP site at qualitis.hu, here are my findings after pulling the root folder contents and the "Catalog Pull" folder...

    4499 - apps pulled from FTP with catalog number prefix on the filename
    4497 - apps after stripping catalog number and de-duping
    1749 - apps pulled from FTP without catalog number prefix, 314 overlapped with catalog
    ----
    5932 - combined, de-duped apps from FTP to be compared against catalog...
    ----
    3556 - app matches from app catalog master file
    678 - alternate versions of app catalog apps
    1698 - apps from archive that were not in the app catalog
    ----
    5932 - apps from FTP indexed by tool against catalog
    ----
    6128 - apps in the catalog that do not have a corresponding ipk

    Automatically generated report attached.
    Attached Files Attached Files
  7.    #7  
    The end result of the tool is two new "masterAppData" json files -- one for missing files, and a really good one for extant files. This one is really good because I have icons and IPKs for them -- I'll work on the screenshots next. This "really good" set can be a fully working App Catalog... once I figure out the PHP side of things!
  8. #8  
    Quote Originally Posted by jonwise80 View Post
    The end result of the tool is two new "masterAppData" json files -- one for missing files, and a really good one for extant files. This one is really good because I have icons and IPKs for them -- I'll work on the screenshots next. This "really good" set can be a fully working App Catalog... once I figure out the PHP side of things!
    I have all these, already provided them to Bannernet guys in the past. Can share with you if needed. Got symetrical 1gbit fiber here, so should be quick enough for you to download. Just PM me if interested

    Sent from my Titan using Tapatalk
    HP Veer (daily driver), HP Pre 3, HP Touchpad Proper 4G/LTE (Sierra MC7710), HP Touchpad 32GB WiFi, Palm Pre 2
    jonwise80 and gazaud like this.
  9.    #9  
    No change in count from the archive I got from Herrie, since it was just metadata (although it did prove to be a useful image repository!)

    I did find another user's Box.com archive, which had 522 Pre apps (with 172 version variations) and 143 TouchPad apps (with 30 version variations), however it netted only a minor change in the total count of extant apps, since it largely duplicated the FTP archive.

    Extant app catalog count: 3563
    Missing app catalog count: 6121

    That said, it increased the number of "Unknown" apps -- apps for which there is no metadata. Previously I had 1458 uncatalogued apps, that count went up by 439 to 1897 apps without metadata (although I need to rule out possible PreWare dupes before I accept this count!) I did notice that Herrie had metadata files that were different in structure to the App Museum, but on a spot check, not appreciably different in content.

    So we're still legitimately missing thousands of apps, and the ones we have are (to date) scattered across multiple archives and folders. I can solve the latter problem -- but doing so is almost useless unless I can also tweak the App Museum to know which ones are downloadable and which are not. The combined/de-duped archive, including metadata where found, alternate versions, and apps without metadata is sitting at 28.9GB.

    If anyone knows how to contact Banneret, please let me know!
    Last edited by jonwise80; 10/11/2020 at 07:53 PM.
  10. #10  
    Quote Originally Posted by jonwise80 View Post
    This set of problems has always seemed accomplishable individually, but combined they appeared overwhelming. I decided to take it one step at a time.
    - The AppMuseum is a good listing of the apps that used to be in the webOS AppStore, but doesn't actually have the apps themselves, and all the images are offline
    - The webosftp.qualitis.hu FTP archive has thousands of apps, but split across multiple folders, and completely lacks meta data
    I've checked on my touchpad, and the images still work. Did you edit the host-file as mentioned in the first post: https://forums.webosnation.com/webos...pp-museum.html
    It is also possible to attach an FTP or HTML location of your choice to the app museum in the preferences.

    If anyone knows how to contact Banneret, please let me know!
    Hi. :-)


    ps. I had some further ideas to implement (as well as some ideas by others). But I never got around to them when life got in the way.
    jonwise80 likes this.
  11.    #11  
    Yay, the gang's all here!

    Yes, I've changed the hosts file to point to the CDN -- icons work, but most screen shots fail to load. And yes, I've pointed the App Museum to the FTP archive, BUT its painful because:
    1) the apps in the archive are spread across two main folders, which largely duplicate each other, except one folder has a catalog prefix and the other doesn't, so you're constantly manually tweaking the download path
    2) problem #1 assumes you've already determined that the App exists in the archive you're pointing at -- itself a manual task

    You pretty much can't use it without also having a computer beside you to find out if/how the Museum can get the app you're interested in.

    I understand and respect your decision not to include an IPK archive, to avoid copyright issues (and probably bandwidth costs!) but I think in 2020, its fairly safe to pull everything that's left together in one place* -- I'm certainly willing to go for it. I can see from dumping the web service directory that you'd also started work on a "missing" list, and from the API call, that you'd at least planned a way to exclude missing apps from showing up in the Museum. Herrie is going to pull together a list of what he has, hopefully we can end up somewhere near half the apps -- but that would still leave 50% of the museum as "browse-only"

    I propose to either:
    Option A) Add features to the App Museum to point to a server that has everything in one place (which I will provide) -- including metadata, images, and ipks for a filtered-on-existing-ipks masterAppData (with possibly a switch to show missing apps for purely historical purposes)
    Option B) Fork the App Museum into "App Catalog II" or something like that, where the download features are built-in, leaving the App Museum as-is with its copyright considerations intact. We could use this new fork as the basis for a new App Store.

    Either way, the goal is to get to the point where I can hand my kid a TouchPad and say "go here to find new games to play" without worrying about FTP sites. I'm actually really close on Option A -- but if I could have access to the .php source code, that would help a lot! I was about to start a NodeJS re-implementation, but if you're willing to share, I'll skip that... and would be happy to work on those other feature ideas you mentioned!

    * Recent evolution of copyright practice in the US is generally fairly protective of preserving digital artifacts for historical purposes.
  12.    #12  
    Update on Pulling Everything Together:

    - Icon scraping from the Palm CDN backup resulted in only 7 missing icons for the Catalog! Given the small set missing, I'll do some manual searching, rather than build a tool to try to find those.
    - Screenshot and Thumbnail scraping from Herrie's archive + the Palm CDN backup resulted in 10845 images missing -- but that's still 58793 that do exist! I'll automate a Wayback Machine search to try to fill in those gaps, but I attached a screenshot of what the tool looks like now when its running -- takes over an hour on a quad core i7, but its super cool how automatic it is.
    - Indexing Herrie's IPK collection identified that he has 4310 of the 6121 IPKs missing from the FTP archive. Once he gets those to me, we'll have 7873 out of 9684, or 81.3%, of the catalog apps in one place!

    Reverse engineering the API is going slowly -- I was able to spoof the initial catalog pull, but the App Museum hangs while pulling Item details -- probably due to a follow-up catalog pull failing in the background. Still hoping to get access to the PHP source code...

    ImageScraper.PNG
  13. #13  
    Hi,

    I talked to Misj' and we agree on handing over the source code of the App Museum. We knew there were some limitations concerning downloading and installing the applications, but somehow it didn't go far.

    Misj' has all the source code of the WebOS-application and of the Server side. So I'll leave this one to him.

    Regards,

    J.
    Running a Pre 3 (daily driver), Pre (advanced alarm clock), Touchpad 32GB with Android (hate it), Touchpad 64GB Panda with WebOS, and a Veer (for my wife).
    jonwise80 likes this.
  14.    #14  
    I updated my Image Scraping utility to fallback to the Wayback Machine. Its kind of brute force and exponentially increases run time, but it worked. I now have 69513 images, with only 35 screenshots and 2 icons missing for the extant App Catalog IPKs!

    I've got the server space, and a beta IPK, sketched out at http://appcatalog.webosarchive.com
    Misj' drop me a line about that PHP
    Last edited by jonwise80; 10/16/2020 at 08:43 AM.
  15.    #15  
    Looks like I'll be able to grow the extant app list by about 272 apps, by building a search on similar package names. The FTP archive had 1897 apps that were not exact matches to the catalog master app list, but by making the match logic a little fuzzier, I was able to find a small pool of them that were close to the master app list -- but maybe off by a major or minor version or two.

    I'm not exactly sure what to do with these at the moment. I can forcibly rename them to make them match the master app list -- but I don't want to lose track of the fact that they weren't exact matches, in case an exact match does come along at a later date. It would be better to modify the master app list to handle multiple versions, but I need the back-end code for that.

    I will try to make the user's group meeting tomorrow, but Sundays are tough with a family of 5 -- especially when we're down to one car for the time being!
  16. #16  
    I'll try to gather everything tomorrow. But I'm a bit busy, so no guarantees.
    jonwise80 likes this.
  17. #17  
    I'll send you a download link via PM in a minute.
    EDIT: scratch that...the PM isn't working for me. Just email me at webos <monkeytail> banneret.nl (and yes, in Dutch the character @ is called a monkeytail...which is a great name) and I'll send you the download information.

    and from the API call, that you'd at least planned a way to exclude missing apps from showing up in the Museum
    Also, anyone who is interested in some of the features I was working on, that never got into the initial release (it's probably not that interesting to most users). Type Palm webOS 2018 in the searchbar. You will find there are some additional options in the preferences now.
    Last edited by Misj'; 10/18/2020 at 10:58 AM.
    jonwise80 likes this.
  18.    #18  
    A Beta version of what is tentatively called "webOS App Museum II" is available at:
    webOS Archive

    To make it clear that this is a different beast than the original App Museum, it has a different skin, icon and package name. The only other changes are on the back-end:

    • There is no longer any need to point to an FTP archive
    • You don't need to modify your hosts file for the Palm CDN
    • Only apps that can be downloaded are listed by the service


    I'm taking suggestions for the name of the app...
    gazaud likes this.
  19. #19  
    I just want to give a big Thank You to everyone that is and has been apart of this. I just downloaded and installed it with Quick Install and it runs and looks great! However, If I go to an app the far right is all gray and don't see a download option. What am I doing wrong?

    Also, I am not too too tech savvy either, but I did download all the ipk files for every app listed a while back from old source through the mega download thing, so if any of those are needed, please let me know. If I can be of any help, I have an Original Pre and a Touchpad.
    Last edited by Eatsleepdrivesc; 10/18/2020 at 10:55 PM.
    jonwise80 likes this.
  20.    #20  
    Quote Originally Posted by Eatsleepdrivesc View Post
    I just downloaded and installed it with Quick Install and it runs and looks great! However, If I go to an app the far right is all gray and don't see a download option. What am I doing wrong?
    You're not doing anything wrong. I had a bug in how I was loading the details, but I was too tired to fix it last night. It was the first thing I did when I got to my computer this morning, and I just tested it. Everything works -- but be aware you'll need Preware installed first to do the actual install. Let me know how it goes, and if you find any new gems!

    Also, this is in "active development" so it may break again -- there'll be new versions of the app that you'll need to download periodically, but if you're using the latest and it doesn't "just work" -- let me know. The goal is that anyone can use it... I just didn't realise there'd be users so fast! I'll update the website to show what version is current, and add some other first-timer info.
    Last edited by jonwise80; 10/19/2020 at 08:49 AM.
Page 1 of 2 12 LastLast

Similar Threads

  1. Once More Unto The Breach
    By Aressel in forum FeedSpider
    Replies: 1
    Last Post: 05/29/2020, 07:16 PM
  2. Replies: 4
    Last Post: 05/22/2020, 09:36 AM
  3. Trying to open my app from the television browser
    By MartianFromMars in forum LG webOS TV
    Replies: 0
    Last Post: 05/09/2020, 08:22 AM
  4. Whats Shaking and TechTray - back as Web Apps!
    By cyberprashant in forum webOS Apps & Games
    Replies: 2
    Last Post: 03/16/2020, 10:36 AM
  5. Cannot boot past the USB icon
    By pepebuho in forum HP TouchPad
    Replies: 2
    Last Post: 02/26/2020, 11:04 PM

Posting Permissions