IPK Fetcher

[Blacklight]

IPK Fetcher - Circumvent geo-filtering
For Windows, Mac OS X, Linux

*** UPDATE 6/8/2011: R.I.P. 'IPK Fetcher'! Today HP informed a few people about the impending shutdown of the official App Catalog feeds. This application uses the WebOS Internals Preware Feeds, and those get their data from the official feeds. The Preware feeds are offline already, which means that this application won't work anymore from now on.

I wanted to offer a convenient and easy method to circumvent the geo filters. Without the feeds, I can't. However, HP did say that in the near future geo filtering won't be an issue anymore for new devices. We will see how that turns out.. I guess legacy devices are out of luck.

You can probably still use the new DummyMaker from michael.schumacker: Link. However, you have to know the app-id of the app you want to install.

Thanks to all users! ***

Download:

Removed! See update above..

Requirements:

- A Java Runtime Environment (Java 6 is required, you should be good if you have the SDK or Quick Install installed). Download here: Java Downloads for All Operating Systems
- For installing the IPKs use "Quick Install": Link

(If you want to install the generated apps from within the application, you will need the WebOS SDK or just Novacom. If you have WebOS Quick Install on your computer, Novacom is probably already installed.)

Doubleclick ipkfetcher.jar to start the application



Instructions:

Once you have started 'IPK Fetcher' (double click ipkfetcher.jar), the application will update the app data from the Preware Palm-Catalog and Palm-Beta feed. The feeds are saved as files within the root directory of 'IPK Fetcher' and get updated on restart if the data is older than 6 hours (you need write access to the application root directory). You can manually reload by hitting "Reload" in the lower right corner.

The beta-feed is now included. You can switch between the two feeds by using the combobox above the table.

You should then filter the apps submitted for your country by excluding them from the list. That way you won't have to deal with apps you can already see in the regular App Catalog on your device.

There is a live-search.. just type and see the results.

You can sort most table-columns (ascending/descending) by clicking on the table headers once or twice.

If you hold your mouse on a table-entry, you will see the countries the app was submitted for as a tooltip. If you select an app, you can hold your mouse on the app-icon in the upper-right corner. After a while you can see three screenshots of the selected app.

For generating apps, just hit the button after selecting the desired application. A "dummy" IPK-file will be created in the folder "IPKFGenApps" within the folder 'IPK Fetcher' is currently running in. If you wish to directly install the selected app, hit the checkbox right next to the button BEFORE you hit the button (if you forgot, just click it again with the checkbox selected). Your WebOS device has to be connected to your computer, in developer-mode (type "webos20090606" in UniversalSearch to enable it) and "just charging", or you will get an error.

The terminal window will inform you about steps or errors concerning the IPK creation/installation.

The status line on the bottom informs you about the current steps or if anything went wrong.

IMPORTANT:
When you are done, it's important that you open up your App Catalog on the device. Go to your installed apps (button in the lower right corner), wait for the update-button to appear, and UPDATE ALL APPS! This is the step where you download the actual application directly from Palm. If you skip this step, you will have nothing on your Pre except freshly generated apps.
On webOS 2.x, updating works differently. You need to open up the Software-Manager, and update your apps there.

It's posssible that you won't see the correct icon or title for an app in the launcher right after the update. The app should start up though. After a Luna restart the launcher icons and titles should be fixed.

You can clear the directory where the IPK-files are stored from within the application. Just hit the options in the menu.

THIS TOOL ONLY GENERATES "DUMMY-APPS". THAT MEANS THE IPK-FILES YOU WILL SEE ON YOUR COMPUTER AFTER YOU HIT THE GENERATE-BUTTON ARE NOT THE "REAL DEAL", THEY ARE JUST BASIC WEBOS APPS WITHOUT ANY FUNCTION. DON'T USE THIS TOOL AS A BACKUP-TOOL BECAUSE YOU ARE NOT BACKING ANYTHING UP.

Tested on Mac OS X 10.6.3, Windows XP Professional, Windows 7 Professional

[palmdoc2005]

thank you thank you thank you !

[rwhitby]

OK, time for me to disclose some stuff ...

This tool uses the palm feeds from ipkg.preware.org, with our permission (you don't actually need to ask permission, but Blacklight did the right thing and did ask permission beforehand).

These feeds include information which is gathered purely from the published Palm App Catalog XML feeds. These XML feeds are designed to be read, processed, and turned into other ways to access the app catalog. There are no secrets here - everything is public knowledge, published by Palm.

There is enough information in the Palm XML feeds to identify the URL by which free app catalog applications may be downloaded. This technique has been known by many different group for a couple of weeks, and is reported widely on this and other forums. Palm has the ability to restrict access to those URLs (as the URLs for paid apps are restricted from being accessed using this technique), but has obviously chosen not to do that so far for free apps.

Note that ipkg.preware.org does not proxy or host or redistribute any actual packages from the palm app catalog, specifically because there are many applications in the palm app catalog which do not allow redistribution elsewhere. You cannot download an app catalog application from ipkg.preware.org, however you can get a processed version of the public Palm XML feed information which contains enough information (all of that information being publicly published by Palm) to access Palm's servers and achieve that download. Note that this is similar to the situation for homebrew applications from PreCentral, where the feed on ipkg.preware.org simply redirects the access to the PreCentral servers, to ensure that redistribution rights are not violated and to ensure that PreCentral's download counts are accurate for applications downloaded through Preware or WebOS Quick Install.

So that's the good news for those outside of the US who are normally unable to access these free applications from the Palm App Catalog.

Now for the ethical dilemma ... and it involves the Hot Apps competition.

If someone outside of the US uses a tool such as this to download and install packages, then my personal opinion is that there is no problem. Authors who choose to limit the distribution of their application outside of the US have already made a decision to forgo any downloads from those countries that may otherwise have counted towards their Hot Apps ranking.

However, if this tool is used within the US, then it will cause the download counts for authors of free applications in the offficial app catalog to be reduced, thereby affecting their chances of winning the Hot Apps competition.

My opinion is that tools such as this should look at the country in which they are being run, and only download applications which are not normally available in the app catalog in that country. If this capability were (hypothetically) to be added to Preware, it would check the Mojo.Locale settings against the Countries JSON array in the preware.org feeds, and if the device is set to a country for which the application is available in the official app catalog, it would simply pass the appid over to the app catalog client for installation (thereby keeping the official download count for that application correct).

Whether or not we add such functionality to Preware is still under discussion. There is a small technical hurdle (the default wget binary in webOS does not handle https URLs), but nothing insurmountable if we do decide to go forward.

Note that paid apps are not available via this mechanism (or any other known mechanism involving information from the public Palm XML feeds), however free apps available in Europe are technically able to be downloaded by US users via this mechanism. The authors of this specific tool have special-cased those applications, and prevented their download using this tool. However, the general technique remains, and it will be interesting to see whether Palm or EA games has any reaction to this situation.

On a final note, WebOS Internals takes copyright and redistribution rights very seriously, and goes to great lengths (for example, see the WebOS Internals Meta-Doctor) to ensure such laws and licenses are not violated. The feeds on ipkg.preware.org are generated from official published information from Palm which is explicitly designed and offered for processing and republication, and ipkg.preware.org does not proxy, host or redistribute any applications from the Palm app catalog directly.

-- Rod

[Blacklight]

Thank you for your post rwhitby!

My opinion is that tools such as this should look at the country in which they are being run, and only download applications which are not normally available in the app catalog in that country.

II couldn't agree more. It's not my intention to harm anyone, after all I like my Pre and the company that made it possible, which is why certain apps are excluded. Also, this tool is not for US users, you guys have a full App Catalog. This is why I posted this in the GSM Section.
But let me stress out again that this tool is simply a fancier version of using techniques that are already well known. If those techniques don't work anymore, this tool won't be able to download the IPKs anymore.

You can also see it as a kind of an outcry.. it's been very frustrating to see all those new free apps and not see them in the Catalog outside of the US, and this questionable, strict geo-filtering. And to be very honest, if it wasn't for my Palm-sympathy, the HUGE potential of WebOS, my wish to see Palm doing well and going stronger, and (not to forget) Preware I might have already switched to Android. Thanks for your attention

[cryptooth]

Who can merge WOSQI and IPKFetcher?

[Blacklight]

Originally Posted by tomi666

Hi Rod, thanks for the explanation. What do you mean with the part quoted above. That EA or Palm may object, even when certain IPK files are made unavailable within the tool itself.

I specifically made those three apps unavailable within my tool, but that doesn't change anything about the fact that the underlying technique is still working.

[rwhitby]

Originally Posted by cryptooth

Who can merge WOSQI and IPKFetcher?

If Jason wishes to face the ethical dilemma, the capabilities already exist in WebOS Quick Install to achieve this with a couple of lines of changed code.

Similarly for Preware (we could simply install a better version of wget, just like we install GNU Patch).

-- Rod

[Griffinx]

Originally Posted by rwhitby

Now for the ethical dilemma ... and it involves the Hot Apps competition.


-- Rod

I suspect Palm will resolve this 'ethical' dilemma.

They cant afford to leave themselves open to accusations of unfair play with regards to the Hot Apps competition. The publicity would be quite damaging.

Gx

[shanerooni]

Originally Posted by Griffinx

I suspect Palm will resolve this 'ethical' dilemma.

They cant afford to leave themselves open to accusations of unfair play with regards to the Hot Apps competition. The publicity would be quite damaging.

Gx

not to mention the drama it will (is, actually) causing within the developer community -- the whole reason the Hot Apps competition exists. palm can't afford to lose their increasing app stream, and i know some developers are on the fence to begin with, and this may be the tipping point.

[DJeremyC]

Did I miss something about this program to work...I start it and the windows is blank with Count: 0

Gives me this message: Error reading InputStream! Please check your internet connection, or the servers might just be overloaded.

???

[Blacklight]

Originally Posted by DJeremyC

Did I miss something about this program to work...I start it and the windows is blank with Count: 0

Gives me this message: Error reading InputStream! Please check your internet connection, or the servers might just be overloaded.

???

Check your connection or firewall, everything is fine.

[Brawler]

This is awesome finally getting to play free apps on my bell palm pre

[rockshox1314]

Can you please include some instructions to use this. I downloaded the ipk executable jar file. I doubleclick and nothing happens?

[Blacklight]

Originally Posted by rockshox1314

Can you please include some instructions to use this. I downloaded the ipk executable jar file. I doubleclick and nothing happens?

Just doubleclick ipkfetcher.jar and the application should start up. If not check if you have the latest Java Runtime Environment. If that doesn't help I can't think of anything else right now.

[petehild]

Originally Posted by rockshox1314

Can you please include some instructions to use this. I downloaded the ipk executable jar file. I doubleclick and nothing happens?

did u check that you java is installed properly?

[aolbites]

Does anyone know why the free EA games aren't coming through? I checked and rechecked and they were in the Catalog a day or two ago, and now they're not there, but they're also not available through this. And I was really looking forward to downloading the free NFS and Monopoly!

[Blacklight]

Originally Posted by aolbites

Does anyone know why the free EA games aren't coming through? I checked and rechecked and they were in the Catalog a day or two ago, and now they're not there, but they're also not available through this. And I was really looking forward to downloading the free NFS and Monopoly!

Where do you live? They are still available in the European App Catalog. In the US, you can buy those apps. And you should really read the notes in my first post

[petehild]

Originally Posted by aolbites

Does anyone know why the free EA games aren't coming through? I checked and rechecked and they were in the Catalog a day or two ago, and now they're not there, but they're also not available through this. And I was really looking forward to downloading the free NFS and Monopoly!

The EA Games are disabled in IPK Fetcher on purpose. I just checked, they are still avail in the EU App Catalog.

[petehild]

Originally Posted by Blacklight

Where do you live? They are still available in the European App Catalog. In the US, you can buy those apps. And you should really read the notes in my first post

Warst schneller

[nathan_pre]

I manually filtered three EA games currently offered in European App Catalogs. So don't look for them.. the reason for this is that this is a limited March-offer from Palm (as a gift, promotion, whatever) because there are still no paid apps available in Europe. I appreciate that offer very much, and since you can buy those apps in the US I excluded them.

I am in Canada using a Bell Mobility Palm Pre. We do not have paid apps and we did not get access to the free EA games apps. Is there a way us Canadians can get it?

Canadian Pre owners really seem to be forgotten by Palm and Bell. We dont even have 1.4 yet!