Did you know there is no way to backup third party data

[Bag of Leaves]

Originally Posted by Jason Robitaille

Wanted to pop in here and mention I should have a desktop solution ready within the coming weeks (no set ETA, but now that WOSQI 2.95 is out, I'll be a high priority)

Now that I have a Pre myself (and am unable to use Palm's backup) I find myself needing local backup too

Will this be something you don't have to hack/homebrew/Prebrew/devmode/etc. your Pre in order to use? Would it completely mirror the Pre so that even high scores and (encrypted) SplashID data and other things are backed up so they can be restored without issues if restoration needs to happen?

[Daemon]

There's nothing in the Mojo SDK that can directly access the database
files that he's backing up (and there never will be), so yes whatever solution
he comes up will always require some level of "hacking" to install and use,
be it QI, or Preware or whatever.

ian

[sorli]

Originally Posted by Jason Robitaille

Wanted to pop in here and mention I should have a desktop solution ready within the coming weeks (no set ETA, but now that WOSQI 2.95 is out, I'll be a high priority)

Now that I have a Pre myself (and am unable to use Palm's backup) I find myself needing local backup too

I'm pretty sure we don't have to reply, quote and give the thumbs up for something like this since I can imagine all WebOS users could use a tool like this, but THANK YOU is always needed and your work is appreciated.

I've been through 4 Pres (3rd for wife and 2nd for me) and have had to rebuild everything, update, and start over again. Would love to test something like this out before 1.3.5 arrives soon and have a better fallback before I continue Patching and otherwise. Moving from 1.2 to 1.3.1 was not very clean, though much of our problems related to Sprint/Verizon issues and had nothing to do with the phones themselves.

Keep up the great work and I'll look forward to your post.

And if you need help, testing or otherwise, please let me know?

Sorli...

[bill.millett]

Originally Posted by deafinoneear

Well, the way I see it is that Palm hasn't built in the archetecture for app developers to have an integrated backup option. One good thing about the sandboxing of iPhone apps (and correct me if I'm wrong) is that it's all self contained within the package. Is there even a standard the Palm has set up in WebOS for the storage of 3rd party app data?

Well, I am not sure about iPhone apps and whether or not they are backed up at all. Allowing for apps to be self contained (or using specific directories that can be backed up) may be something they provide, but even if they did that on the Pre there is no guarantee that the apps would utilize it. I mean, maybe an export function, etc. Could it be easier to build apps that are backed up? Is there a lack of capability in the SDK? Maybe. Somebody else that writes apps can comment. I just don't think that because I download an app that the Palm cloud would automatically back it up. Hell, what if it stored 3Gb of data?

Originally Posted by Jason Robitaille

Wanted to pop in here and mention I should have a desktop solution ready within the coming weeks (no set ETA, but now that WOSQI 2.95 is out, I'll be a high priority)

Now that I have a Pre myself (and am unable to use Palm's backup) I find myself needing local backup too

Once again, you are a rockstar. When it is out I'll donate (again) for sure. Hmmm - I suppose that with winter coming you will be stuck indoors avoid the cold and doing even more development. Good for us.

[hparsons]

Originally Posted by Jason Robitaille

Wanted to pop in here and mention I should have a desktop solution ready within the coming weeks (no set ETA, but now that WOSQI 2.95 is out, I'll be a high priority)

Now that I have a Pre myself (and am unable to use Palm's backup) I find myself needing local backup too

Jason, out of curiosity, how will the 1.3.5 change to where Palm saves install data affect all of this?

If it's simply an encrypted file, can it be duplicated to the same phone after a wipe? Can it be duplicated to a different phone?

[bbaratz]

Originally Posted by bbaratz

My original Pre from day 1 had to be replaced, no longer answered calls.
I had bought some apps and palm profile restored all the apps, but no data. I use pocketmirror to sync to Outlook 2003, so no problem, all my contacts, appointments and memos were all synced over perfectly.
Tealshopper, a good shopping app, I had spent a long time setting up stores, items, prices, etc. and I can't believe all that time was wasted, because palm profile does not backup third party data. In fact, with the OS, the palm rep told me there is no way to back it up at this time. Not in the clouds, not on your harddrive. Through USB, I had copied the old Pre to my computer, but that data is not copied over.
This is not acceptable. I hesitate to spend time adding data to these apps since there is no way to back them up.
What can be done to prevent this? Since Palm is so negligent to not include this basic computing premise of complete backup, we need some of the great programers on this site to come to the rescue..

I requested the developer of Tealshopper, a great shopping app, to give me his take on this problem of no third party data backup. Here is his reply:
Hi.


Thanks for the link. Here's a summary of what I get from the messages and our own experience with webOS that I hope you'll find useful. Feel free to share this with others:


BACKGROUND


Under webOS, almost all third party applications are enhanced javascript web apps that only have limited access to the device and no access to the file system itself. This architecture keeps the device stable, user data secure, and prevents apps from interfering with each other. This architecture is similar to the "sandboxing" on the iPhone, but is a little more restrictive due to the programming language used. The only exceptions are a few specific native apps like "Classic" that have been developed with special permission from Palm and native development tools not generally available.


FILE ACCESS


According to Palm rules, any apps submitted to the app catalog must work solely within the Javascript framework. This means that in most cases they cannot read or write files. In their own installation folder, they can read static files, but not create or modify them. Outside of their installation folder, they can do neither. As we understand it, apps can only ask the user to select a file from the USB drive (the special partition viewable when in USB drive mode) and then pass the name of the selected file to the built-in system media, document, and image viewers.


DATABASES


Since apps can't write to files, to locally store their data, they create and access databases through a webOS system API. The databases are give a name by the application, but there seems to be no good documentation in the SDK about whether the database name corresponds to a matching file name, or where that file (if it's an individual file) might be located. This obfuscation is assumably intentional by Palm, as they might even swap out the underlying technology in a future release of webOS.


Knowing this, it's easy to see how third party apps are limited in how they can manage and backup their data. WebOS runs linux under the hood, however, so it's possible to make a third party backup app compiled for linux to do a backup. Indeed, some of the developing tar-based solutions are probably exactly this. It's important to note that these apps break Palm's guidelines, however, so they can only currently be made available through Homebrew, and this functionality cannot be included in an app sold through the app catalog. It also doesn't answer the question of where the data is located. We third party developers don't know either, since the data managed by webOS.


It's probably Palm's intention that apps should instead store all their data "in the cloud" and to have each app sync its data with a server-based sync solution. That might be fine for calendars and apps with a web-based counterpart. But for other apps, is it really reasonable to expect each third party developer to create their own incompatible web-based backup system, and then force users to log in separately into a separate account for every third party app? All this to say, backup the state of your sudoku game? We think not.


Instead, when the system "Backup" app says "Your data is backed up automatically every day", IMHO it should be *all* your data, just as the text implies. Just like everybody else, we were surprised that this is not what it actually meant. Since webOS manages the creation and storage of third party databases, it seems reasonable that it should be responsible for backing them all up as well. If indeed all data belongs in the cloud, then Palm should put it there with one unified consistent interface.


Like other developers, we're now exploring interim solutions, but hopefully Palm will fix this problem soon.


Best regards,


--tex

Customer Support

[Dr.Grace]

Wow, a very clear answer! I don't think I've seen this explained so thoroughly anywhere else. I hope Palm re-examines this issue. Their sandboxing also makes it impossible, for example, for companies like Mark/Space to sync existing calendars or contacts it does not create, e.g. those originating from Google or the Palm Profile. Synergy makes things unnecessarily complicated for users, rather than simplifying them, IMO.

However, one thing to consider regarding the idea that Palm should backup all data is that the Palm Profile simply cannot be trusted to do this reliably. There's no way for users even to find out what is actually stored there!

If we acknowledge that the Palm backup is untrustworthy, then as things stand, every app developer has to buy into the cloud storage philosophy and provide for server space somewhere. As Tex wrote, this is also impractical. Thus, I'd rather see a solution for comprehensive backup to a computer.

[Audemars02]

While I absolutely hate the fact that there is no way to back up all my data, I want to go on record with one quick statement: There has NEVER been a native Palm-based solution to backup 3rd Party Data! I have had Treos since 2003 with the Treo 600, and I know that we had Palm Desktop to sync PIMS data (and I sorely miss that solution), but even the old Hotsync functionality never synced 3rd party data. However, there were 3rd party applications (e.g. BackupBuddy) that would perform a complete snapshot of the device and allow you to restore that to a new device, which I did many times!

So, for everyone complaining that Palm should have come out with a better native solution for 3rd party data/apps (I would have hoped they would have done this), I am not surprised they didn't. What does surprise me is that there is no way to make the complete backup that the old PalmOS apps let you do...

[Jason Robitaille]

Originally Posted by hparsons

Jason, out of curiosity, how will the 1.3.5 change to where Palm saves install data affect all of this?

If it's simply an encrypted file, can it be duplicated to the same phone after a wipe? Can it be duplicated to a different phone?

I'm sorry, but I'm not able to comment on that at this time. And am not able to say where or not there is a purported 1.3.5.

I'll post back once I'm able to

[ktessner]

Originally Posted by Audemars02

While I absolutely hate the fact that there is no way to back up all my data, I want to go on record with one quick statement: There has NEVER been a native Palm-based solution to backup 3rd Party Data!

Yeah there was, it was called HotSync. The majority of 3rd party apps that used the native PalmOS database APIs never had any trouble with their associated .pdb files being backed up via HotSync and restored by recovery syncs to either the same or replacement devices. The only issue was things like preferences (depending on which APIs apps used to save their preferences, even those were restored just fine by a recovery sync for a lot of apps) and license keys. Admittedly there were 3rd party solutions like Resco Backup or the free/brilliant NvBackup required to put you completely in the clear with all of your applications, but I owned Palms and Treos for years before I found out about these and for the most part upgraded painlessly without them.

In my opinion if the Mojo SDK forces catalog apps to use sandboxed APIs that store data in proprietary ways behind the scenes, then the Palm backup app should take care of transparently saving and restoring that data as part of its overall job function. This is what they've bred legacy Palm users and developers to expect...

[Spader]

Originally Posted by Jason Robitaille

I'm sorry, but I'm not able to comment on that at this time. And am not able to say where or not there is a purported 1.3.5.

I'll post back once I'm able to

Ah... you have to love a good NDA.

[rc46]

PIM data is one matter but 3rd party application data is a whole different story. There is an API for backing up and restoring 3rd party application data using Google Documents and it is very slick indeed. Unfortunately most developers are too lazy to use it or they dont know about it.

Just take a look at some of the well written apps like Checkbook and Fuel. You will see how easy it is to import and export data using Google Documents. The nice thing is that these backup files do not get over written so you can have as many different versions as you want. Its a shame more developers dont use this.

[Audemars02]

Originally Posted by ktessner

Yeah there was, it was called HotSync. The majority of 3rd party apps that used the native PalmOS database APIs never had any trouble with their associated .pdb files being backed up via HotSync and restored by recovery syncs to either the same or replacement devices. The only issue was things like preferences (depending on which APIs apps used to save their preferences, even those were restored just fine by a recovery sync for a lot of apps) and license keys. Admittedly there were 3rd party solutions like Resco Backup or the free/brilliant NvBackup required to put you completely in the clear with all of your applications, but I owned Palms and Treos for years before I found out about these and for the most part upgraded painlessly without them.

In my opinion if the Mojo SDK forces catalog apps to use sandboxed APIs that store data in proprietary ways behind the scenes, then the Palm backup app should take care of transparently saving and restoring that data as part of its overall job function. This is what they've bred legacy Palm users and developers to expect...

Really? my experience with hotsync was that it did NOT keep most of my 3rd party data.. I relied on BackupBuddy to do my full backups... either way, thats all in the past. for 2009 (and soon 2010), we need a solution to this problem!!!

[kuoirad]

Originally Posted by Daemon

I then just created a little shell script called fullbackup.sh with the following lines

cd /
gnutar -cvf /media/internal/bigbackup.tar / --exclude /media --exclude /sys --exclude /proc --exclude dev

When I want to do a full backup, I start terminal, and run
sh fullbackup.sh
It backs up the entire Pre from the root directory excluding the /media partition
which is already visible via USB drive and some real-time sensitive system directories.
This backup will grab the palm databases which have all your contacts/messages/etc,
email, plus all the front (app catalog) and side loaded (homebrew, etc) apps and all
their own databases. It creates the backup tar file in root of /media/internal where
you can see it from the USB drive and copy to your PC.
A look through the contents of the tar and it starts to become clear
where everything is stored on the Pre.
/var/usr/palm/applications has most of the side loaded apps themselves
/var/palm/data has some of the databases for external apps

Once you know where stuff is stored there are ways to restore it.

Given that cron exists on the system, have you tried setting up a cronjob to have this fire off every night, or on some other schedule, out of curiousity?

And if one ended up with a new phone, for some odd reason I can't see a tar xvf -C / working happily. Do you have an idea of how to do a restore, then?

I've also had thoughts along the lines of using rsync, but that'd be a slightly more complex script...