Thinking about... Locally hosted backup servers

[GMMan]

I'm digging through the code for cloud backup. Because the code is all JavaScript, I'm thinking it could be easy to reverse engineer it and create a local backup server from that. So for Open webOS, if the cloud backup is not provided by HP, users could host their own backup server on their network, copy over the backup framework, and instead back up to their own network.

From what I've read so far, the backup system generates a manifest with some statistics, fires off an event for different services to generate backup files, group and list those files in the manifest, then sends the manifest and files off to the backup servers. There are three servers used in this process: an authentication server, a metadata server, and blob storage server. There's an additional deviceImages server that I haven't figured out yet.

There are a lot of stats sent over with the manifests: battery percentage, signal strength, OS version, free disk space, and CPU/RAM load (I wonder what that's for). Backups are typically done incrementally, and each service comes with a description of what part of the system is backed up. Because it's modularized on services, maybe in the future people could select what parts they want to restore. And because of this modularization, new services could potentially be added. If that is indeed possible, maybe the WebOS Internals Backup/Restore app could be tied in and make it easier to restore flat file app settings (like used in games) without shuffling things on the USB drive.

[GMMan]

File access looks easy. You request a storage auth token with your standard authentication details, and then you can access your files that is hosted under a subdomain within the main storage cluster's subdomain, with your storage auth token. Then it becomes sort of like an FTP, where an HTTP GET will either return a list of files or the file requested, and an HTTP PUT will upload the file to the storage. All the while there is a metadata server to keep track of the session. Checking how files are populated now.

Update: There are two ways to install backup services: a) put a manifest under /etc/palm/backup, with the file name being the service ID (not enforced), and a schema that looks like this:

{
    "id": "com.service.id",
    "preBackup": "preBackupMethodName",
    "postBackup": "postBackupMethodName",
    "preRestore": "preRestoreMethodName",
    "postRestore": "postRestoreMethodName"
}

[xandros9]

woah, thats so cool!

i just got myself an itanium

[GMMan]

Here's how I think local backups could be implemented:

- A single server hosts all three components.
- A patch overrides the backup server paths, configurable through Tweaks (I'm surprised that the Palm Profile service is actually implemented in JavaScript).
- Authentication and metadata is wrapped in a database, where there will be a table of devices (persistent), table of authentication tokens (temporary), and a table for metadata.
- An online interface is used to control all three parts: you can add/delete devices from one interface, and be able to see the backup history (metadata) for each device. The metadata forms a log, and for each backup instance the manifest could be viewed.
- Auth tokens will be generated in the same general format as the way it's done currently. However, because of the way the backup system is programmed, there might need to be a DNS server installed locally so the storage server can be reached (it's like uniqueid.storageservername, instead of 168.45.6.7/backups/storage/uniqueid)
- There could be a manifest editor that can allow choosing which files to restore when a restore is necessary.
- When a restore is needed, connect the device to a computer to install the local backup patch, and then proceed through logging in to Palm Profile and restoring data.
- There can be a way to select which apps are installed on to the device instead of dumping the last known list of apps installed back (which always seems to be a month late).

About implementing Save/Restore backup on current system:
- Save/Restore would back up and zip together a file to be uploaded when a system backup is requested.
- When the system needs to be restored, the file will be downloaded back for Save/Restore to use.
- Immediately after erasing data, Save/Restore will not be installed, so it would be a good idea if this was saved to somewhere external to the Save/Restore app directory.
- Since Save/Restore wouldn't be installed, the backup system can't notify the app it needs to restore. So check for backup file the first time Save/Restore starts and restore data if backup file is found.
- This way even if the USB Drive partition is erased all app data can still be restored.
- There is no need for Save/Restore to do any additional processing to upload the backup. All it needs to do is backup, zip backup, and return the path of the archive to the backup service.

[rolladude1166]

This seems like a good idea. It may just be that I'm new to this whole javascript thing, but it seems kinda tough to do?

[gizmo21]

oh my own iCloud aahh wCloud for device perferences and third party apps backup and sync. That would be really cool and one major point missing in Open webOS:
- What's missing from Open webOS 1.0: Nothing that can't be fixed | webOS Nation

And the grounds are layed since 2.x newly introduced db8 clouddb for apps

[GMMan]

Originally Posted by rolladude1166

This seems like a good idea. It may just be that I'm new to this whole javascript thing, but it seems kinda tough to do?

It would be implemented serverside using Java or ASP .NET. (Or maybe PHP.) Of course, server side JavaScript can also be an option.

[john mcconnell]

Hum got me think. I like the ideal. What little i know about about net working wont help much but could you mod
the host file? instead of a dns server

john

[GMMan]

Originally Posted by john mcconnell

Hum got me think. I like the ideal. What little i know about about net working wont help much but could you mod
the host file? instead of a dns server

john

The goal is to allow the system to freely add and remove devices. Entering a hosts entry would be device-specific, and things can't be dynamically changed.

[gizmo21]

for many users a 'NAS-enabled' serverside would be great as these are servers at home that run 24/7. So thinking about php or nodes.js on serverside would be nice. Just a suggestion for the 'man on the streets' server possiblities.

[Garfonso]

I love the idea and what you already did investigate...

Am willing to help you as much as I can.

[GMMan]

I chanced upon an interesting server yesterday. It's called BarracudaDrive, and it's a file/application server with server side coding done in Lua. I think it's a good candidate for implementing the backup server, since it's light, fast, has built in database and JSON features, and was designed for file storage. Comes in both Windows and Linux versions, though I'm not sure if they offer non-x86 builds.

BarracudaDrive Web Server

[Garfonso]

At the bottom of the page it says that it runs on the raspberry pi and some other embedded device. So there seems to be ARM support...

[xandros9]

there's no info whatsoever on architecture support.

my itanium has memory issues, so i'll have to talk to some people

eventually

[GMMan]

Looking at this webOS Nation article, it seems to scream cloud backup as the easiest path for implementation. A new provider that pulls out app data and does an incremental compare; uploads it to the backup server, then other devices pull the backups and import the changes.

The webOS Wish List: Synchronized webOS Accounts | webOS Nation

[Isandunk]

This is probably more related than helpful, but have you heard of www.owncloud.com?

[GMMan]

That solution looks expensive. I don't know if there are anyone or group of 50 people who want to fork over $1000 a year for phone backups. (Of course, they can use the service for something else, but exclusively for backup it's really expensive.)

[hans109h]

I like this idea, would love to host my own backup.

Hans

[xandros9]

i have a couple spare computers i could possibly use for it