self-hosted/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2024-12-03 08:45:34 +02:00
Code Issues Releases Activity

# Fixed

Browse Source 
- Illegible coloring of the Diff dialog.

# Implemented
- On-the-fly encryption and decryption in replication.
- Text splitting algorithms updated
(use a bit more memory (which is saved by On-the-fly enc-dec), but faster than old algorithms.)
- Garbage collector is now decent and memory saving.

# Internal things
- Refactored so much.
This commit is contained in:
vorotamoroz 2022-06-09 17:44:08 +09:00
parent f613f1b887
commit e2da4ec454
13 changed files with 763 additions and 457 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
docs/settings.md
View File

@ -1,3 +1,5 @@
NOTE: This document surely became outdated. I'll improve this doc in a while. but your contributions are always welcome.
# Settings of this plugin
## Remote Database Configurations
@ -19,14 +21,51 @@ The Database name to synchronize.
⚠️If not exist, created automatically.
### Use the old connecting method
Since v0.8.0, Self-hosted LiveSync uses Obsidian's API to connect to the CouchDB instead of the browser API.
This method will increase the performance and avoid troubles with the CORS.
But it doesn't been well tested yet. If you are troubled, please disable this option once.
This option has been removed at v0.10.0
### End to End Encryption
Encrypt your database. It affects only the database, your files are left as plain.
The encryption algorithm is AES-GCM.
Note: If you want to use "Plugins and their settings", you have to enable this.
### Passphrase
The passphrase to used as the key of encryption. Please use the long text.
### Apply
To enable End-to-End encryption, there must be no items of the same content encrypted with different passphrases to avoid attackers guessing passphrases. Self-hosted LiveSync uses crc32 of the chunks, It is really a must.
So, this plugin completely deletes everything from both local and remote databases before enabling it and then synchronizing again.
To enable, "Apply and send" from the most powerful device.
If you want to synchronize an existing database, set passphrase and press "Just apply".
- Apply and send
1. Initialize the Local Database and set (or clear) passphrase, put all files into the database again.
2. Initialize the Remote Database.
3. Lock the Remote Database.
4. Send it all.
This process is simply heavy. Using a PC or Mac is preferred.
- Apply and receive
1. Initialize the Local Database and set (or clear) the passphrase.
2. Unlock the Remote Database.
3. Retrieve all and decrypt to file.
When running these operations, every synchronization settings is disabled.
### Test Database connection
You can check the connection by clicking this button.
### Check database configuration
You can check and modify your CouchDB's configuration from here directly.
## Local Database Configurations
"Local Database" is created inside your obsidian.
@ -56,39 +95,6 @@ Note: If you want to use "Use history", this vault must be set to 0.
### Manual Garbage Collect
Run "Garbage Collection" manually.
### End to End Encryption
Encrypt your database. It affects only the database, your files are left as plain.
The encryption algorithm is AES-GCM.
Note: If you want to use "Plugins and their settings", you have to enable this.
### Passphrase
The passphrase to used as the key of encryption. Please use the long text.
### Apply
To enable End-to-End encryption, there must be no items of the same content encrypted with different passphrases to avoid attackers guessing passphrases. Self-hosted LiveSync uses crc32 of the chunks, It is really a must.
So, this plugin completely deletes everything from both local and remote databases before enabling it and then synchronizing again.
To enable, "Apply and send" from the most powerful device, and "Apply and receive" from every other device.
- Apply and send
1. Initialize the Local Database and set (or clear) passphrase, put all files into the database again.
2. Initialize the Remote Database.
3. Lock the Remote Database.
4. Send it all.
This process is simply heavy. Using a PC or Mac is preferred.
- Apply and receive
1. Initialize the Local Database and set (or clear) the passphrase.
2. Unlock the Remote Database.
3. Retrieve all and decrypt to file.
When running these operations, every synchronization settings is disabled.
**And even your passphrase is wrong, It doesn't be checked before the plugin really decrypts. So If you set the wrong passphrase and run "Apply and Receive", you will get an amount of decryption error. But, this is the specification.**
### minimum chunk size and LongLine threshold
The configuration of chunk splitting.

66
docs/settings_ja.md
View File

@ -1,3 +1,5 @@
注意：少し内容が古くなっています。
# このプラグインの設定項目
## Remote Database Configurations
@ -19,9 +21,44 @@ CouchDBのURIを入力します。Cloudantの場合は「External Endpoint(prefe
⚠️存在しない場合は、テストや接続を行った際、自動的に作成されます[^1]。
[^1]:権限がない場合は自動作成には失敗します。
### End to End Encryption
データベースを暗号化します。この効果はデータベースに格納されるデータに限られ、ディスク上のファイルは平文のままです。
暗号化はAES-GCMを使用して行っています。
### Passphrase
暗号化を行う際に使用するパスフレーズです。充分に長いものを使用してください。
### Apply
End to End 暗号化を行うに当たって、異なるパスフレーズで暗号化された同一の内容を入手されることは避けるべきです。また、Self-hosted LiveSyncはコンテンツのcrc32を重複回避に使用しているため、その点でも攻撃が有効になってしまいます。
そのため、End to End 暗号化を有効にする際には、ローカル、リモートすべてのデータベースをいったん破棄し、新しいパスフレーズで暗号化された内容のみを、改めて同期し直します。
有効化するには、一番体力のある端末からApply and sendを行います。
既に存在するリモートと同期する場合は、設定してJust applyを行ってください。
- Apply and send
1. ローカルのデータベースを初期化しパスフレーズを設定（またはクリア）します。その後、すべてのファイルをもう一度データベースに登録します。
2. リモートのデータベースを初期化します。
3. リモートのデータベースをロックし、他の端末を締め出します。
4. すべて再送信します。
負荷と時間がかかるため、デスクトップから行う方が好ましいです。
- Apply and receive
1. ローカルのデータベースを初期化し、パスフレーズを設定（またはクリア）します。
2. リモートのデータベースにかかっているロックを解除します。
3. すべて受信して、復号します。
どちらのオペレーションも、実行するとすべての同期設定が無効化されます。
### Test Database connection
上記の設定でデータベースに接続できるか確認します。
### Check database configuration
ここから直接CouchDBの設定を確認・変更できます。
## Local Database Configurations
端末内に作成されるデータベースの設定です。
@ -49,35 +86,6 @@ Obsidianでのファイル操作が終わってから指定秒数が経過した
### Manual Garbage Collect
上記のGarbage Collectionを手動で行います。
### End to End Encryption
データベースを暗号化します。この効果はデータベースに格納されるデータに限られ、ディスク上のファイルは平文のままです。
暗号化はAES-GCMを使用して行っています。
### Passphrase
暗号化を行う際に使用するパスフレーズです。充分に長いものを使用してください。
### Apply
End to End 暗号化を行うに当たって、異なるパスフレーズで暗号化された同一の内容を入手されることは避けるべきです。また、Self-hosted LiveSyncはコンテンツのcrc32を重複回避に使用しているため、その点でも攻撃が有効になってしまいます。
そのため、End to End 暗号化を有効にする際には、ローカル、リモートすべてのデータベースをいったん破棄し、新しいパスフレーズで暗号化された内容のみを、改めて同期し直します。
有効化するには、一番体力のある端末からApply and sendを行い、他の端末でApply and receiveを行います。
- Apply and send
1. ローカルのデータベースを初期化しパスフレーズを設定（またはクリア）します。その後、すべてのファイルをもう一度データベースに登録します。
2. リモートのデータベースを初期化します。
3. リモートのデータベースをロックし、他の端末を締め出します。
4. すべて再送信します。
負荷と時間がかかるため、デスクトップから行う方が好ましいです。
- Apply and receive
1. ローカルのデータベースを初期化し、パスフレーズを設定（またはクリア）します。
2. リモートのデータベースにかかっているロックを解除します。
3. すべて受信して、復号します。
どちらのオペレーションも、実行するとすべての同期設定が無効化されます。
**また、パスフレーズのチェックは、実際に復号するまで行いません。そのため、パスフレーズを間違えて設定し、Apply and receiveで同期を行うと、大量のエラーが発生します。これは仕様です。**
### minimum chunk size と LongLine threshold
チャンクの分割についての設定です。
Self-hosted LiveSyncは一つのチャンクのサイズを最低minimum chunk size文字確保した上で、できるだけ効率的に同期できるよう、ノートを分割してチャンクを作成します。

11
package-lock.json generated
View File

@ -12,6 +12,7 @@
"diff-match-patch": "^1.0.5",
"esbuild": "0.13.12",
"esbuild-svelte": "^0.6.0",
"idb": "^7.0.1",
"svelte-preprocess": "^4.10.2",
"xxhash-wasm": "^0.4.2"
},
@ -2125,6 +2126,11 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/idb": {
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/idb/-/idb-7.0.1.tgz",
"integrity": "sha512-UUxlE7vGWK5RfB/fDwEGgRf84DY/ieqNha6msMV99UsEMQhJ1RwbCd8AYBj3QMgnE3VZnfQvm4oKVCJTYlqIgg=="
},
"node_modules/ignore": {
"version": "5.1.9",
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.1.9.tgz",
@ -5077,6 +5083,11 @@
"has-symbols": "^1.0.2"
}
},
"idb": {
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/idb/-/idb-7.0.1.tgz",
"integrity": "sha512-UUxlE7vGWK5RfB/fDwEGgRf84DY/ieqNha6msMV99UsEMQhJ1RwbCd8AYBj3QMgnE3VZnfQvm4oKVCJTYlqIgg=="
},
"ignore": {
"version": "5.1.9",
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.1.9.tgz",

1
package.json
View File

@ -37,6 +37,7 @@
"diff-match-patch": "^1.0.5",
"esbuild": "0.13.12",
"esbuild-svelte": "^0.6.0",
"idb": "^7.0.1",
"svelte-preprocess": "^4.10.2",
"xxhash-wasm": "^0.4.2"
}

2
pouchdb-browser-webpack/dist/pouchdb-browser.js vendored
View File

File diff suppressed because one or more lines are too long

44
pouchdb-browser-webpack/package.json
View File

@ -1,23 +1,25 @@
{
"name": "pouchdb-browser-webpack",
"version": "1.0.0",
"description": "pouchdb-browser webpack",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"build": "webpack --mode=production --node-env=production",
"build:dev": "webpack --mode=development",
"build:prod": "webpack --mode=production --node-env=production",
"watch": "webpack --watch"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"pouchdb-browser": "^7.3.0"
},
"devDependencies": {
"webpack": "^5.58.1",
"webpack-cli": "^4.9.0"
}
"name": "pouchdb-browser-webpack",
"version": "1.0.0",
"description": "pouchdb-browser webpack",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"build": "webpack --mode=production --node-env=production",
"build:dev": "webpack --mode=development",
"build:prod": "webpack --mode=production --node-env=production",
"watch": "webpack --watch"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"pouchdb-browser": "^7.3.0",
"transform-pouch": "^2.0.0",
"pouchdb-find": "^7.3.0"
},
"devDependencies": {
"webpack": "^5.58.1",
"webpack-cli": "^4.9.0"
}
}

10
pouchdb-browser-webpack/src/index.js
View File

@ -1,4 +1,10 @@
/* eslint-disable no-undef */
/* eslint-disable @typescript-eslint/no-var-requires */
// This module just webpacks pouchdb-browser
import * as PouchDB_src from "pouchdb-browser";
const PouchDB = PouchDB_src.default;
// import * as PouchDB_src from "pouchdb-browser";
const pouch = require("pouchdb-browser").default;
const find = require("pouchdb-find").default;
const transform = require("transform-pouch");
const PouchDB = pouch.plugin(find).plugin(transform);
export { PouchDB };

547
src/LocalPouchDB.ts
View File

@ -21,15 +21,56 @@ import {
MILSTONE_DOCID,
DatabaseConnectingStatus,
} from "./lib/src/types";
import { decrypt, encrypt } from "./lib/src/e2ee";
import { RemoteDBSettings } from "./lib/src/types";
import { resolveWithIgnoreKnownError, delay, runWithLock, isPlainText, splitPieces, NewNotice, WrappedNotice } from "./lib/src/utils";
import { resolveWithIgnoreKnownError, delay, runWithLock, NewNotice, WrappedNotice, shouldSplitAsPlainText, splitPieces2, enableEncryption } from "./lib/src/utils";
import { path2id } from "./utils";
import { Logger } from "./lib/src/logger";
import { checkRemoteVersion, connectRemoteCouchDB, getLastPostFailedBySize } from "./utils_couchdb";
import { checkRemoteVersion, connectRemoteCouchDBWithSetting, getLastPostFailedBySize } from "./utils_couchdb";
import { openDB, deleteDB, IDBPDatabase } from "idb";
type ReplicationCallback = (e: PouchDB.Core.ExistingDocument<EntryDoc>[]) => Promise<void>;
class LRUCache {
cache = new Map<string, string>([]);
revCache = new Map<string, string>([]);
maxCache = 100;
constructor() {}
get(key: string) {
// debugger
const v = this.cache.get(key);
if (v) {
// update the key to recently used.
this.cache.delete(key);
this.revCache.delete(v);
this.cache.set(key, v);
this.revCache.set(v, key);
}
return v;
}
revGet(value: string) {
// debugger
const key = this.revCache.get(value);
if (value) {
// update the key to recently used.
this.cache.delete(key);
this.revCache.delete(value);
this.cache.set(key, value);
this.revCache.set(value, key);
}
return key;
}
set(key: string, value: string) {
this.cache.set(key, value);
this.revCache.set(value, key);
if (this.cache.size > this.maxCache) {
for (const kv of this.cache) {
this.revCache.delete(kv[1]);
this.cache.delete(kv[0]);
if (this.cache.size <= this.maxCache) break;
}
}
}
}
export class LocalPouchDB {
auth: Credential;
dbname: string;
@ -42,12 +83,13 @@ export class LocalPouchDB {
h32: (input: string, seed?: number) => string;
h64: (input: string, seedHigh?: number, seedLow?: number) => string;
h32Raw: (input: Uint8Array, seed?: number) => number;
hashCache: {
[key: string]: string;
} = {};
hashCacheRev: {
[key: string]: string;
} = {};
// hashCache: {
// [key: string]: string;
// } = {};
// hashCacheRev: {
// [key: string]: string;
// } = {};
hashCaches = new LRUCache();
corruptedEntries: { [key: string]: EntryDoc } = {};
remoteLocked = false;
@ -90,8 +132,6 @@ export class LocalPouchDB {
this.settings = settings;
this.cancelHandler = this.cancelHandler.bind(this);
this.isMobile = isMobile;
// this.initializeDatabase();
}
close() {
Logger("Database closed (by close)");
@ -101,10 +141,6 @@ export class LocalPouchDB {
this.localDatabase.close();
}
}
disposeHashCache() {
this.hashCache = {};
this.hashCacheRev = {};
}
updateRecentModifiedDocs(id: string, rev: string, deleted: boolean) {
const idrev = id + rev;
@ -119,52 +155,132 @@ export class LocalPouchDB {
const idrev = id + rev;
return this.recentModifiedDocs.indexOf(idrev) !== -1;
}
async initializeDatabase() {
async isOldDatabaseExists() {
const db = new PouchDB<EntryDoc>(this.dbname + "-livesync", {
auto_compaction: this.settings.useHistory ? false : true,
revs_limit: 100,
deterministic_revs: true,
skip_setup: true,
});
try {
const info = await db.info();
Logger(info, LOG_LEVEL.VERBOSE);
return db;
} catch (ex) {
return false;
}
}
async initializeDatabase(): Promise<boolean> {
await this.prepareHashFunctions();
if (this.localDatabase != null) this.localDatabase.close();
this.changeHandler = this.cancelHandler(this.changeHandler);
this.localDatabase = null;
this.localDatabase = new PouchDB<EntryDoc>(this.dbname + "-livesync", {
this.localDatabase = new PouchDB<EntryDoc>(this.dbname + "-livesync-v2", {
auto_compaction: this.settings.useHistory ? false : true,
revs_limit: 100,
deterministic_revs: true,
});
Logger("Database Info");
Logger("Database info", LOG_LEVEL.VERBOSE);
Logger(await this.localDatabase.info(), LOG_LEVEL.VERBOSE);
// initialize local node information.
const nodeinfo: EntryNodeInfo = await resolveWithIgnoreKnownError<EntryNodeInfo>(this.localDatabase.get(NODEINFO_DOCID), {
_id: NODEINFO_DOCID,
type: "nodeinfo",
nodeid: "",
});
if (nodeinfo.nodeid == "") {
nodeinfo.nodeid = Math.random().toString(36).slice(-10);
await this.localDatabase.put(nodeinfo);
}
this.localDatabase.on("close", () => {
Logger("Database closed.");
this.isReady = false;
this.localDatabase.removeAllListeners();
});
this.nodeid = nodeinfo.nodeid;
// Traceing the leaf id
const changes = this.localDatabase
.changes({
since: "now",
live: true,
filter: (doc) => doc.type == "leaf",
})
.on("change", (e) => {
if (e.deleted) return;
this.leafArrived(e.id);
this.docSeq = `${e.seq}`;
Logger("Open Database...");
// The sequence after migration.
const nextSeq = async (): Promise<boolean> => {
Logger("Database Info");
Logger(await this.localDatabase.info(), LOG_LEVEL.VERBOSE);
// initialize local node information.
const nodeinfo: EntryNodeInfo = await resolveWithIgnoreKnownError<EntryNodeInfo>(this.localDatabase.get(NODEINFO_DOCID), {
_id: NODEINFO_DOCID,
type: "nodeinfo",
nodeid: "",
v20220607: true,
});
this.changeHandler = changes;
this.isReady = true;
Logger("Database is now ready.");
if (nodeinfo.nodeid == "") {
nodeinfo.nodeid = Math.random().toString(36).slice(-10);
await this.localDatabase.put(nodeinfo);
}
this.localDatabase.on("close", () => {
Logger("Database closed.");
this.isReady = false;
this.localDatabase.removeAllListeners();
});
this.nodeid = nodeinfo.nodeid;
// Traceing the leaf id
const changes = this.localDatabase
.changes({
since: "now",
live: true,
filter: (doc) => doc.type == "leaf",
})
.on("change", (e) => {
if (e.deleted) return;
this.leafArrived(e.id);
this.docSeq = `${e.seq}`;
});
this.changeHandler = changes;
this.isReady = true;
Logger("Database is now ready.");
return true;
};
Logger("Checking old database", LOG_LEVEL.VERBOSE);
const old = await this.isOldDatabaseExists();
//Migrate.
if (old) {
const oi = await old.info();
if (oi.doc_count == 0) {
Logger("Old database is empty, proceed to next step", LOG_LEVEL.VERBOSE);
// aleady converted.
return nextSeq();
}
//
const progress = NewNotice("Converting..", 0);
try {
Logger("We have to upgrade database..", LOG_LEVEL.NOTICE);
// To debug , uncomment below.
// this.localDatabase.destroy();
// await delay(100);
// this.localDatabase = new PouchDB<EntryDoc>(this.dbname + "-livesync-v2", {
// auto_compaction: this.settings.useHistory ? false : true,
// revs_limit: 100,
// deterministic_revs: true,
// });
const newDbStatus = await this.localDatabase.info();
Logger("New database is initialized");
Logger(newDbStatus);
if (this.settings.encrypt) {
enableEncryption(old, this.settings.passphrase);
}
const rep = old.replicate.to(this.localDatabase);
rep.on("change", (e) => {
progress.setMessage(`Converting ${e.docs_written} docs...`);
Logger(`Converting ${e.docs_written} docs...`, LOG_LEVEL.VERBOSE);
});
const w = await rep;
progress.hide();
if (w.ok) {
Logger("Conversion completed!", LOG_LEVEL.NOTICE);
old.destroy(); // delete the old database.
this.isReady = true;
return nextSeq();
} else {
throw new Error("Conversion failed!");
}
} catch (ex) {
progress.hide();
Logger("Conversion failed!, If you are fully synchronized, please drop the old database in the Hatch pane in setting dialog. or please make an issue on Github.", LOG_LEVEL.NOTICE);
Logger(ex);
this.isReady = false;
return false;
}
} else {
return nextSeq();
}
}
async prepareHashFunctions() {
@ -203,55 +319,28 @@ export class LocalPouchDB {
async getDBLeaf(id: string, waitForReady: boolean): Promise<string> {
await this.waitForGCComplete();
// when in cache, use that.
if (this.hashCacheRev[id]) {
return this.hashCacheRev[id];
const leaf = this.hashCaches.revGet(id);
if (leaf) {
return leaf;
}
try {
const w = await this.localDatabase.get(id);
if (w.type == "leaf") {
if (id.startsWith("h:+")) {
try {
w.data = await decrypt(w.data, this.settings.passphrase);
} catch (e) {
Logger("The element of the document has been encrypted, but decryption failed.", LOG_LEVEL.NOTICE);
throw e;
}
}
this.hashCache[w.data] = id;
this.hashCacheRev[id] = w.data;
this.hashCaches.set(id, w.data);
return w.data;
}
throw new Error(`retrive leaf, but it was not leaf.`);
} catch (ex) {
if (ex.status && ex.status == 404 && waitForReady) {
// just leaf is not ready.
// wait for on
if ((await this.waitForLeafReady(id)) === false) {
throw new Error(`time out (waiting leaf)`);
}
try {
// retrive again.
const w = await this.localDatabase.get(id);
if (w.type == "leaf") {
if (id.startsWith("h:+")) {
try {
w.data = await decrypt(w.data, this.settings.passphrase);
} catch (e) {
Logger("The element of the document has been encrypted, but decryption failed.", LOG_LEVEL.NOTICE);
throw e;
}
}
this.hashCache[w.data] = id;
this.hashCacheRev[id] = w.data;
return w.data;
if (ex.status && ex.status == 404) {
if (waitForReady) {
// just leaf is not ready.
// wait for on
if ((await this.waitForLeafReady(id)) === false) {
throw new Error(`time out (waiting leaf)`);
}
throw new Error(`retrive leaf, but it was not leaf.`);
} catch (ex) {
if (ex.status && ex.status == 404) {
throw new Error("leaf is not found");
}
Logger(`Something went wrong on retriving leaf`);
throw ex;
return this.getDBLeaf(id, false);
} else {
throw new Error("Leaf was not found");
}
} else {
Logger(`Something went wrong on retriving leaf`);
@ -517,7 +606,7 @@ export class LocalPouchDB {
let plainSplit = false;
let cacheUsed = 0;
const userpasswordHash = this.h32Raw(new TextEncoder().encode(this.settings.passphrase));
if (isPlainText(note._id)) {
if (shouldSplitAsPlainText(note._id)) {
pieceSize = MAX_DOC_SIZE;
plainSplit = true;
}
@ -535,7 +624,7 @@ export class LocalPouchDB {
let longLineThreshold = this.settings.longLineThreshold;
if (longLineThreshold < 100) longLineThreshold = 100;
const pieces = splitPieces(note.data, pieceSize, plainSplit, minimumChunkSize, longLineThreshold);
const pieces = splitPieces2(note.data, pieceSize, plainSplit, minimumChunkSize, longLineThreshold);
for (const piece of pieces()) {
processed++;
let leafid = "";
@ -544,9 +633,10 @@ export class LocalPouchDB {
let hashQ = 0; // if hash collided, **IF**, count it up.
let tryNextHash = false;
let needMake = true;
if (typeof this.hashCache[piece] !== "undefined") {
const cache = this.hashCaches.get(piece);
if (cache) {
hashedPiece = "";
leafid = this.hashCache[piece];
leafid = cache;
needMake = false;
skiped++;
cacheUsed++;
@ -563,21 +653,11 @@ export class LocalPouchDB {
try {
nleafid = `${leafid}${hashQ}`;
const pieceData = await this.localDatabase.get<EntryLeaf>(nleafid);
//try decode
if (pieceData._id.startsWith("h:+")) {
try {
pieceData.data = await decrypt(pieceData.data, this.settings.passphrase);
} catch (e) {
Logger("Decode failed!");
throw e;
}
}
if (pieceData.type == "leaf" && pieceData.data == piece) {
leafid = nleafid;
needMake = false;
tryNextHash = false;
this.hashCache[piece] = leafid;
this.hashCacheRev[leafid] = piece;
this.hashCaches.set(piece, leafid);
} else if (pieceData.type == "leaf") {
Logger("hash:collision!!");
hashQ++;
@ -601,19 +681,15 @@ export class LocalPouchDB {
} while (tryNextHash);
if (needMake) {
//have to make
let savePiece = piece;
if (this.settings.encrypt) {
const passphrase = this.settings.passphrase;
savePiece = await encrypt(piece, passphrase);
}
const savePiece = piece;
const d: EntryLeaf = {
_id: leafid,
data: savePiece,
type: "leaf",
};
newLeafs.push(d);
this.hashCache[piece] = leafid;
this.hashCacheRev[leafid] = piece;
this.hashCaches.set(piece, leafid);
made++;
} else {
skiped++;
@ -635,7 +711,6 @@ export class LocalPouchDB {
} else {
Logger(`save failed:id:${item.id} rev:${item.rev}`, LOG_LEVEL.NOTICE);
Logger(item);
// this.disposeHashCache();
saved = false;
}
}
@ -707,7 +782,7 @@ export class LocalPouchDB {
this.openOneshotReplication(
setting,
showingNotice,
async (e) => { },
async (e) => {},
false,
(e) => {
if (e === true) res(e);
@ -731,16 +806,12 @@ export class LocalPouchDB {
return false;
}
const uri = setting.couchDB_URI + (setting.couchDB_DBNAME == "" ? "" : "/" + setting.couchDB_DBNAME);
const auth: Credential = {
username: setting.couchDB_USER,
password: setting.couchDB_PASSWORD,
};
if (this.syncHandler != null) {
Logger("Another replication running.");
return false;
}
const dbret = await connectRemoteCouchDB(uri, auth, setting.disableRequestURI || this.isMobile);
const dbret = await connectRemoteCouchDBWithSetting(setting, this.isMobile);
if (typeof dbret === "string") {
Logger(`could not connect to ${uri}: ${dbret}`, showResult ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO);
return false;
@ -1038,6 +1109,15 @@ export class LocalPouchDB {
Logger("Replication closed");
}
async resetLocalOldDatabase() {
const oldDB = await this.isOldDatabaseExists();
if (oldDB) {
oldDB.destroy();
NewNotice("Deleted! Please re-launch obsidian.", LOG_LEVEL.NOTICE);
} else {
NewNotice("Old database is not exist.", LOG_LEVEL.NOTICE);
}
}
async resetDatabase() {
await this.waitForGCComplete();
this.changeHandler = this.cancelHandler(this.changeHandler);
@ -1047,17 +1127,11 @@ export class LocalPouchDB {
await this.localDatabase.destroy();
this.localDatabase = null;
await this.initializeDatabase();
this.disposeHashCache();
Logger("Local Database Reset", LOG_LEVEL.NOTICE);
}
async tryResetRemoteDatabase(setting: RemoteDBSettings) {
await this.closeReplication();
const uri = setting.couchDB_URI + (setting.couchDB_DBNAME == "" ? "" : "/" + setting.couchDB_DBNAME);
const auth: Credential = {
username: setting.couchDB_USER,
password: setting.couchDB_PASSWORD,
};
const con = await connectRemoteCouchDB(uri, auth, setting.disableRequestURI || this.isMobile);
const con = await connectRemoteCouchDBWithSetting(setting, this.isMobile);
if (typeof con == "string") return;
try {
await con.db.destroy();
@ -1070,22 +1144,14 @@ export class LocalPouchDB {
}
async tryCreateRemoteDatabase(setting: RemoteDBSettings) {
await this.closeReplication();
const uri = setting.couchDB_URI + (setting.couchDB_DBNAME == "" ? "" : "/" + setting.couchDB_DBNAME);
const auth: Credential = {
username: setting.couchDB_USER,
password: setting.couchDB_PASSWORD,
};
const con2 = await connectRemoteCouchDB(uri, auth, setting.disableRequestURI || this.isMobile);
const con2 = await connectRemoteCouchDBWithSetting(setting, this.isMobile);
if (typeof con2 === "string") return;
Logger("Remote Database Created or Connected", LOG_LEVEL.NOTICE);
}
async markRemoteLocked(setting: RemoteDBSettings, locked: boolean) {
const uri = setting.couchDB_URI + (setting.couchDB_DBNAME == "" ? "" : "/" + setting.couchDB_DBNAME);
const auth: Credential = {
username: setting.couchDB_USER,
password: setting.couchDB_PASSWORD,
};
const dbret = await connectRemoteCouchDB(uri, auth, setting.disableRequestURI || this.isMobile);
const dbret = await connectRemoteCouchDBWithSetting(setting, this.isMobile);
if (typeof dbret === "string") {
Logger(`could not connect to ${uri}:${dbret}`, LOG_LEVEL.NOTICE);
return;
@ -1115,11 +1181,7 @@ export class LocalPouchDB {
}
async markRemoteResolved(setting: RemoteDBSettings) {
const uri = setting.couchDB_URI + (setting.couchDB_DBNAME == "" ? "" : "/" + setting.couchDB_DBNAME);
const auth: Credential = {
username: setting.couchDB_USER,
password: setting.couchDB_PASSWORD,
};
const dbret = await connectRemoteCouchDB(uri, auth, setting.disableRequestURI || this.isMobile);
const dbret = await connectRemoteCouchDBWithSetting(setting, this.isMobile);
if (typeof dbret === "string") {
Logger(`could not connect to ${uri}:${dbret}`, LOG_LEVEL.NOTICE);
return;
@ -1159,7 +1221,6 @@ export class LocalPouchDB {
const dc = await this.localDatabase.allDocs({ keys: [...children] });
if (dc.rows.some((e) => "error" in e)) {
this.corruptedEntries[entry._id] = entry;
this.disposeHashCache();
Logger(`sancheck:corrupted:${entry._id} : ${children.length}`, LOG_LEVEL.VERBOSE);
return false;
}
@ -1180,84 +1241,150 @@ export class LocalPouchDB {
await runWithLock("replicate", true, async () => {
if (this.gcRunning) return;
this.gcRunning = true;
let idbGC: IDBPDatabase<{ id: string }> = null;
const storeIDB = "gc";
const idbname = "idb-" + this.dbname + "-idb-gcx";
try {
// get all documents of NewEntry2
// we don't use queries , just use allDocs();
this.disposeHashCache();
let c = 0;
let readCount = 0;
let hashPieces: string[] = [];
let usedPieces: string[] = [];
Logger("Collecting Garbage");
do {
const result = await this.localDatabase.allDocs({ include_docs: false, skip: c, limit: 2000, conflicts: true });
readCount = result.rows.length;
Logger("checked:" + readCount);
if (readCount > 0) {
//there are some result
for (const v of result.rows) {
if (v.id.startsWith("h:")) {
hashPieces = Array.from(new Set([...hashPieces, v.id]));
} else {
const docT = await this.localDatabase.get(v.id, { revs_info: true });
const revs = docT._revs_info;
// console.log(`revs:${revs.length}`)
for (const rev of revs) {
if (rev.status != "available") continue;
// console.log(`id:${docT._id},rev:${rev.rev}`);
const doc = await this.localDatabase.get(v.id, { rev: rev.rev });
if ("children" in doc) {
// used pieces memo.
usedPieces = Array.from(new Set([...usedPieces, ...doc.children]));
if (doc._conflicts) {
for (const cid of doc._conflicts) {
const p = await this.localDatabase.get<EntryDoc>(doc._id, { rev: cid });
if (p.type == "newnote" || p.type == "plain") {
usedPieces = Array.from(new Set([...usedPieces, ...p.children]));
}
}
}
}
const procAllDocs = async (getLeaf: boolean, startkey: string, endkey: string, callback: (idordoc: string[]) => Promise<void>) => {
let c = 0;
let readCount = 0;
do {
const result = await this.localDatabase.allDocs({ include_docs: false, skip: c, limit: 2000, conflicts: !getLeaf, startkey: startkey, endkey: endkey });
readCount = result.rows.length;
if (readCount > 0) {
await callback(result.rows.map((e) => e.id));
}
c += readCount;
} while (readCount != 0);
};
// Delete working indexedDB once.
await deleteDB(idbname);
idbGC = await openDB(idbname, 1, {
upgrade(db) {
db.createObjectStore(storeIDB, { keyPath: "id" });
},
});
// Mark all chunks once.
await procAllDocs(true, "h:", "h_", async (docs) => {
Logger(`Chunks marked - :${docs.length}`);
const tx = idbGC.transaction(storeIDB, "readwrite");
const store = tx.objectStore(storeIDB);
for (const docId of docs) {
await store.put({ id: docId });
}
await tx.done;
});
Logger("All chunks are marked once");
const unmarkUsedByHashId = async (doc: EntryDoc) => {
if ("children" in doc) {
const tx = idbGC.transaction(storeIDB, "readwrite");
const store = tx.objectStore(storeIDB);
for (const hashId of doc.children) {
await store.delete(hashId);
}
await tx.done;
}
};
Logger("Processing existen docs");
let procDocs = 0;
await procAllDocs(false, null, null, async (doc) => {
const docIds = (doc as string[]).filter((e) => !e.startsWith("h:") && !e.startsWith("ps:"));
for (const docId of docIds) {
procDocs++;
if (procDocs % 25 == 0) Logger(`${procDocs} Processed`);
const docT = await this.localDatabase.get(docId, { revs_info: true });
if (docT._deleted) continue;
// Unmark about latest doc.
unmarkUsedByHashId(docT);
const revs = docT._revs_info;
// Unmark old revisions
for (const rev of revs) {
if (rev.status != "available") continue;
const docRev = await this.localDatabase.get(docId, { rev: rev.rev });
unmarkUsedByHashId(docRev);
if (docRev._conflicts) {
// Unmark the conflicted chunks of old revisions.
for (const cid of docRev._conflicts) {
const docConflict = await this.localDatabase.get<EntryDoc>(docId, { rev: cid });
unmarkUsedByHashId(docConflict);
}
}
}
}
c += readCount;
} while (readCount != 0);
// items collected.
Logger("Finding unused pieces");
this.disposeHashCache();
const garbages = hashPieces.filter((e) => usedPieces.indexOf(e) == -1);
let deleteCount = 0;
Logger("we have to delete:" + garbages.length);
let deleteDoc: EntryDoc[] = [];
for (const v of garbages) {
try {
const item = await this.localDatabase.get(v);
item._deleted = true;
deleteDoc.push(item);
if (deleteDoc.length > 50) {
await this.localDatabase.bulkDocs<EntryDoc>(deleteDoc);
deleteDoc = [];
Logger("delete:" + deleteCount);
// Unmark the conflicted chunk.
if (docT._conflicts) {
for (const cid of docT._conflicts) {
const docConflict = await this.localDatabase.get<EntryDoc>(docId, { rev: cid });
unmarkUsedByHashId(docConflict);
}
}
deleteCount++;
} catch (ex) {
if (ex.status && ex.status == 404) {
// NO OP. It should be timing problem.
}
});
// All marked chunks could be deleted.
Logger("Delete non-used chunks");
let dataLeft = false;
let chunkKeys: string[] = [];
let totalDelCount = 0;
do {
const tx = idbGC.transaction(storeIDB, "readonly");
const store = tx.objectStore(storeIDB);
let cursor = await store.openCursor();
if (cursor == null) break;
const maxconcurrentDocs = 10;
let delChunkCount = 0;
do {
// console.log(cursor.key, cursor.value);
if (cursor) {
chunkKeys.push(cursor.key as string);
delChunkCount++;
dataLeft = true;
} else {
throw ex;
dataLeft = false;
}
cursor = await cursor.continue();
} while (cursor && dataLeft && delChunkCount < maxconcurrentDocs);
// if (chunkKeys.length > 0) {
totalDelCount += delChunkCount;
const delDocResult = await this.localDatabase.allDocs({ keys: chunkKeys, include_docs: true });
const delDocs = delDocResult.rows.map((e) => ({ ...e.doc, _deleted: true }));
await this.localDatabase.bulkDocs(delDocs);
Logger(`deleted from pouchdb:${delDocs.length}`);
const tx2 = idbGC.transaction(storeIDB, "readwrite");
const store2 = tx2.objectStore(storeIDB);
for (const doc of chunkKeys) {
await store2.delete(doc);
}
Logger(`deleted from workspace:${chunkKeys.length}`);
await tx2.done;
// }
chunkKeys = [];
} while (dataLeft);
Logger(`Deleted ${totalDelCount} chunks`);
Logger("Teardown the database");
if (idbGC != null) {
idbGC.close();
idbGC = null;
}
if (deleteDoc.length > 0) {
await this.localDatabase.bulkDocs<EntryDoc>(deleteDoc);
}
Logger(`GC:deleted ${deleteCount} items.`);
await deleteDB(idbname);
this.gcRunning = false;
Logger("Done");
} catch (ex) {
Logger("Error on garbage collection");
Logger(ex);
} finally {
if (idbGC != null) {
idbGC.close();
}
await deleteDB(idbname);
this.gcRunning = false;
}
});
this.disposeHashCache();
}
}

250
src/ObsidianLiveSyncSettingTab.ts
View File

@ -1,9 +1,9 @@
import { App, PluginSettingTab, Setting, sanitizeHTMLToDom, RequestUrlParam, requestUrl } from "obsidian";
import { EntryDoc, LOG_LEVEL } from "./lib/src/types";
import { EntryDoc, LOG_LEVEL, RemoteDBSettings } from "./lib/src/types";
import { path2id, id2path } from "./utils";
import { NewNotice, runWithLock } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { connectRemoteCouchDB } from "./utils_couchdb";
import { checkSyncInfo, connectRemoteCouchDBWithSetting } from "./utils_couchdb";
import { testCrypt } from "./lib/src/e2ee";
import ObsidianLiveSyncPlugin from "./main";
@ -15,14 +15,16 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin = plugin;
}
async testConnection(): Promise<void> {
const db = await connectRemoteCouchDB(
this.plugin.settings.couchDB_URI + (this.plugin.settings.couchDB_DBNAME == "" ? "" : "/" + this.plugin.settings.couchDB_DBNAME),
{
username: this.plugin.settings.couchDB_USER,
password: this.plugin.settings.couchDB_PASSWORD,
},
this.plugin.settings.disableRequestURI
);
// const db = await connectRemoteCouchDB(
// this.plugin.settings.couchDB_URI + (this.plugin.settings.couchDB_DBNAME == "" ? "" : "/" + this.plugin.settings.couchDB_DBNAME),
// {
// username: this.plugin.settings.couchDB_USER,
// password: this.plugin.settings.couchDB_PASSWORD,
// },
// this.plugin.settings.disableRequestURI,
// this.plugin.settings.encrypt ? this.plugin.settings.passphrase : this.plugin.settings.encrypt
// );
const db = await connectRemoteCouchDBWithSetting(this.plugin.settings, this.plugin.localDatabase.isMobile);
if (typeof db === "string") {
this.plugin.addLog(`could not connect to ${this.plugin.settings.couchDB_URI} : ${this.plugin.settings.couchDB_DBNAME} \n(${db})`, LOG_LEVEL.NOTICE);
return;
@ -78,7 +80,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const containerRemoteDatabaseEl = containerEl.createDiv();
containerRemoteDatabaseEl.createEl("h3", { text: "Remote Database configuration" });
const syncWarn = containerRemoteDatabaseEl.createEl("div", { text: `These settings are kept locked while automatic synchronization options are enabled. Disable these options in the "Sync Settings" tab to unlock.` });
syncWarn.addClass("op-warn");
syncWarn.addClass("op-warn-info");
syncWarn.addClass("sls-hidden");
const isAnySyncEnabled = (): boolean => {
@ -170,20 +172,126 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin.settings.couchDB_DBNAME = value;
await this.plugin.saveSettings();
})
),
)
new Setting(containerRemoteDatabaseEl)
.setDesc("This feature is locked in mobile")
.setName("Use the old connecting method")
.addToggle((toggle) => {
toggle.setValue(this.plugin.settings.disableRequestURI).onChange(async (value) => {
this.plugin.settings.disableRequestURI = value;
// new Setting(containerRemoteDatabaseEl)
// .setDesc("This feature is locked in mobile")
// .setName("Use the old connecting method")
// .addToggle((toggle) => {
// toggle.setValue(this.plugin.settings.disableRequestURI).onChange(async (value) => {
// this.plugin.settings.disableRequestURI = value;
// await this.plugin.saveSettings();
// });
// toggle.setDisabled(this.plugin.isMobile);
// return toggle;
// })
);
new Setting(containerRemoteDatabaseEl)
.setName("End to End Encryption")
.setDesc("Encrypting contents on the database.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.workingEncrypt).onChange(async (value) => {
this.plugin.settings.workingEncrypt = value;
phasspharase.setDisabled(!value);
await this.plugin.saveSettings();
})
);
const phasspharase = new Setting(containerRemoteDatabaseEl)
.setName("Passphrase")
.setDesc("Encrypting passphrase")
.addText((text) => {
text.setPlaceholder("")
.setValue(this.plugin.settings.workingPassphrase)
.onChange(async (value) => {
this.plugin.settings.workingPassphrase = value;
await this.plugin.saveSettings();
});
toggle.setDisabled(this.plugin.isMobile);
return toggle;
})
);
text.inputEl.setAttribute("type", "password");
});
phasspharase.setDisabled(!this.plugin.settings.workingEncrypt);
containerRemoteDatabaseEl.createEl("div", {
text: "If you change the passphrase, rebuilding the remote database is required. Please press 'Apply and send'. Or, If you have configured it to connect to an existing database, click 'Just apply'.",
});
const checkWorkingPassphrase = async (): Promise<boolean> => {
const settingForCheck: RemoteDBSettings = {
...this.plugin.settings,
encrypt: this.plugin.settings.workingEncrypt,
passphrase: this.plugin.settings.workingPassphrase,
};
console.dir(settingForCheck);
const db = await connectRemoteCouchDBWithSetting(settingForCheck, this.plugin.localDatabase.isMobile);
if (typeof db === "string") {
Logger("Could not connect to the database.", LOG_LEVEL.NOTICE);
return false;
} else {
if (await checkSyncInfo(db.db)) {
// Logger("Database connected", LOG_LEVEL.NOTICE);
return true;
} else {
Logger("Failed to read remote database", LOG_LEVEL.NOTICE);
return false;
}
}
};
const applyEncryption = async (sendToServer: boolean) => {
if (this.plugin.settings.workingEncrypt && this.plugin.settings.workingPassphrase == "") {
Logger("If you enable encryption, you have to set the passphrase", LOG_LEVEL.NOTICE);
return;
}
if (this.plugin.settings.workingEncrypt && !(await testCrypt())) {
Logger("WARNING! Your device would not support encryption.", LOG_LEVEL.NOTICE);
return;
}
if (!(await checkWorkingPassphrase())) {
return;
}
if (!this.plugin.settings.workingEncrypt) {
this.plugin.settings.workingPassphrase = "";
}
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.encrypt = this.plugin.settings.workingEncrypt;
this.plugin.settings.passphrase = this.plugin.settings.workingPassphrase;
await this.plugin.saveSettings();
// await this.plugin.resetLocalDatabase();
if (sendToServer) {
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.replicateAllToServer(true);
} else {
await this.plugin.markRemoteResolved();
await this.plugin.replicate(true);
}
};
new Setting(containerRemoteDatabaseEl)
.setName("Apply")
.setDesc("apply encryption settinngs, and re-initialize remote database")
.addButton((button) =>
button
.setButtonText("Apply and send")
.setWarning()
.setDisabled(false)
.setClass("sls-btn-left")
.onClick(async () => {
await applyEncryption(true);
})
)
.addButton((button) =>
button
.setButtonText("Just apply")
.setWarning()
.setDisabled(false)
.setClass("sls-btn-right")
.onClick(async () => {
await applyEncryption(false);
})
);
new Setting(containerRemoteDatabaseEl)
.setName("Test Database Connection")
@ -408,88 +516,6 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
await this.plugin.garbageCollect();
})
);
new Setting(containerLocalDatabaseEl)
.setName("End to End Encryption")
.setDesc("Encrypting contents on the database.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.workingEncrypt).onChange(async (value) => {
this.plugin.settings.workingEncrypt = value;
phasspharase.setDisabled(!value);
await this.plugin.saveSettings();
})
);
const phasspharase = new Setting(containerLocalDatabaseEl)
.setName("Passphrase")
.setDesc("Encrypting passphrase")
.addText((text) => {
text.setPlaceholder("")
.setValue(this.plugin.settings.workingPassphrase)
.onChange(async (value) => {
this.plugin.settings.workingPassphrase = value;
await this.plugin.saveSettings();
});
text.inputEl.setAttribute("type", "password");
});
phasspharase.setDisabled(!this.plugin.settings.workingEncrypt);
containerLocalDatabaseEl.createEl("div", {
text: "When you change any encryption enabled or passphrase, you have to reset all databases to make sure that the last password is unused and erase encrypted data from anywhere. This operation will not lost your vault if you are fully synced.",
});
const applyEncryption = async (sendToServer: boolean) => {
if (this.plugin.settings.workingEncrypt && this.plugin.settings.workingPassphrase == "") {
Logger("If you enable encryption, you have to set the passphrase", LOG_LEVEL.NOTICE);
return;
}
if (this.plugin.settings.workingEncrypt && !(await testCrypt())) {
Logger("WARNING! Your device would not support encryption.", LOG_LEVEL.NOTICE);
return;
}
if (!this.plugin.settings.workingEncrypt) {
this.plugin.settings.workingPassphrase = "";
}
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.encrypt = this.plugin.settings.workingEncrypt;
this.plugin.settings.passphrase = this.plugin.settings.workingPassphrase;
await this.plugin.saveSettings();
await this.plugin.resetLocalDatabase();
if (sendToServer) {
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.replicateAllToServer(true);
} else {
await this.plugin.markRemoteResolved();
await this.plugin.replicate(true);
}
};
new Setting(containerLocalDatabaseEl)
.setName("Apply")
.setDesc("apply encryption settinngs, and re-initialize database")
.addButton((button) =>
button
.setButtonText("Apply and send")
.setWarning()
.setDisabled(false)
.setClass("sls-btn-left")
.onClick(async () => {
await applyEncryption(true);
})
)
.addButton((button) =>
button
.setButtonText("Apply and receive")
.setWarning()
.setDisabled(false)
.setClass("sls-btn-right")
.onClick(async () => {
await applyEncryption(false);
})
);
containerLocalDatabaseEl.createEl("div", {
text: sanitizeHTMLToDom(`Advanced settings<br>
@ -830,7 +856,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
}
const hatchWarn = containerHatchEl.createEl("div", { text: `To stop the bootup sequence for fixing problems on databases, you can put redflag.md on top of your vault (Rebooting obsidian is required).` });
hatchWarn.addClass("op-warn");
hatchWarn.addClass("op-warn-info");
const dropHistory = async (sendToServer: boolean) => {
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
@ -1001,6 +1027,20 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
})
);
new Setting(containerHatchEl)
.setName("Drop old encrypted database")
.setDesc("WARNING: Please use this button only when you have failed on converting old-style localdatabase at v0.10.0.")
.addButton((button) =>
button
.setButtonText("Drop")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await this.plugin.resetLocalOldDatabase();
await this.plugin.initializeDatabase();
})
);
addScreenElement("50", containerHatchEl);
// With great respect, thank you TfTHacker!
// refered: https://github.com/TfTHacker/obsidian42-brat/blob/main/src/features/BetaPlugins.ts

2
src/lib

@ -1 +1 @@
Subproject commit d495f4577ab2a90f3471d21336b3dcf38b861d40
Subproject commit 92c1bbc45dc7961d3af65dd151a9b3d3b48b18ec

123
src/main.ts
View File

@ -1,7 +1,7 @@
import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, PluginManifest, Modal, App } from "obsidian";
import { diff_match_patch } from "diff-match-patch";
import { EntryDoc, LoadedEntry, ObsidianLiveSyncSettings, diff_check_result, diff_result_leaf, EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, diff_result, FLAGMD_REDFLAG } from "./lib/src/types";
import { EntryDoc, LoadedEntry, ObsidianLiveSyncSettings, diff_check_result, diff_result_leaf, EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, diff_result, FLAGMD_REDFLAG, SYNCINFO_ID } from "./lib/src/types";
import { PluginDataEntry, PERIODIC_PLUGIN_SWEEP, PluginList, DevicePluginList } from "./types";
import {
base64ToString,
@ -17,6 +17,7 @@ import {
setNoticeClass,
NewNotice,
allSettledWithConcurrencyLimit,
getLocks,
} from "./lib/src/utils";
import { Logger, setLogger } from "./lib/src/logger";
import { LocalPouchDB } from "./LocalPouchDB";
@ -27,6 +28,7 @@ import { DocumentHistoryModal } from "./DocumentHistoryModal";
import PluginPane from "./PluginPane.svelte";
import { id2path, path2id } from "./utils";
const isDebug = false;
setNoticeClass(Notice);
class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
@ -161,37 +163,42 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
this.addSettingTab(new ObsidianLiveSyncSettingTab(this.app, this));
this.app.workspace.onLayoutReady(async () => {
try {
if (this.isRedFlagRaised()) {
this.settings.batchSave = false;
this.settings.liveSync = false;
this.settings.periodicReplication = false;
this.settings.syncOnSave = false;
this.settings.syncOnStart = false;
this.settings.syncOnFileOpen = false;
this.settings.autoSweepPlugins = false;
this.settings.usePluginSync = false;
this.settings.suspendFileWatching = true;
await this.saveSettings();
await this.openDatabase();
const warningMessage = "The red flag is raised! The whole initialize steps are skipped, and any file changes are not captured.";
Logger(warningMessage, LOG_LEVEL.NOTICE);
this.setStatusBarText(warningMessage);
} else {
if (this.settings.suspendFileWatching) {
Logger("'Suspend file watching' turned on. Are you sure this is what you intended? Every modification on the vault will be ignored.", LOG_LEVEL.NOTICE);
if (this.localDatabase.isReady)
try {
if (this.isRedFlagRaised()) {
this.settings.batchSave = false;
this.settings.liveSync = false;
this.settings.periodicReplication = false;
this.settings.syncOnSave = false;
this.settings.syncOnStart = false;
this.settings.syncOnFileOpen = false;
this.settings.autoSweepPlugins = false;
this.settings.usePluginSync = false;
this.settings.suspendFileWatching = true;
await this.saveSettings();
await this.openDatabase();
const warningMessage = "The red flag is raised! The whole initialize steps are skipped, and any file changes are not captured.";
Logger(warningMessage, LOG_LEVEL.NOTICE);
this.setStatusBarText(warningMessage);
} else {
if (this.settings.suspendFileWatching) {
Logger("'Suspend file watching' turned on. Are you sure this is what you intended? Every modification on the vault will be ignored.", LOG_LEVEL.NOTICE);
}
const isInitalized = await this.initializeDatabase();
if (!isInitalized) {
//TODO:stop all sync.
return false;
}
}
await this.initializeDatabase();
await this.realizeSettingSyncMode();
this.registerWatchEvents();
if (this.settings.syncOnStart) {
this.localDatabase.openReplication(this.settings, false, false, this.parseReplicationResult);
}
} catch (ex) {
Logger("Error while loading Self-hosted LiveSync", LOG_LEVEL.NOTICE);
Logger(ex, LOG_LEVEL.VERBOSE);
}
await this.realizeSettingSyncMode();
this.registerWatchEvents();
if (this.settings.syncOnStart) {
this.localDatabase.openReplication(this.settings, false, false, this.parseReplicationResult);
}
} catch (ex) {
Logger("Error while loading Self-hosted LiveSync", LOG_LEVEL.NOTICE);
Logger(ex, LOG_LEVEL.VERBOSE);
}
});
this.addCommand({
id: "livesync-replicate",
@ -204,7 +211,6 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
id: "livesync-dump",
name: "Dump informations of this doc ",
editorCallback: (editor: Editor, view: MarkdownView) => {
this.localDatabase.disposeHashCache();
this.localDatabase.getDBEntry(view.file.path, {}, true, false);
},
});
@ -318,7 +324,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
this.localDatabase.updateInfo = () => {
this.refreshStatusText();
};
await this.localDatabase.initializeDatabase();
return await this.localDatabase.initializeDatabase();
}
async garbageCollect() {
@ -329,6 +335,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
this.settings.workingEncrypt = this.settings.encrypt;
this.settings.workingPassphrase = this.settings.passphrase;
// Delete this feature to avoid problems on mobile.
this.settings.disableRequestURI = true;
const lsname = "obsidian-live-sync-vaultanddevicename-" + this.app.vault.getName();
if (this.settings.deviceAndVaultName != "") {
if (!localStorage.getItem(lsname)) {
@ -422,7 +430,6 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
if (this.settings.syncOnFileOpen && !this.suspended) {
await this.replicate();
}
this.localDatabase.disposeHashCache();
await this.showIfConflicted(file);
this.gcHook();
}
@ -581,7 +588,11 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
notifies: { [key: string]: { notice: Notice; timer: NodeJS.Timeout; count: number } } = {};
// eslint-disable-next-line require-await
lastLog = "";
async addLog(message: any, level: LOG_LEVEL = LOG_LEVEL.INFO) {
if (level == LOG_LEVEL.DEBUG && !isDebug) {
return;
}
if (level < LOG_LEVEL.INFO && this.settings && this.settings.lessInformationInLog) {
return;
}
@ -595,6 +606,10 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
this.logMessage = [].concat(this.logMessage).concat([newmessage]).slice(-100);
console.log(valutName + ":" + newmessage);
this.setStatusBarText(null, messagecontent.substring(0, 30));
if (message instanceof Error) {
console.trace(message);
}
if (level >= LOG_LEVEL.NOTICE) {
if (messagecontent in this.notifies) {
@ -814,6 +829,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
if (change._id.startsWith("h:")) {
continue;
}
if (change._id == SYNCINFO_ID) {
continue;
}
if (change.type != "leaf" && change.type != "versioninfo" && change.type != "milestoneinfo" && change.type != "nodeinfo") {
Logger("replication change arrived", LOG_LEVEL.VERBOSE);
await this.handleDBChanged(change);
@ -956,22 +974,36 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
const procs = getProcessingCounts();
const procsDisp = procs == 0 ? "" : ` ⏳${procs}`;
const message = `Sync:${w} ↑${sent} ↓${arrived}${waiting}${procsDisp}`;
this.setStatusBarText(message);
const locks = getLocks();
const pendingTask = locks.pending.length ? `\nPending:${locks.pending.join(", ")}` : "";
const runningTask = locks.running.length ? `\nRunning:${locks.running.join(", ")}` : "";
this.setStatusBarText(message + pendingTask + runningTask);
}
setStatusBarText(message: string) {
if (this.lastMessage != message) {
this.statusBar.setText(message);
logHideTimer: NodeJS.Timeout = null;
setStatusBarText(message: string = null, log: string = null) {
if (!this.statusBar) return;
const newMsg = typeof message == "string" ? message : this.lastMessage;
const newLog = typeof log == "string" ? log : this.lastLog;
if (`${this.lastMessage}-${this.lastLog}` != `${newMsg}-${newLog}`) {
this.statusBar.setText(newMsg.split("\n")[0]);
if (this.settings.showStatusOnEditor) {
const root = document.documentElement;
root.style.setProperty("--slsmessage", '"' + message + '"');
root.style.setProperty("--slsmessage", '"' + (newMsg + "\n" + newLog).split("\n").join("\\a ") + '"');
} else {
const root = document.documentElement;
root.style.setProperty("--slsmessage", '""');
}
this.lastMessage = message;
if (this.logHideTimer != null) {
clearTimeout(this.logHideTimer);
}
this.logHideTimer = setTimeout(() => this.setStatusBarText(null, ""), 3000);
this.lastMessage = newMsg;
this.lastLog = newLog;
}
}
updateStatusBarText() {}
async replicate(showMessage?: boolean) {
if (this.settings.versionUpFlash != "") {
@ -986,8 +1018,14 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
}
async initializeDatabase(showingNotice?: boolean) {
await this.openDatabase();
await this.syncAllFiles(showingNotice);
if (await this.openDatabase()) {
if (this.localDatabase.isReady) {
await this.syncAllFiles(showingNotice);
}
return true;
} else {
return false;
}
}
async replicateAllToServer(showingNotice?: boolean) {
@ -1428,6 +1466,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
async resetLocalDatabase() {
await this.localDatabase.resetDatabase();
}
async resetLocalOldDatabase() {
await this.localDatabase.resetLocalOldDatabase();
}
async tryResetRemoteDatabase() {
await this.localDatabase.tryResetRemoteDatabase(this.settings);

53
src/utils_couchdb.ts
View File

@ -1,6 +1,6 @@
import { Logger } from "./lib/src/logger";
import { LOG_LEVEL, VER, VERSIONINFO_DOCID, EntryVersionInfo, EntryDoc } from "./lib/src/types";
import { resolveWithIgnoreKnownError } from "./lib/src/utils";
import { LOG_LEVEL, VER, VERSIONINFO_DOCID, EntryVersionInfo, EntryDoc, RemoteDBSettings, SYNCINFO_ID, SyncInfo } from "./lib/src/types";
import { enableEncryption, resolveWithIgnoreKnownError } from "./lib/src/utils";
import { PouchDB } from "./pouchdb-browser";
import { requestUrl, RequestUrlParam, RequestUrlResponse } from "obsidian";
@ -27,7 +27,18 @@ const fetchByAPI = async (request: RequestUrlParam): Promise<RequestUrlResponse>
return ret;
};
export const connectRemoteCouchDB = async (uri: string, auth: { username: string; password: string }, disableRequestURI: boolean): Promise<string | { db: PouchDB.Database<EntryDoc>; info: PouchDB.Core.DatabaseInfo }> => {
export const connectRemoteCouchDBWithSetting = (settings: RemoteDBSettings, isMobile: boolean) =>
connectRemoteCouchDB(
settings.couchDB_URI + (settings.couchDB_DBNAME == "" ? "" : "/" + settings.couchDB_DBNAME),
{
username: settings.couchDB_USER,
password: settings.couchDB_PASSWORD,
},
settings.disableRequestURI || isMobile,
settings.encrypt ? settings.passphrase : settings.encrypt
);
const connectRemoteCouchDB = async (uri: string, auth: { username: string; password: string }, disableRequestURI: boolean, passphrase: string | boolean): Promise<string | { db: PouchDB.Database<EntryDoc>; info: PouchDB.Core.DatabaseInfo }> => {
if (!isValidRemoteCouchDBURI(uri)) return "Remote URI is not valid";
let authHeader = "";
if (auth.username && auth.password) {
@ -82,7 +93,7 @@ export const connectRemoteCouchDB = async (uri: string, auth: { username: string
} else {
last_post_successed = true;
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${r.status}`, LOG_LEVEL.VERBOSE);
Logger(`HTTP:${method}${size} to:${localURL} -> ${r.status}`, LOG_LEVEL.DEBUG);
return new Response(r.arrayBuffer, {
headers: r.headers,
@ -109,7 +120,7 @@ export const connectRemoteCouchDB = async (uri: string, auth: { username: string
} else {
last_post_successed = true;
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${responce.status}`, LOG_LEVEL.VERBOSE);
Logger(`HTTP:${method}${size} to:${localURL} -> ${responce.status}`, LOG_LEVEL.DEBUG);
return responce;
} catch (ex) {
Logger(`HTTP:${method}${size} to:${localURL} -> failed`, LOG_LEVEL.VERBOSE);
@ -124,6 +135,9 @@ export const connectRemoteCouchDB = async (uri: string, auth: { username: string
},
};
const db: PouchDB.Database<EntryDoc> = new PouchDB<EntryDoc>(uri, conf);
if (passphrase && typeof passphrase === "string") {
enableEncryption(db, passphrase);
}
try {
const info = await db.info();
return { db: db, info: info };
@ -179,3 +193,32 @@ export const bumpRemoteVersion = async (db: PouchDB.Database, barrier: number =
await db.put(vi);
return true;
};
export const checkSyncInfo = async (db: PouchDB.Database): Promise<boolean> => {
try {
const syncinfo = (await db.get(SYNCINFO_ID)) as SyncInfo;
console.log(syncinfo);
// if we could decrypt the doc, it must be ok.
return true;
} catch (ex) {
if (ex.status && ex.status == 404) {
const randomStrSrc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
const temp = [...Array(30)]
.map((e) => Math.floor(Math.random() * randomStrSrc.length))
.map((e) => randomStrSrc[e])
.join("");
const newSyncInfo: SyncInfo = {
_id: SYNCINFO_ID,
type: "syncinfo",
data: temp,
};
if (await db.put(newSyncInfo)) {
return true;
}
return false;
} else {
console.dir(ex);
return false;
}
}
};

33
styles.css
View File

@ -1,14 +1,14 @@
.added {
color: black;
background-color: white;
color: var(--text-on-accent);
background-color: var(--text-accent);
}
.normal {
color: lightgray;
color: var(--text-normal);
}
.deleted {
color: white;
background-color: black;
/* text-decoration: line-through; */
color: var(--text-on-accent);
background-color: var(--text-muted);
text-decoration: line-through;
}
.op-scrollable {
overflow-y: scroll;
@ -24,6 +24,25 @@
padding: 2px;
border-radius: 4px;
}
.op-warn::before {
content: "Warning";
font-weight: bold;
color: salmon;
position: relative;
display: block;
}
.op-warn-info {
border: 1px solid rgb(255, 209, 81);
padding: 2px;
border-radius: 4px;
}
.op-warn-info::before {
content: "Notice";
font-weight: bold;
color: rgb(255, 209, 81);
position: relative;
display: block;
}
.syncstatusbar {
-webkit-filter: grayscale(100%);
filter: grayscale(100%);
@ -69,6 +88,8 @@
.CodeMirror-wrap::before,
.cm-s-obsidian > .cm-editor::before {
content: var(--slsmessage);
text-align: right;
white-space: pre-wrap;
position: absolute;
border-radius: 4px;
/* border:1px solid --background-modifier-border; */