Removed bthreads dependency in favor of native worker_threads
fixes https://github.com/TryGhost/Toolbox/issues/370 - we no longer need `bthreads` because we can use native `worker_threads` now we don't have to support Node 10 any longer - this allows us to clean up a dependency and stick with native libraries - the referenced node-sqlite3 issue should be fixed (or at least, we now maintain it so we can fix it if not)
This commit is contained in:
Daniel Lockyer
parent
1d06439633
commit
c2e45b657f
@ -31,10 +31,6 @@ function configure(dbConfig) {
|
||||
}
|
||||
};
|
||||
|
||||
// Force bthreads to use child_process backend until a worker_thread-compatible version of sqlite3 is published
|
||||
// https://github.com/mapbox/node-sqlite3/issues/1386
|
||||
process.env.BTHREADS_BACKEND = 'child_process';
|
||||
|
||||
// In the default SQLite test config we set the path to /tmp/ghost-test.db,
|
||||
// but this won't work on Windows, so we need to replace the /tmp bit with
|
||||
// the Windows temp folder
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
const {parentPort} = require('bthreads');
|
||||
const {parentPort} = require('worker_threads');
|
||||
|
||||
const postParentPortMessage = (message) => {
|
||||
if (parentPort) {
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
const {parentPort} = require('bthreads');
|
||||
const {parentPort} = require('worker_threads');
|
||||
const debug = require('@tryghost/debug')('jobs:email-analytics:fetch-latest');
|
||||
|
||||
// recurring job to fetch analytics since the most recently seen event timestamp
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
const {parentPort} = require('bthreads');
|
||||
const {parentPort} = require('worker_threads');
|
||||
const util = require('util');
|
||||
|
||||
let shutdown = false;
|
||||
|
||||
@ -123,7 +123,6 @@
|
||||
"bookshelf-relations": "2.4.0",
|
||||
"brute-knex": "4.0.1",
|
||||
"bson-objectid": "2.0.3",
|
||||
"bthreads": "0.5.1",
|
||||
"chalk": "4.1.2",
|
||||
"cheerio": "0.22.0",
|
||||
"common-tags": "1.8.2",
|
||||
|
||||
@ -82,7 +82,7 @@ For more examples of JobManager initialization check [test/examples](https://git
|
||||
|
||||
There are two types of jobs distinguished based on purpose and environment they run in:
|
||||
- **"inline"** - job which is run in the same even loop as the caller. Should be used in situations when there is no even loop blocking operations and no need to manage memory leaks in sandboxed way. Sometimes
|
||||
- **"offloaded"** - job which is executed in separate to caller's event loop. For Node >v12 clients it spawns a [Worker thread](https://nodejs.org/dist/latest-v12.x/docs/api/worker_threads.html#worker_threads_new_worker_filename_options), for older Node runtimes it is executed in separate process through [child_process](https://nodejs.org/docs/latest-v10.x/api/child_process.html). Comparing to **inline** jobs, **offloaded** jobs are safer to execute as they are run on a dedicated thread (or process) acting like a sandbox. These jobs also give better utilization of multi-core CPUs. This type of jobs is useful when there are heavy computations needed to be done blocking the event loop or need a sandboxed environment to run in safely. Example jobs would be: statistical information processing, memory intensive computations (e.g. recursive algorithms), processing that requires blocking I/O operations etc.
|
||||
- **"offloaded"** - job which is executed in separate to caller's event loop. Comparing to **inline** jobs, **offloaded** jobs are safer to execute as they are run on a dedicated thread (or process) acting like a sandbox. These jobs also give better utilization of multi-core CPUs. This type of jobs is useful when there are heavy computations needed to be done blocking the event loop or need a sandboxed environment to run in safely. Example jobs would be: statistical information processing, memory intensive computations (e.g. recursive algorithms), processing that requires blocking I/O operations etc.
|
||||
- **"one-off"** - job that would only ever run once. It is persisted in storage keeping the record of the job state between process restarts. One-off jobs can be of both "inline" and "offloaded" types. They do not support scheduled recurring execution as that's against the nature of being *one-off*. Apart from being persisted one-off jobs can be rescheduled for execution in case they have a "failed" execution state.
|
||||
|
||||
Job manager's instance registers jobs through `addJob` and `addOneOffJob` methods. The `offloaded` parameter controls if the job is **inline** (executed in the same event loop) or is **offloaded** (executed in worker thread/separate process). By default `offloaded` is set to `true` - creates an "offloaded" job.
|
||||
@ -108,18 +108,3 @@ Offloaded jobs are running on dedicated worker threads which makes their lifecyc
|
||||
4. When **exceptions** happen and expected outcome is to terminate current job, leave the exception unhandled allowing it to bubble up to the job manager. Unhandled exceptions [terminate current thread](https://nodejs.org/dist/latest-v14.x/docs/api/worker_threads.html#worker_threads_event_error) and allow for next scheduled job execution to happen.
|
||||
|
||||
For more nuances on job structure best practices check [bree documentation](https://github.com/breejs/bree#writing-jobs-with-promises-and-async-await).
|
||||
|
||||
### Offloaded job script quirks
|
||||
|
||||
⚠️ to ensure worker thread back compatibility and correct inter-thread communication use [btrheads](https://github.com/chjj/bthreads) polyfill instead of native [worker_threads](https://nodejs.org/api/worker_threads.html#worker_threads) module in job scripts.
|
||||
|
||||
Instead of:
|
||||
```js
|
||||
const {isMainThread, parentPort} = require('worker_threads');
|
||||
```
|
||||
use
|
||||
```js
|
||||
const {isMainThread, parentPort} = require('bthreads');
|
||||
```
|
||||
|
||||
It should be possible to use native `worker_threads` module once Node v10 [hits EOL](https://nodejs.org/en/about/releases/) (2021-04-30).
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
/* eslint-disable no-console */
|
||||
|
||||
const setTimeoutPromise = require('util').promisify(setTimeout);
|
||||
const {isMainThread, parentPort} = require('bthreads');
|
||||
const {isMainThread, parentPort} = require('worker_threads');
|
||||
|
||||
let shutdown = false;
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
const {parentPort} = require('bthreads');
|
||||
const {parentPort} = require('worker_threads');
|
||||
|
||||
setInterval(() => { }, 10);
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
const {isMainThread, parentPort, workerData} = require('bthreads');
|
||||
const {isMainThread, parentPort, workerData} = require('worker_threads');
|
||||
const util = require('util');
|
||||
const setTimeoutPromise = util.promisify(setTimeout);
|
||||
|
||||
|
||||
@ -6780,7 +6780,7 @@ bson-objectid@2.0.3:
|
||||
resolved "https://registry.yarnpkg.com/bson-objectid/-/bson-objectid-2.0.3.tgz#d840185172846b2f10c42ce2bcdb4a50956a9db5"
|
||||
integrity sha512-WYwVtY9yqk179EPMNuF3vcxufdrGLEo2XwqdRVbfLVe9X6jLt7WKZQgP+ObOcprakBGbHxzl76tgTaieqsH29g==
|
||||
|
||||
bthreads@0.5.1, bthreads@^0.5.1:
|
||||
bthreads@^0.5.1:
|
||||
version "0.5.1"
|
||||
resolved "https://registry.yarnpkg.com/bthreads/-/bthreads-0.5.1.tgz#c7a4dacc2d159c50de08b37b1e2a7da836171063"
|
||||
integrity sha512-nK7Jo9ll+r1FRMNPWEFRTZMQrX6HhX8JjPAofxmbTNILHqWVIJPmWzCi9JlX/K0DL5AKZTFZg2Qser5C6gVs9A==
|
||||
@ -10309,7 +10309,6 @@ ember-power-calendar@^0.16.3:
|
||||
|
||||
ember-power-datepicker@cibernox/ember-power-datepicker:
|
||||
version "0.8.1"
|
||||
uid da580474a2c449b715444934ddb626b7c07f46a7
|
||||
resolved "https://codeload.github.com/cibernox/ember-power-datepicker/tar.gz/da580474a2c449b715444934ddb626b7c07f46a7"
|
||||
dependencies:
|
||||
ember-basic-dropdown "^3.0.11"
|
||||
@ -12644,7 +12643,6 @@ globrex@^0.1.2:
|
||||
|
||||
"google-caja-bower@https://github.com/acburdine/google-caja-bower#ghost":
|
||||
version "6011.0.0"
|
||||
uid "275cb75249f038492094a499756a73719ae071fd"
|
||||
resolved "https://github.com/acburdine/google-caja-bower#275cb75249f038492094a499756a73719ae071fd"
|
||||
|
||||
got@9.6.0:
|
||||
@ -14815,7 +14813,6 @@ keygrip@~1.1.0:
|
||||
|
||||
"keymaster@https://github.com/madrobby/keymaster.git":
|
||||
version "1.6.3"
|
||||
uid f8f43ddafad663b505dc0908e72853bcf8daea49
|
||||
resolved "https://github.com/madrobby/keymaster.git#f8f43ddafad663b505dc0908e72853bcf8daea49"
|
||||
|
||||
keypair@1.0.4:
|
||||
@ -16583,7 +16580,6 @@ mocha@^2.5.3:
|
||||
|
||||
mock-knex@TryGhost/mock-knex#8ecb8c227bf463c991c3d820d33f59efc3ab9682:
|
||||
version "0.4.9"
|
||||
uid "8ecb8c227bf463c991c3d820d33f59efc3ab9682"
|
||||
resolved "https://codeload.github.com/TryGhost/mock-knex/tar.gz/8ecb8c227bf463c991c3d820d33f59efc3ab9682"
|
||||
dependencies:
|
||||
bluebird "^3.4.1"
|
||||
@ -20539,7 +20535,6 @@ simple-update-notifier@^1.0.7:
|
||||
|
||||
"simplemde@https://github.com/kevinansfield/simplemde-markdown-editor.git#ghost":
|
||||
version "1.11.2"
|
||||
uid "4c39702de7d97f9b32d5c101f39237b6dab7c3ee"
|
||||
resolved "https://github.com/kevinansfield/simplemde-markdown-editor.git#4c39702de7d97f9b32d5c101f39237b6dab7c3ee"
|
||||
|
||||
sinon@14.0.0:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user