Ghost/ghost/email-service/lib/email-event-processor.js

237 lines
7.8 KiB
JavaScript
Raw Normal View History

const {EmailDeliveredEvent, EmailOpenedEvent, EmailBouncedEvent, SpamComplaintEvent, EmailUnsubscribedEvent, EmailTemporaryBouncedEvent} = require('@tryghost/email-events');
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
async function waitForEvent() {
return new Promise((resolve) => {
setTimeout(resolve, 70);
});
}
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
/**
* @typedef EmailIdentification
* @property {string} email
* @property {string} providerId
* @property {string} [emailId] Optional email id
*/
/**
* @typedef EmailRecipientInformation
* @property {string} emailRecipientId
* @property {string} memberId
* @property {string} emailId
*/
/**
* @typedef EmailEventStorage
* @property {(event: EmailDeliveredEvent) => Promise<void>} handleDelivered
* @property {(event: EmailOpenedEvent) => Promise<void>} handleOpened
* @property {(event: EmailBouncedEvent) => Promise<void>} handlePermanentFailed
* @property {(event: EmailTemporaryBouncedEvent) => Promise<void>} handleTemporaryFailed
* @property {(event: EmailUnsubscribedEvent) => Promise<void>} handleUnsubscribed
* @property {(event: SpamComplaintEvent) => Promise<void>} handleComplained
*/
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
/**
* WARNING: this class is used in a separate thread (an offloaded job). Be careful when working with settings and models.
*/
class EmailEventProcessor {
#domainEvents;
#db;
#eventStorage;
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
constructor({domainEvents, db, eventStorage}) {
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
this.#domainEvents = domainEvents;
this.#db = db;
this.#eventStorage = eventStorage;
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
// Avoid having to query email_batch by provider_id for every event
this.providerIdEmailIdMap = {};
}
/**
* @param {EmailIdentification} emailIdentification
* @param {Date} timestamp
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handleDelivered(emailIdentification, timestamp) {
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = EmailDeliveredEvent.create({
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
email: emailIdentification.email,
emailRecipientId: recipient.emailRecipientId,
memberId: recipient.memberId,
emailId: recipient.emailId,
timestamp
});
await this.#eventStorage.handleDelivered(event);
this.#domainEvents.dispatch(event);
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
}
return recipient;
}
/**
* @param {EmailIdentification} emailIdentification
* @param {Date} timestamp
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handleOpened(emailIdentification, timestamp) {
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = EmailOpenedEvent.create({
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
email: emailIdentification.email,
emailRecipientId: recipient.emailRecipientId,
memberId: recipient.memberId,
emailId: recipient.emailId,
timestamp
});
this.#domainEvents.dispatch(event);
Implemented email analytics retrying (#16273) fixes https://github.com/TryGhost/Team/issues/2562 New event fetching loops: - Reworked the analytics fetching algorithm. Instead of starting again where we stopped during the last fetching minus 30 minutes, we now just continue where we stopped. But with ms precision (because no longer database dependent after first fetch), and we stop at NOW - 1 minute to reduce chance of missing events. - Apart from that, a missing fetching loop is introduced. This fetches events that are older than 30 minutes, and just processes all events a second time to make sure we didn't skip any because of storage delays in the Mailgun API. - A new scheduled fetching loop, that allows us to schedule between a given start/end date (currently only persisted in memory, so stops after a reboot) UI and endpoint changes: - New UI to show the state of the analytics 'loops' - New endpoint to request the analytics loop status - New endpoint to schedule analytics - New endpoint to cancel scheduled analytics - Some number formatting improvements, and introduction of 'opened' count in debug screen - Live reload of data in the debug screen Other changes: - This also improves the support for maxEvents. We can now stop a fetching loop after x events without worrying about lost events. This is used to reduce the fetched events in the missing and scheduled event loop (e.g. when the main one is fetching lots of events, we skip the other loops). - Prevents fetching the same events over and over again if no new events come in (because we always started at the same begin timestamp). The code increases the begin timestamp with 1 second if it is safe to do so, to prevent the API from returning the same events over and over again. - Some optimisations in handing the processing results (less merges to reduce CPU usage in cases we have lots of events). Testing: - You can test with lots of events using the new mailgun mocking server (Toolbox repo `scripts/mailgun-mock-server`). This can also simulate events that are only returned after x minutes because of storage delays.
2023-02-20 18:44:13 +03:00
await this.#eventStorage.handleOpened(event);
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
}
return recipient;
}
/**
* @param {EmailIdentification} emailIdentification
* @param {{id: string, timestamp: Date, error: {code: number; message: string; enhandedCode: string|number} | null}} event
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handleTemporaryFailed(emailIdentification, {timestamp, error, id}) {
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = EmailTemporaryBouncedEvent.create({
id,
error,
email: emailIdentification.email,
memberId: recipient.memberId,
emailId: recipient.emailId,
emailRecipientId: recipient.emailRecipientId,
timestamp
});
await this.#eventStorage.handleTemporaryFailed(event);
this.#domainEvents.dispatch(event);
}
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
return recipient;
}
/**
* @param {EmailIdentification} emailIdentification
* @param {{id: string, timestamp: Date, error: {code: number; message: string; enhandedCode: string|number} | null}} event
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handlePermanentFailed(emailIdentification, {timestamp, error, id}) {
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = EmailBouncedEvent.create({
id,
error,
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
email: emailIdentification.email,
memberId: recipient.memberId,
emailId: recipient.emailId,
emailRecipientId: recipient.emailRecipientId,
timestamp
});
await this.#eventStorage.handlePermanentFailed(event);
this.#domainEvents.dispatch(event);
await waitForEvent(); // Avoids knex connection pool to run dry
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
}
return recipient;
}
/**
* @param {EmailIdentification} emailIdentification
* @param {Date} timestamp
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handleUnsubscribed(emailIdentification, timestamp) {
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = EmailUnsubscribedEvent.create({
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
email: emailIdentification.email,
memberId: recipient.memberId,
emailId: recipient.emailId,
timestamp
});
await this.#eventStorage.handleUnsubscribed(event);
this.#domainEvents.dispatch(event);
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
}
return recipient;
}
/**
* @param {EmailIdentification} emailIdentification
* @param {Date} timestamp
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
*/
async handleComplained(emailIdentification, timestamp) {
const recipient = await this.getRecipient(emailIdentification);
if (recipient) {
const event = SpamComplaintEvent.create({
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
email: emailIdentification.email,
memberId: recipient.memberId,
emailId: recipient.emailId,
timestamp
});
await this.#eventStorage.handleComplained(event);
this.#domainEvents.dispatch(event);
await waitForEvent(); // Avoids knex connection pool to run dry
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
}
return recipient;
}
/**
* @private
* @param {EmailIdentification} emailIdentification
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
* @returns {Promise<EmailRecipientInformation|undefined>}
*/
async getRecipient(emailIdentification) {
if (!emailIdentification.emailId && !emailIdentification.providerId) {
// Protection if both are null or undefined
return;
}
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
// With the provider_id and email address we can look for the EmailRecipient
const emailId = emailIdentification.emailId ?? await this.getEmailId(emailIdentification.providerId);
if (!emailId) {
// Invalid
return;
}
const {id: emailRecipientId, member_id: memberId} = await this.#db.knex('email_recipients')
.select('id', 'member_id')
.where('member_email', emailIdentification.email)
.where('email_id', emailId)
.first() || {};
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
if (emailRecipientId && memberId) {
return {
emailRecipientId,
memberId,
emailId
};
}
}
/**
* @private
* @param {string} providerId
Added new email event processor (#15879) fixes https://github.com/TryGhost/Team/issues/2310 This moves the processing of the events from the event-processor to a new email-event-processor in the email-service package. - The `EmailEventProcessor` only translates events from providerId/emailId to their known emailId, memberId and recipientId, and dispatches the corresponding events. - Since `EmailEventProcessor` runs in a separate worker thread, we can't listen for the dispatched events on the main thread. To accomplish this communication, the events dispatched from the `EmailEventProcessor` class are 'posted' via the postMessage method and redispatched on the main thread. - A new `EmailEventStorage` class reacts to the email events and stores it in the database. This code mostly corresponds to the (now deleted) subclass of the old `EmailEventProcessor` - Updating a members last_seen_at timestamp has moved to the lastSeenAtUpdater. - Email events no longer store `ObjectID` because these are not encodable across threads via postMessage - Includes new E2E tests that test the storage of all supported Mailgun events. Note that in these tests we run the processing on the main thread instead of on a separate thread (couldn't do this because stubbing is not possible across threads) There are some missing pieces that will get added in later PRs (this PR focuses on porting the existing functionality): - Handling temporary failures/bounces - Capturing the error messages of bounce events
2022-11-29 13:15:19 +03:00
* @returns {Promise<string|undefined>}
*/
async getEmailId(providerId) {
if (this.providerIdEmailIdMap[providerId]) {
return this.providerIdEmailIdMap[providerId];
}
const {emailId} = await this.#db.knex('email_batches')
.select('email_id as emailId')
.where('provider_id', providerId)
.first() || {};
if (!emailId) {
return;
}
this.providerIdEmailIdMap[providerId] = emailId;
return emailId;
}
}
module.exports = EmailEventProcessor;