refs https://ghost.slack.com/archives/CTH5NDJMS/p1699359241142969
It's possible for `ObjectIDs` to have only numeric characters. We were
previously letting the type be inferred, which created a very rare but
possible edge case where the last recipient of an email batch had a
numeric ObjectID, resulting in a numeric comparison against alphanumeric
`ObjectIDs` in the database.
- updated the filter to add `'`'s around the `lastId` parameter
- updated tests to check for the type of the id filter parameter value
- can't fully test for numeric object IDs using what we have because
javascript cannot handle numerics of that size; may be able to look at
using fixture data loaded directly into the db
refs https://github.com/TryGhost/Product/issues/4053
This adds the feature flag. If enabled, the list-unsubscribe header
should be set. The value currently is only for testing purposes and
probably won't work yet.
fixes https://github.com/TryGhost/Product/issues/3687
After this change, relative URLs in emails will be replaced with
absolute URLs using the post URL. Making relative Portal URLs possible
etc.
Updates the test data generator to fix invalid URL encoding (somehow a
backslash + escaped double quote was added when it wasn't required).
refs https://github.com/TryGhost/Product/issues/3651
- This is a security fix that addresses an issue causing malicious users
to abuse the test / preview email API endpoint.
- We have multiple procedures in place now to limit such users.
- First, we now only allow one email address to be passed into the
`sendTestEmail` method. This method only have one purpose, which is to
compliment the test email functionality within the Editor in Admin and
therefore have no reason to send to more than one email address at a
time.
- We then add an additional rate limiter to prevent a user from making
multiple requests, eg via a script.
- The new imposed limit is 10 test emails per hour.
refs TryGhost/Product#3647
- The latest version of juice (which Ghost uses to inline css in email
newsletters) included new functionality to add height="auto" and
width="auto" for any images with dimensions set to auto in css
- This was causing rendering issues in Outlook, which would render the
image at full width, which often added a horizontal scroll and generally
messed up the flow of the document
- This change prevents juice from modifying the height or width of `<img
/>` tags
fixes https://github.com/TryGhost/Team/issues/3541
The email preheader, which is only present in the html version of an
email, is also included in the plaintext version of all emails. This
results in all text being duplicated twice in plaintext emails.
fixes https://github.com/TryGhost/Ghost/issues/16131
Members only content was incorrectly being shown in a plaintext email
due to the email `preheader` using the post model `plaintext` field
directly (which contained the members-only content). This changes this
behaviour so that the post html content is utilised for the `preheader`
but has all members-only content (post-preview content + segmented
content) removed
refs https://github.com/TryGhost/Ghost/issues/15725
This pull request adds a new configuration option for the Mailgun email
provider that allows the user to set the maximum number of recipients
per email batch via a new config option `bulkEmail.batchSize`
refs: https://github.com/TryGhost/Toolbox/issues/595
We're rolling out new rules around the node assert library, the first of which is enforcing the use of assert/strict. This means we don't need to use the strict version of methods, as the standard version will work that way by default.
This caught some gotchas in our existing usage of assert where the lack of strict mode had unexpected results:
- Url matching needs to be done on `url.href` see aa58b354a4
- Null and undefined are not the same thing, there were a few cases of this being confused
- Particularly questionable changes in [PostExporter tests](c1a468744b) tracked [here](https://github.com/TryGhost/Team/issues/3505).
- A typo see eaac9c293a
Moving forward, using assert strict should help us to catch unexpected behaviour, particularly around nulls and undefineds during implementation.
refs 551532f874
refs https://github.com/TryGhost/Team/issues/3324
- After analyzing data dumps, the data revealed that we have extra data from a stray batch. The filtering logic manually filters out the data to the recipients that belong to a "current batch".
- Hunting down the root cause of the data mixup proved to be too expensive of an investigation, so this is a "good enough patch" to deal with the problem.
- Most likely cause is the concurrent batch sending, but reducing the concurrency would be too expensive of a performance price to pay instead of filtering the data rarely.
closes https://github.com/TryGhost/Team/issues/3324
- When the recipients batch size is larger than the limit in addition to logging the error we need extra data to figure out what exactly is inside those `2000` or `3000` records causing faulty behavior.
- This change grabs all available models and dumps them into a file inside of the `content/data` folder. The code is temporary and should be removed once the problem is narrowed down
refs TryGhost/Team#3229
- The issue we are observing that even though the returned amount of email recipients should not ever accede the max batch size (1000 in case of MailGun), there are rare glitches when this number is doubled and we fetch 2000 records instead.
- The fix takes it's best guess in de-duping data in the batch and then truncates it if the amount of records is still above the threshold. This ensures we at least end up sending the emails out to some of the recipients instead of none.
As discussed with the product team we want to enforce kebab-case file names for
all files, with the exception of files which export a single class, in which
case they should be PascalCase and reflect the class which they export.
This will help find classes faster, and should push better naming for them too.
Some files and packages have been excluded from this linting, specifically when
a library or framework depends on the naming of a file for the functionality
e.g. Ember, knex-migrator, adapter-manager
refs TryGhost/Team#2974
- currently the unsubscribeFromNewsletters event is failing with 'member
not found' in elastic
- this change catches the error and logs it, which should allow the rest
of the event(s) to be processed
refs https://github.com/TryGhost/Team/issues/2671
The inline style display: none; isn't applied to the images in Outlook for some reason. This change manually removes the images in the backend.
refs https://github.com/TryGhost/Team/issues/2845
Ideally the calculation of these values would be handled by a Newsletter entity
but we don't have one yet, we can look to fix this if we have time. For now
we're calculating them in separate methods to make it easier to extract in
future.
no issue
Bookshelf by default returns an empty model when requesting .related('email') for a post without an email. So we need to be a bit smarter to know if a post has an email or not. This fixed an issue where we always showed 'published and emailed' instead of 'published only'.
Since this change also included some changes to test helpers, it also made some changes to the email service because coverage dropped below 100% as a result of fixing the .related method mocking. Ideally we want to move test test helpers to a seperate package in the future.
Refs https://github.com/TryGhost/Team/issues/2801
- It was not possible to click latest post links in Outlook due to <a>
tag wrapping around a table
- The post meta data wouldn't display properly when centered in Outlook
---------
Co-authored-by: Simon Backx <simon@ghost.org>
refs https://github.com/TryGhost/Team/issues/2805
When we render mobiledoc to HTML, it automatically escapes HTML entities in the process, so a button or directly pasted link with href="https://example.com?code=test" will be rendered as href="https://example.com?code=test" as the url is encoded in the rendered HTML. Our link tracking was using the encoded URL as the redirect URL in newsletters, causing certain links to break.
This change updates the link tracking to decode the URL with `entities.decode(url)` so we store the correct redirect URL in our DB and ensure link tracking redirects to the correct url from newsletters.
---------
Co-authored-by: Rishabh <zrishabhgarg@gmail.com>
refs https://github.com/TryGhost/Team/issues/2674
- The segment detection doesn't work outside the main post content. So the data-gh-segment attribute didn't work. It is now replaced with just a simple email replacement that is empty for a free member.
- Fixed that a trialing member was shown as 'paid'. This is now replaced with 'trialing'.
This commit also includes E2E tests for a couple of member statusses.
refs https://github.com/TryGhost/Team/issues/2736
If the name is not known for a member, we'll hide the name row in the subscription details in an email. This method is supported in most email clients, and requires the support of `<style>` in `<head>`.
fixes https://github.com/TryGhost/Team/issues/2672
- Moves the feature from behind the flag
- Also hides the comment CTA for email only posts
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
fixes https://github.com/TryGhost/Team/issues/2725
- Added to emails if labs flag enabled, comments enabled and comment CTA button enabled
- Links to comment section
- Design and styling not added yet
fixes https://github.com/TryGhost/Team/issues/2705
- Added showPostTitleSection to newsletter model in admin
- Wired up UI to admin model so it saves to the database
- Implemented showPostTitleSection in newsletter preview and added some
minor temporary css styling
- Implemented showPostTitleSection in newsletter template in backend,
and added some extra CSS styling to fix spacing
fixes https://github.com/TryGhost/Team/issues/2560
When an email fails, and you reschedule the post, the error dialog was
shown (from the previous try). The retry button on that page allowed you
to retry sending the email immediately, which could be very confusing.
- The email error dialog is no longer shown for scheduled emails
- The email status is no longer polled for scheduled emails
- Retrying an email is not possible via the API if the post status is
not published or sent
- Added some extra snapshot tests
- When retrying an email, we immediately update the email status to
'pending' to have a better API response (instead of still returning
failed).
- Disabled email sending retrying in development (otherwise very hard to
test failed emails if it takes 10 mins before it gives up automatic
retrying)
fixes https://github.com/TryGhost/Team/issues/2683
When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.
This commit fixes the issue and adds some tests to make sure it doesn't
happen again.
The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
fixes https://github.com/TryGhost/Team/issues/2433
- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
fixes https://github.com/TryGhost/Team/issues/2557
When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
no issue
- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
fixes https://github.com/TryGhost/Team/issues/2522
When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.
The cache is short lived. After sending the email it is automatically
garbage collected.
