A look behind Dense Discovery: creating a fully customised weekly newsletter

Readers frequently send me questions about how I created and run Dense Discovery. Here’s an updated summary of all the tools I use and how I connect them to create my weekly newsletter, including the membership and sponsorship programs.

Email app EmailOctopus

After Campaign Monitor, then MailChimp, then Mailblast, I now use EmailOctopus.

The email platform

I’ve been a happy customer of both MailChimp and Campaign Monitor for several years. They are easy to use and extremely reliable, but once your subscriber base grows to five digits, the monthly bill quickly grows to several hundred dollars. In recent years, both MailChimp and Campaign Monitor have also morphed into marketing rather than publishing platforms, with a clear focus on generating sales. As a publisher of a weekly newsletter I have no use for sales-driven features, so I made the move to a smaller competitor.

There are tons of options to choose from. After some research I first settled on Mailblast and more recently moved to EmailOctopus. Both use Amazon’s SES infrastructure to send emails cheaply. Mailblast served me well for about two years, but I ended up changing to EmailOctopus for its integration with Zapier, basic segmentation features, and being able to use conditional content. (A big shoutout and thanks to the EmailOctopus support crew, who were open to listening to my feedback and kindly extended their Zapier features!)

While Substack seems to work great for a lot of people, I decided against it for the same reasons that I don’t like publishing on Medium: a limiting set of features and the lack of design flexibility makes every newsletter ‘feel’ the same. I also prefer to own my content. As I describe below, it’s refreshing to be able to control of your own public archive and customise the entire experience for subscribers.

Email template Antwort

The open-source email template Antwort provides the basic table structure I use for Dense Discovery.

A responsive template

Many email composers, especially MailChimp’s editor, inject a lot of markup to ensure emails looks decent in every possible client. With so much redundant markup, long emails quickly hit Gmail’s size limit and get truncated. Knowing that Dense Discovery emails were going to be quite long, I had to create my own, light-weight template and optimise the markup by removing redundant CSS for outdated email clients.

I went looking for a barebone, responsive email template and found one in Antwort. To ensure DD looks half-decent on less popular clients, I used the very handy Litmus app to do some cross-browser, cross-platform testing and tweaking.

Manually coding each issue

Using this custom email template, I compose each issue offline in my HTML editor. This isn’t the most visually elegant workflow but it has several advantages:

I can use a few simple PHP snippets inside the email markup to reduce complexity. For example, I use PHP to detect the name/number of the folder the file sits in (which is the issue number) and store it in a variable <?=$issue?>. I can then use this number throughout the file. For instance, I can generate the UTM code for each outgoing link like so:

?utm_source=densediscovery&utm_medium=email&utm_campaign=newsletter-issue-<?=$issue?>

I also use PHP to create conditional sections that only show up in the archive version of that issue. For example, the comment section shows up under each issue in the archive version but not in the actual email I send out, even though the markup of both comes from the same file.

Once I’m ready to send an issue, I ‘parse’ the PHP, then copy the HTML and run it through a CSS inliner tool. This injects the CSS styling into the HTML elements which makes the markup pretty messy but is important for the email to work across different clients. I then transfer this inlined version to EmailOctopus where I paste it into a new campaign and the issue is ready to send.

Before I send the campaign out, I upload the online version of the email to the archive, i.e. my server. Doing this first is important because I host all images used in the email on my own server. So whenever you open an issue of Dense Discovery in your email client, the images you see are being pulled straight from the online/archival version of that email. (I use Cloudflare to lessen the load on my server.)

The Dense Discovery archive

All previous issues can be accessed through an online archive.

A custom archive

It’s a shame that most email platforms only offer an archive feature that feels like an afterthought. Being able to browse through past issues of a newsletter is a great way to entice new subscribers (unless, of course, all you do is send out marketing emails). A lot of work goes into each issue of Dense Discovery, so being able to preserve and showcase older issues was one of the many reasons for going with a custom approach.

On my server, each issue sits inside a folder (its name being the issue number) that contains all images (compressed/minimised using ImageOptim) and a PHP file.

The archive page itself uses a bit of PHP to automatically look for the folder with the highest number – i.e. the latest issue – and then displays the file in that folder by default. Easy.

One of the benefits of having full control over the archive is that I can make changes even after the email has been sent. Of course, this doesn’t change the content of the email itself (unless it’s about replacing an image), but at least I can fix minor issues such as a broken link or a typo and provide an error-free version for posterity.

I get quite lot of responses to my newsletter, often with related link suggestions and further reading material. To make comments and suggestions accessible for everyone, I recently added a comment section to each issue. I use Commento for this, a very minimal, JS-based commenting tool.

The Dense Discovery sign-up page

I created a very minimal sign-up page with a basic captcha to help fend off bots.

A simple landing page

For the actually Dense Discovery website, I chose a very minimal look with a clear focus on the sign-up field that’s connected to EmailOctopus’ API.

Besides using the double opt-in method and a simple Honeypot in the form, I also implemented a captcha script to filter out bots. The captcha is shown based on certain rules I can set in the code. I’m surprised about how persistent spammers are, because even with a captcha like this, I regularly get fake signups using email addresses like ‘[email protected]’. I don’t think they are bots because these sign-ups often occur in small batches of 10–20.

For even more control, I recently also added an IP blocker that allows me to block individual repeat offenders from being able to submit more email addresses.

Booking an ad in Dense Discovery

I use a set of Google Spreadsheets connected to Formsite through Zapier to manage ad bookings.

A DIY ad manager

I offer two different types of advertising in Dense Discovery: one main sponsor slot and four plain-text classified ads at the bottom.

To automate and manage the bookings I hacked together a bunch of apps through Zapier. It works like this:

  • On a Google Spreadsheet I list all available slots.
  • When an advertiser clicks on a slot, they are referred to a Formsite form with some data like dates and spreadsheet rows being transmitted through the URL. They enter their details and book the slot.
  • The successful payment kicks off a Zap that automatically adds the booking to another Google Spreadsheet (where I manage the content of all ads) and it updates the original spreadsheet to show that particular slot as booked.

I’m really happy with this workflow. It’s been an absolute pleasure using Zapier to automate all the steps. So much that I decided to also use it for my membership program:

Dense Discovery’s membership program ‘Friends of DD’

Friends of DD help keep DD going and ensure it remains accessible to everyone.

An (optional) membership program

To become less depended on advertising, I recently decided to add a membership program called ‘Friends of DD’. At the moment, paid members are supporting me with a small yearly amount to help keep the newsletter freely accessible. There aren’t many other perks. However, I set up this system so that I can easily send out members-only campaigns or even show specific sections of the newsletter to members only.

A lot of newsletter memberships run on Memberful. It looks like a great tool, but I’m only charging a modest $20/year and don’t expect thousands of members, so paying $25/month plus 4.9% of each transaction is quite expensive. I found a cheaper alternative in Moonclerk which lacks many features compared to other membership apps but it offers everything I need to make it work.

Friends of DD sign up on my website through the embedded Moonclerk form. After successful payment of the annual membership fee, Moonclerk sends an email to Zapier with all the necessary details. Zapier then parses those details and triggers the following actions: it sends out a nicer looking ‘welcome’ email to new members (Moonclerk itself only sends basic plain-text emails) and it adds a custom field with a timestamp to the subscriber on my EmailOctopus list.

There is a surprising amount of complexity in managing an email list on one platform (EmailOctopus) and members on another (Moonclerk). One of the trickier issues: if a paying member signs up for an annual subscription today, but then cancels their membership two weeks later, they remain a member for 50 more weeks. Only after a full year is their membership status revoked. Making this work was a lot trickier than I thought, but I eventually found a nice little hack:

I convert the time of the payment to Unix time, then add one year, also in Unix time. This timestamp now gives me the exact time (365 days after first signing up) when the paid membership expires. By adding that timestamp as a custom field to the subscriber on EmailOctopus, I can now show content in the newsletter using conditional if/else statements. For instance, if the timestamp is after today’s date (1596758400 in Unix time), it means the subscriber is active:

{% if timestamp > 1596758400 %}
show member-only content
{% else %}
show normal content
{% endif %}

Whenever the membership is renewed (the next payment occurs) the process repeats itself and the timestamp is updated.

I know this setup seems complicated and relies on Zapier to work properly, but once it’s running it really doesn’t require much maintenance. Figuring out how to make this process work was actually a lot of fun and reminded me how powerful Zapier really is.

Tools summary

To sum up, here’s a quick overview of all the tools I’m using to run Dense Discovery:

To send out the newsletter and manage subscribers. The actual emails are sent through Amazon SES, the only Amazon product I still reluctantly use.

Atom →

My preferred code editor to work on each issue. I use a few plugins to make my life easier, such as Emmet.

I’m not proud of still using Google, but their email, calendar and Drive products are unmatched in their reliability and are widely integrated in other tools like Zapier. I use Google Spreadsheets to manage Dense Discovery’s ads.

Formsite →

I use two different Formsite forms to collect details and payment from advertisers. (Payments through Stripe, of course.)

Moonclerk →

Moonclerk makes it easy to start and manage yearly subscriptions. After signing up, Members get access to a ‘payer portal’ where they can self-manage their subscription.

Zapier →

Zapier does all the automation under the hood: it connects Formsite with Google Docs and Moonclerk with EmailOctopus.

Cloudflare →

To lessen the load on my own server I use Cloudflare as my CDN – quite handy with those chunky GIFs.

Commento →

A simple commenting tool that I embed under each issue in my archive.

Last updated: September 2020

← All Notes Home