July 12, 2024: All About our Emails and Lists, Lessons Learned
We have managed email, both direct and list, for the thirty years we've been in existence. Initially we distributed avalanche advisories for many US Forest Service centers, the Canadians, and the Swiss. Nobody else knew how to do it. We used majordomo, which had no web interface built it. (The web was new at the time anyway.) Later the government people figured things out and there was no need for us to do that. We tried using mailman, a Python program, but it was not very smooth. Then, until recently, we used a commercial service called MadMimi. This seemed to work ok, although we only now found some problems with it. And it's not free, although we kept our mailings below the initial free allotment.
Over the past few months we have implemented an in-house mail system again, with a robust web management interface. We are now using that, and in the coming season we will be able to make regular mailings with no per-message cost. But this transition has been an adventure. We've had to learn about spf, dkim and dmarc records (which was had mostly right for our own servers) and we found that the major email providers often bounce mail for no good reason. Especially hotmail/msn and pacbell/sbcglobal/att. The msn issue seems fixed, the att related ones may not be.
Our mail comes from different servers as well, and if you want to be sure to receive our mail you can whitelist the domains csac.org and avalancheassociation.org.
What follows is a summary of our systems, and of the problems and technicalities. If you have problems this might explain them, and for anyone managing their own email this may be helpful.
Our Addresses, Domains and Servers
Historically we have used csac.org and still use this for direct (non-list) emailing. The website domain will forward to avalanche-center.org which we now use as the primary web domain. We do not send mail from avalanche-center.org. Our webserver is with Digital Ocean and they have many spammer problems. As a result entire IP ranges are on blacklists and sending through our own SMTP server became impractical. We now use smtp2go for outgoing mail which is reliable.
The only mail through our own server is that sent via the web, such as contact forms, password reminders, etc. Most of this is sent to our own server, and if it's outbound we get a copy ourselves. Problems are not common. The only blocklist we were on recently allowed for easy removal - they blocked a range but let individual IP's remove themselves quickly and easily. (Unlike microsoft and others, per below.)
When we set up the new list mail we did so on a different server dedicated to that. We needed a different domain, and we have numerous domains we don't use (including some left over from projects for others that they ended up not pursuing) so we chose one. The domain avalancheassociation.org is only used for list mail and is technically set up 100% correctly and is not on any block lists. While mail is sent From: this domain the Reply-To: is still csac.org
While our DNS records for csac.org were set up ok for our server and for smtp2go it turns out they were never set up for MadMimi and some of that mail may have been going to spam/junk depending on the recipient provider.
Getting Our Mail in your inbox, Keeping our lists clean
There are some things you can do to help our mail go to the inbox. Not only yours but others as well, for people who want to receive it.
First, whitelist the domains csac.org and avalancheassociation.org. This is often enough.
Second, watch your spam/junk folder. For our mail as well as other mail you may want. Be sure to mark what you want as "not junk". Many or most providers will periodically test things by putting something in junk, even things you open regularly and in some cases whitelist. Moving it out of junk should keep it out after that.
Third, filter mail if you prefer to do that and know how. You can filter mail from our domains into a dedicated box or explicitly into your inbox. We also prefix the subject of all list mail with [CSAC], partly for this reason.
Finally, it helps if you open our mail. Even if you are busy and don't read that one. Just open before deleting. Clicking a link is even better.
If you really do think our mail is junk there is an unsubscribe link at the top of all list mail. We hate to lose subscribers but there is no benefit to trying to serve people that are not interested. Please use our link, unsubscribe links from your provider are taken by them as a mark against us. (Or any sender you use them for.) Unsubscribes done that way are added to a ban list and cannot resubscribe in the future.
Problem Email Providers
When we first sent out list mails we were surprised to find hotmail (and msn in general) summarily rejecting us from all their clients. We are not on any bad lists, and other providers had no problems. (Except sbcglobal and other att domains which also are a problem.)
Their bounce message says our IP is in a blocked range. So a few bad IP's used by spammers cause all the other addresses to be blocked as well, for no reason. It's not clear if this is the real reason or not though. It also appears that if you are a new sender from a new IP their default is to reject you. Until you build a reputation, which is hard to do when rejected.
It's not easy to find the right contacts. We did finally submit requests to two places. It's necessary to use an MSN domain to contact them at all, so we emailed from hotmail. They did respond, and ironically enough their own email from their support was put into the junk folder. In their own system. But in the end the problem was resolved. For msn domains, not for sbcglobal or pacbell which are bouncing now. But we don't have many of those addresses.
If you use these services you should be aware that they may be blocking mail you sign up for and want. You have no chance to whitelist it or do anything as a customer, you are not even aware of this happening. It can happen with any sender, not just us. One solution is to use a free forwarding domain and have it sent to hotmail (or wherever). We've done this with 33mail.com and there are others. 33mail is not blocking anything (unless you log in and set it to), and msn seems to accept forwarded mail from them. Although I also forward a copy of mail to a professional academic address to hotmail (since I don't check the account directly much) and some of it appears while some does not.
The Technicalities
So for any techies this section explains our DNS records lessons.
DNS is the Domain Name System and records there tell the internet which addresses (domains) go to which machines (identified by IP address). It includes information on who can send mail for a domain, authentication details, and more.
There is an SPF record which lists the servers and domains allowed to send mail for the domain. For csac.org this is our web server, smtp2go, and madmimi. For avalancheassociation.org it is only that domain. This has been correct for a long time with the possible exception on madmimi.
There is a DKIM record which has a public encryption key. This was also set up correctly, except possibly for madmimi again.
Now, as if those are not enough, there is a DMARC record which tests both of the previous two things. This is required for high volume senders (which we are not, by any stretch), but is also helpful for any sender. We either did not have this or had some problems on csac.org. This has not affected direct mail or server mail but may have been a problem for the madmimi mail. This record was correct for avalancheassociation.org from the start.
These records have never been a problem for our new list mail since it was all set up correctly and mail is only sent from one machine. For csac.org I think the DMARC was missing for smtp2go, or they were omitted. But for our low volumes it has never been a problem. It is fixed now anyway.
It turns out that mail through madmimi is more complicated than they indicate, unless you go searching for the technical requirements. (And know what to search for.) On mail-tester.com we had a lower score using them, while the new system consistently scores 10/10. We had to make sure madmimi was included in all of the csac.org records, and since it was not some mail was probably being put into junk folders. We are phasing our madmimi but fixed it anyway.
The reason madmimi is a bit more complicated is that the sender is csac.org so the records for csac.org need to reflect that. Our new server sends mail as avalancheassociation.org which is also the server name so it's simpler. We use snow@csac.org as a Reply-To: and that can be done without problems.
Conclusion
It's not clear how many people will care about some of this, or any of it. But it's all here for the record.
If you want mail from a certain domain be sure to whitelist it. If you don't see it at all try using a forwarding address.
If your provider is bouncing mail you want before you ever see it consider changing providers. While hotmail/msn has been problematic Yahoo and Gmail have not been.
The only way to take control of your incoming email is to manage it yourself. Mail sent to csac.org arrives on our webserver, a FreeBSD unix machine. We do block a few TLD's such as .school .cc and other 2 letter domains rarely used for non-spam. Other than that we filter spam ourselves using spam assassin which can be fine-tuned the way one wishes. And it is read on the server using Alpine, a text based email program. We don't miss much mail this way and have control over everything ourselves.
It's unfortunate that the internet has reached a point where big entities can and sometimes do control the flow of information right down to desired emails. They can do this for whatever reason they decide to. In our case it's just technical, or inherent suspicion of anyone new. We're not political in any way. But that could also be a reason in other cases. In theory, at least for now.
Over the past few months we have implemented an in-house mail system again, with a robust web management interface. We are now using that, and in the coming season we will be able to make regular mailings with no per-message cost. But this transition has been an adventure. We've had to learn about spf, dkim and dmarc records (which was had mostly right for our own servers) and we found that the major email providers often bounce mail for no good reason. Especially hotmail/msn and pacbell/sbcglobal/att. The msn issue seems fixed, the att related ones may not be.
Our mail comes from different servers as well, and if you want to be sure to receive our mail you can whitelist the domains csac.org and avalancheassociation.org.
What follows is a summary of our systems, and of the problems and technicalities. If you have problems this might explain them, and for anyone managing their own email this may be helpful.
Our Addresses, Domains and Servers
Historically we have used csac.org and still use this for direct (non-list) emailing. The website domain will forward to avalanche-center.org which we now use as the primary web domain. We do not send mail from avalanche-center.org. Our webserver is with Digital Ocean and they have many spammer problems. As a result entire IP ranges are on blacklists and sending through our own SMTP server became impractical. We now use smtp2go for outgoing mail which is reliable.
The only mail through our own server is that sent via the web, such as contact forms, password reminders, etc. Most of this is sent to our own server, and if it's outbound we get a copy ourselves. Problems are not common. The only blocklist we were on recently allowed for easy removal - they blocked a range but let individual IP's remove themselves quickly and easily. (Unlike microsoft and others, per below.)
When we set up the new list mail we did so on a different server dedicated to that. We needed a different domain, and we have numerous domains we don't use (including some left over from projects for others that they ended up not pursuing) so we chose one. The domain avalancheassociation.org is only used for list mail and is technically set up 100% correctly and is not on any block lists. While mail is sent From: this domain the Reply-To: is still csac.org
While our DNS records for csac.org were set up ok for our server and for smtp2go it turns out they were never set up for MadMimi and some of that mail may have been going to spam/junk depending on the recipient provider.
Getting Our Mail in your inbox, Keeping our lists clean
There are some things you can do to help our mail go to the inbox. Not only yours but others as well, for people who want to receive it.
First, whitelist the domains csac.org and avalancheassociation.org. This is often enough.
Second, watch your spam/junk folder. For our mail as well as other mail you may want. Be sure to mark what you want as "not junk". Many or most providers will periodically test things by putting something in junk, even things you open regularly and in some cases whitelist. Moving it out of junk should keep it out after that.
Third, filter mail if you prefer to do that and know how. You can filter mail from our domains into a dedicated box or explicitly into your inbox. We also prefix the subject of all list mail with [CSAC], partly for this reason.
Finally, it helps if you open our mail. Even if you are busy and don't read that one. Just open before deleting. Clicking a link is even better.
If you really do think our mail is junk there is an unsubscribe link at the top of all list mail. We hate to lose subscribers but there is no benefit to trying to serve people that are not interested. Please use our link, unsubscribe links from your provider are taken by them as a mark against us. (Or any sender you use them for.) Unsubscribes done that way are added to a ban list and cannot resubscribe in the future.
Problem Email Providers
When we first sent out list mails we were surprised to find hotmail (and msn in general) summarily rejecting us from all their clients. We are not on any bad lists, and other providers had no problems. (Except sbcglobal and other att domains which also are a problem.)
Their bounce message says our IP is in a blocked range. So a few bad IP's used by spammers cause all the other addresses to be blocked as well, for no reason. It's not clear if this is the real reason or not though. It also appears that if you are a new sender from a new IP their default is to reject you. Until you build a reputation, which is hard to do when rejected.
It's not easy to find the right contacts. We did finally submit requests to two places. It's necessary to use an MSN domain to contact them at all, so we emailed from hotmail. They did respond, and ironically enough their own email from their support was put into the junk folder. In their own system. But in the end the problem was resolved. For msn domains, not for sbcglobal or pacbell which are bouncing now. But we don't have many of those addresses.
If you use these services you should be aware that they may be blocking mail you sign up for and want. You have no chance to whitelist it or do anything as a customer, you are not even aware of this happening. It can happen with any sender, not just us. One solution is to use a free forwarding domain and have it sent to hotmail (or wherever). We've done this with 33mail.com and there are others. 33mail is not blocking anything (unless you log in and set it to), and msn seems to accept forwarded mail from them. Although I also forward a copy of mail to a professional academic address to hotmail (since I don't check the account directly much) and some of it appears while some does not.
The Technicalities
So for any techies this section explains our DNS records lessons.
DNS is the Domain Name System and records there tell the internet which addresses (domains) go to which machines (identified by IP address). It includes information on who can send mail for a domain, authentication details, and more.
There is an SPF record which lists the servers and domains allowed to send mail for the domain. For csac.org this is our web server, smtp2go, and madmimi. For avalancheassociation.org it is only that domain. This has been correct for a long time with the possible exception on madmimi.
There is a DKIM record which has a public encryption key. This was also set up correctly, except possibly for madmimi again.
Now, as if those are not enough, there is a DMARC record which tests both of the previous two things. This is required for high volume senders (which we are not, by any stretch), but is also helpful for any sender. We either did not have this or had some problems on csac.org. This has not affected direct mail or server mail but may have been a problem for the madmimi mail. This record was correct for avalancheassociation.org from the start.
These records have never been a problem for our new list mail since it was all set up correctly and mail is only sent from one machine. For csac.org I think the DMARC was missing for smtp2go, or they were omitted. But for our low volumes it has never been a problem. It is fixed now anyway.
It turns out that mail through madmimi is more complicated than they indicate, unless you go searching for the technical requirements. (And know what to search for.) On mail-tester.com we had a lower score using them, while the new system consistently scores 10/10. We had to make sure madmimi was included in all of the csac.org records, and since it was not some mail was probably being put into junk folders. We are phasing our madmimi but fixed it anyway.
The reason madmimi is a bit more complicated is that the sender is csac.org so the records for csac.org need to reflect that. Our new server sends mail as avalancheassociation.org which is also the server name so it's simpler. We use snow@csac.org as a Reply-To: and that can be done without problems.
Conclusion
It's not clear how many people will care about some of this, or any of it. But it's all here for the record.
If you want mail from a certain domain be sure to whitelist it. If you don't see it at all try using a forwarding address.
If your provider is bouncing mail you want before you ever see it consider changing providers. While hotmail/msn has been problematic Yahoo and Gmail have not been.
The only way to take control of your incoming email is to manage it yourself. Mail sent to csac.org arrives on our webserver, a FreeBSD unix machine. We do block a few TLD's such as .school .cc and other 2 letter domains rarely used for non-spam. Other than that we filter spam ourselves using spam assassin which can be fine-tuned the way one wishes. And it is read on the server using Alpine, a text based email program. We don't miss much mail this way and have control over everything ourselves.
It's unfortunate that the internet has reached a point where big entities can and sometimes do control the flow of information right down to desired emails. They can do this for whatever reason they decide to. In our case it's just technical, or inherent suspicion of anyone new. We're not political in any way. But that could also be a reason in other cases. In theory, at least for now.
