Mail server...why oh why art-thou so difficult to install?

[drozwood90]

So, for the longest time, I have used lots of cloud services.
With all the recent craziness going on, I thought it might be better to just start pulling in all my own "stuff".
With the recomendation from a few people I know, that do this type of thing for a living...along with some inputs from a internet security "expert" I know, I have what I think is a pretty decent plan, with part of it already setup and functioning quite well.  Well enough for me to continue putting in time and effort to get all this running.
 
I would like to replace:

• Google Drive
• Pandora
• Netflix (probably not 100%, since they get all the new content, and I like previewing movies to see if they are good, prior to purchase)
• RSS Feeds (this is what forced me to start doing this, since Google Reader went away)
• Access to my local files (currently use SFTP)
• Allow RSYNC to work 100% (using Tasker backup my phone, allow my file server to sync it's main drive to my other server each night - unless there is a problem with the file server's drive array
• Maybe blog - right now I just use a .ini file that is hosted through Homeseer's webserver
• Eliminate SSH
• remove dependance on Lattitude for lattitude functions
• possibly run a "Walkie-Talkie server" to do one button press voice coms with Wife/House "intercoms"

NOTE, skip to *** if you do not want to read EVERYTHING and jump JUST to the e-mail server rant
 
My wife and I use Android phones, so there is a LOT of integration that needed to be worked out.
 
To eliminate SSH, I am going to institute VPN...need to learn a LOT about that.  I'd like to link my house through a VPN tunnel to my brother's house for offsite backups (RSYNC).
 
For the walkie-talkie, I've been messing with Virtual Walkie Talkie https://play.google.com/store/apps/details?id=ix.com.android.VirtualWalkieTalkie
That app has it's own server you can run yourself.  I liked the features of I think it was called Zeem...but their server costs $9000...so I think I'm good.  The nice part is, I have been slowly migrating to Android tablets/old phones for touchscreen interfaces in my house.  I could create a intercom system in my house/ at the front door that would interact with my mobile phone.  That coupled with my cameras at the front door and Z-Wave locks could prove quite useful.
At the very least, setup a Voice Message server to send voice messages as I'm driving...seems to me this would be more reliable than voice transcoding to text.  I could hit record, talk, send to my wife's mailbox she could listen to it later.
 
To replace Google Drive, I use Owncloud + Kingsoft Office.  Kingsoft for Android is free, does presentations, excel, word, pdf docs.    It also can directly access webdav drives (essentially what OwnCloud is under the hood).  Owncloud lets me access my files through the web page as well - with BUILT in MP3 streaming and a few other features that I really do not use.  So, as soon as I get samba working, Owncloud also has an application for android that gives me a lot of the same integration that the "Drive" app does.  One thing it does not provide is the new scan to drive feature.  However, Genius Scanner takes care of that (just share the "scanned" pdf to Owncloud's app).
With these features, Owncloud should allow me to remove Google Drive, Pandora (when on a PC), access to my local files.
To eliminate Pandora when on my phone, I use BubbleUPnP.  This is a DLNA "remote" app, that has a server you can run on your LAN.  That server interfaces with your local DLNA server and allows me to stream my videos/pictures/music to my cellphone (transcoded/shrunk/re-encoded for optimal bandwidth - Over the Air VS on WIFI).
 
For RSS, I am using TT-RSS.  Great server and can not speak highly enough about it!  It does all the RSS feed consolidation.  Then using the RSS Client for my Android phone, I can see all my threads - or use the Web app.  Fantastic!
 
The blog thing was a - ehh...let's see how hard it is to make a Wordpress blog...mostly because I find I need a place to keep my notes and write-ups about my automation projects.  Like a build log or something like that.  Right now, I just use Google-Word or Google Presentation documents.
 
I am also now, running all this behind a SSL reverse proxy - using Abyss Webserver.  I tried Squid and just COULD not figure it out.  Abyss does a pretty good job, except with Homeseer itself.  Apparently, from a post on the HS forum, there is a lot of things one must do to get Homeseer to run through a reverse proxy.  Owncloud was hard...but there were good guides on what to do to make it work.  FOr the most part, I end up logging in, using the WEbDav page, then get redirected to the fancy interactive page.  Works well.  And I get a small layer of security.  Plus, makes it easy to only have to put my SSL certificate in ONE place and everything flows through that one server.
 
***
So, the last thing, I do not want to run a full up mail server.  I.e. I do not want it to be able to send mail from that server to my gmail account.  No need for MX Servers or anything like that.  What I want to do is setup a mail server that does IMAP Idle, SMTP, but is all self contained, all by itself.  Just local to my LAN.  The reason, I've been using Google Voice to push messages from my Homeseer server to my phone.  What if that goes away as lattitude just did, and Google Reader did?  Or, if the API changes, I am at the mercy of other people who have written python scripts to interface to the Google API to allow me to send my texts.  Why not e-mail?  Well, now I've got clutter in my phone or have to have multiple GMAIL accounts that are hard to trigger Tasker from.
Why not just use some service that allows you to push a message to Google's servers for pushing to my phone.  I am beginning to like the idea of Google recording EVERYTHING I do less and less.  Especially with all this Government involvement junk.
So, why not Exchange/plain old IMAP with IDLE?  I get push messages then, minimal data use on the phone, minimal battery waste?  Plus, I can use K-9 mail client which integrates so well with Tasker for triggering actions.
 
I have tried 2-3 Free IMAP Mail servers.  I'm thinking because I do not OWN my own domain, that is what is causing all my issues with those.  The closest I got was with http://www.xeams.com/ Mail Server.  Not because it was easier than any others - hMail was really easy to setup as well...but I got this far, because I edited the hosts file on my phone to point imap.mydomainname.net to the IP address of the PC running that server.  I was able to authenticate to the machine, BUT was not able to actually send/receive any mail.  I am not sure why.  I spent the better part of the last two to three days messing around with it.  All of this JUST to allow my HOMESEER server to directly contact my phone...or vice-versus using e-mail (in case I am in an area where I lose cell contact...the e-mail client will keep trying to send the mail to my server...
 
As for Exchange, I used to ADMIN an exchange server when it was version 5.5.  I have a license for 2003, which I tried to install...Active Directory/domains/etc. JUST INSTALL!!!  I spent 20 minutes on it.  I decided that the amount of effort it was going to take, just to get the stupid thing installed was not worth my time.
 
Maybe it is time to get a linux mail server running?  Or maybe I do not fully understand the client side?  I guess this is more a venting and ranting because I wasted so much time on something that is SIMPLE and easy.
 
Of course, any suggestions on the best way to go about this, I would be greatly appreciative of.  The hardest part was getting something that was mostly free (since this is ONLY to connect to my phone for the occational message...it isn't worth paying $300 for a mail server), and did IMAP.  From my searching, the VAST majority of mail servers did not have IMAP unless they charged for it.
 
I suppose I could temporarily setup my phone with IMAP to a GMAIL account.  Then I can use K-9 mail client to at least prove the concept to see if it is WORTH pursuing, as part of my pulling everything out of the cloud.
 
And here is a cookie for anyone that actually read this far:
 


       _.:::::._
     .:::'_|_':::.
    /::' --|-- '::\
   |:" .---"---. ':|
   |: ( O R E O ) :|
   |:: `-------' ::|
    \:::.......:::/
  ':::::::::::'
        `'"""'`
--Dan

 

[tmbrown97]

Sounds like quite the endeavor!  It was long so I can't speak to many specific parts, but here's a couple things that came to mind...
 
For your walkie-talking and intercoms and all that, have you looked at SIP? Maybe run a little asterisk or freepbx installation; you could even get an XMPP enabled chat interface as well that might well eliminate your need for IMAP email; if not, there are open source projects such as http://www.hmailserver.com - that might suit your need.
 
I like owncloud and will probably do something similar...
 
For the VPN to your brother, there are plenty of ways; one very simple one is the $99 ubiquiti edgerouter - or if you like the DDWRT route, I'm pretty sure that has something built in...  I like the edgerouter because it can be an easy site to site VPN as well as VPN server for remote access and supports all the basic protocols (VPN, LT2P, OpenVPN).

 

[drozwood90]

Work,
 
Thanks for the ideas.  I will play around with hMail a bit more.  I think I honestly just need to find a mail expert and have them look at my setup.  I personally feel like the problem is external to the mail server.
 
I did not consider SIP, mostly because I thought it would be good to just have the thing connected to everyone at the same time.  I thought SIP was more like a phone - point to point?
 
I honestly didn't expect most to read the whole thing.  I was more just rolling with it and probably rambled too much in there!
 
Thank you for the reply!
 
--Dan

 

[Frunple]

Take a look at pfSense.
It'll take care of a few of your 'wants' with the available packages it has. 
FusionPBX and OpenVPN to name a couple.
As for the VPN, I have OpenVPN setup between my house, my brothers house and my parents house. Each can access the others. Very simple to setup and use.

 

[tmbrown97]

SIP is more of a phone than all-at-once, but you can do intercom-like features with it... the thing is it's much easier to set up and you can find cameras and SIP clients that work with it.
 
I won't call myself a mail expert but if you need a hand looking at things, lemme know - I've been running Exchange and SMTP servers for about 15 years - long enough to hate but fully understand email servers.

 

[drvnbysound]

To replace all of that... I sure hope you have an excellent internet connection - great bandwidth both up and down, and high availability.
 
Speaking of replacement of Pandora alone, if I was on my way to work today listening to Pandora and it stopped working... I'd be pissed, and it's free. I couldn't imagine how I'd feel if I were using my own streaming service and it stopped working... I could only blame myself.
 
There are plenty of things that I use on a daily basis and as much as I'd like to control everything, I know that I don't have the ability to always create the same experience that others already have. That being said, there are plenty of things that I do myself, but as many or more that I leave alone because I know they wouldn't be as good if I did it myself. The goals above are certainly ambitious, and at some point I think you will have to consider if you can create the same experience or better than what you are using today... or if you'd just be wasting your time; unless of coarse the building/maintaining of it is the hobby itself and you are OK working on these things all the time.
 
For the mail server... you mention that you may not have had a good experience because you don't own your own domain. You do realize that you can get free DDNS domain names and map that to your own server right? Alternatively, I do own a domain and pay for web hosting, which also includes email among many other things - I pay $44/yr for hosting and $12/yr for my domain. I choose to pay for hosting so that all of the intrusion attempts were not on my home network... and the fact that I know literally thousands of them occur everyday. I'd much rather pay $44/yr and have someone else manage and worry about that...

 

[sic0048]

Have you looked at Play Music for your music streaming?  It allows you to "upload" your music collection to Google and they will stream it back to you on any device for free.  You can also subscribe to a music streaming service if you like, but there is not charge to access your music.  It's a great backup service for your collection if nothing else! 

 

[wuench]

Drvn makes a good point, replacing streaming services is going to reduce your reliability/quality of service.  Offloading to the cloud gives you a lot of hidden benefits.  There is a lot that goes on over the internet to make these services reliable.  Services like Akamai are used to give sites like Netflix and Pandora geographic diversity and even give them an actual presence in your ISP's data center and there are lots of contracts and agreements in place to make streaming services reliable.  So their streams don't all come from one point on the internet like your solution will, but instead come from the closest available point, and then the local ISP gives preference to stuff on it's own network so it doesn't get dropped.  And this will probably only become more true in the future.  The internet backbone and inter-isp connections simply can't handle anything to anywhere at anytime.
 
Additionally there is no Quality of Service on the internet, so you won't be able to give your video, music streaming or "walkie/talkie/voip" (for example) preference over file transfers externally.  QoS settings on your router only apply to your internal network interfaces, they won't apply to the WAN port and the ISP won't honor them if they did.
 
Hopefully it works for you but if you run into issues the first thing to check after you are sure you aren't exceeding bandwidth on your home router, is that you don't have multiple things going on.  Turn on QoS anyway giving preference to your streaming sources (if it is available on your router), to at least let it manage your internal interfaces.  If you still have issues you may have to select some stuff to offload back to the cloud.

 

[drozwood90]

Hi All,
 
Thanks for the points about bandwidth...I'm not worried about that since I am the only person who does this remote.  My kids are 3 and 4.  My wife barely likes to use her phone for anything but e-mail - which I am not replacing.  I am only setting up the mail server to just be able to "activesync" or do IMAP idles to get the push mail ability.
 
I also only transcode any video I do stream.  I do not do any HD.  We are talking 128K audio and for video, it is transcoded to 120P/64k audio.  Note, we have cable modem at 1Mup/10Mdown, so seems like it is good enough for all that.  I've tested video at higher rates, up to 360p/128K audio without issue...but when I'm burning mobile OTA data (I have 2GB 4G / mo)...ehh 120P is good.  I'm not so much into quality anyhow.  More ofthen than not, I'll end up streaming data once or twice a year.  mostly when I'm stuck in the DMV waiting. 
 
IF I were trying to get full quality, I completly agree with all the comments regarding this.  I don't even use QoS at home...everything seems to work fine.  And that is with the network loading up full 1080P video running across the servers into the various media playing devices on the network.
 
However, as I said, almost all this stuff is really for a single user.  Me.  At a later date, probably others, but for now, I'd like it to just work for me...and I can not forsee any case where I'd be streaming a movie and working OwnCloud files (the two largest users of bandwidth IMHO).
 
Thank you all for taking the time to read my messages and comment.  I think I was mostly posting out of frustration.  My brother told me that is why he went with the small business server edition, so Exchange was all integrated and setup for him.  He just tossed in the LAN configuration data and off he went (he uses that for a similar purpose of being able to get push messages direct from his house server to his phone).
 
--Dan

 

[pete_c]

Dan,
 
How far does your brother live from you?
 
If its not that far you can maybe create a wireless bridge between the houses.  A DIY experiment for you?

 

[drozwood90]

Pete,
 
Good idea, but he's in Maryland.  I'm in New York near Rochester.  I'm sure there is some FCC regulation I'd be violating!
 

 
Although, to keep all of that backup traffic off the WEB is ideal.  I've yet to get a neighbor that I'd trust with something like that, but it would be nice to be able to drop a dockstar with a WIFI connector+cantenna pointed at my house...and vise-verus to perform OUT of home backups like that.  That is why I decided to seed my data to my brother (fill a USB drive, mail to him, or any time we meet up).  Then he has the MAIN data on his server and I need only run rsync to keep everything up to snuff.  Also learned that rsync can properly maintain a few days of backups.  With some nice script trickery even get a rolling backup going. 
Final thing that really made me happy with this configuration, I found rsync, ssh, and curl client executables for Android.  I'd like to get samba as well...but fear I'd have to install debian.  So, for the moment, I sync my pictures with rsync.  I'd really like to use something like samba to just offload the pictures to safe keeping on the server.   I suppose I could use sftp...just so many options and I am trying to keep things SIMPLE.
 
So, since OwnCloud is really webdav, I have found many python scripts that will run webdav type stuff.  Maybe I can get one of those to run.  I also believe there is command line vpn tools for Android, so I should be able to script connecting with VPN to perform these "sync" operations from remote.
 
If not, I just found an article:
http://www.pocketables.com/2013/07/how-to-emulate-touch-input-using-tasker-and-root.html
 
which talks about well, emulating touching of the touchscreen.  This opened up a host of new things for automating things on my phone with Tasker.  Apprently this command they talk about is either new, or the current ROM on my phone is the first one to support it.  Either way, it is making me quite happy!
 
--Dan

 

[tmbrown97]

If you are really into off-siting backups, could play around with Amazon's EC2 - the 1 year free tier with some storage blocks added would work, and you can run a VPN client on it that'll connect back to the house.

 

[drvnbysound]

It's not the general upload speed/bandwidth that I would have any issue with... it's the part of relying on a residential ISP for connectivity that would concern me. Free or not, when I load Pandora on my phone I expect it to work 99.999% of the time, and if it's not working I assume that it's my carrier. That's just a SMALL part of what you've listed above, which is why I prefer to offload most/all of that to someone else who has already done the work and pays for a commercial grade ISP with a much better uptime than I get via my service level agreement.
 
That said, I think the amount of work you are putting into offsite storage is impressive. I'm a member of a large photography forum with a LOT of professional photographers who hold digital files for a LONG time in case of natural disasters that could ruin family heirlooms. That being said, the most robust solution that I've seen/heard of is also the most simple. External HDDs. Sync 2 of them together - there are plenty of applications that do this. Whatever you put on one of them, should automatically be applied to the second (not once a week, or once a month, but immediately). This eliminates the concern for general drive failure (or most of it anyway). In your case, you could have a third rotating drive that you could sync to the primary drive and give that to your brother for your offsite backup. Each time you meet, you can hand him a new drive, and he hands you the other one back. So he always has a copy of the latest version.
 
I actually heard about this approach (actually the suggestion was to just do it manually) from a guy who used to work in the database group at Microsoft. He said when you run the numbers (time included) it's the most cost effective and easiest to implement solution. Once the main drive is full you can store it in a safe deposit box or where ever you desire. To make this as easy as possible you can do it with standard internal HDDs using the toaster style docking stations; this leaves no reason to need to open your PC case and it's easy to swap them out regularly for updating.

 

[video321]

It's not the general upload speed/bandwidth that I would have any issue with... it's the part of relying on a residential ISP for connectivity that would concern me. Free or not, when I load Pandora on my phone I expect it to work 99.999% of the time, and if it's not working I assume that it's my carrier. That's just a SMALL part of what you've listed above, which is why I prefer to offload most/all of that to someone else who has already done the work and pays for a commercial grade ISP with a much better uptime than I get via my service level agreement.
 
That said, I think the amount of work you are putting into offsite storage is impressive. I'm a member of a large photography forum with a LOT of professional photographers who hold digital files for a LONG time in case of natural disasters that could ruin family heirlooms. That being said, the most robust solution that I've seen/heard of is also the most simple. External HDDs. Sync 2 of them together - there are plenty of applications that do this. Whatever you put on one of them, should automatically be applied to the second (not once a week, or once a month, but immediately). This eliminates the concern for general drive failure (or most of it anyway). In your case, you could have a third rotating drive that you could sync to the primary drive and give that to your brother for your offsite backup. Each time you meet, you can hand him a new drive, and he hands you the other one back. So he always has a copy of the latest version.
 
I actually heard about this approach (actually the suggestion was to just do it manually) from a guy who used to work in the database group at Microsoft. He said when you run the numbers (time included) it's the most cost effective and easiest to implement solution. Once the main drive is full you can store it in a safe deposit box or where ever you desire. To make this as easy as possible you can do it with standard internal HDDs using the toaster style docking stations; this leaves no reason to need to open your PC case and it's easy to swap them out regularly for updating.

 
Pretty much describes my backup strategy that I've been doing for many, many years. Since I'm the "master" of all that is precious data (pics/vids/etc.) I have a copy on my workstation which, along with the other machines, gets sync'd to my server via robocopy (simply the best utility if on a Win platform - create scripts and let it fly!) and shared out. Then I use a docking station to sync to a bare drive as well which then gets brought to my parent's house and swapped out with the bare drive there. This effectively gives my sync'd data on (4) drives! Currently this is being done on 2TB drives purchased when they were $50-60 so the cost for such redundancy is nothing and it easily holds everything I can't replace. Since I'm a little off :nutz:... I even create md5 hashes so the data can be verified. So ~$200 buys me quad redundancy plus off-site which can be verified.
 

Once the main drive is full you can store it in a safe deposit box or where ever you desire.

 
This is one thing I'd advise against simply because of bit rot/magnetism issues unless you have 2 drives stored away.

 

[tmbrown97]

The completely Off-Topic Ubiquiti stuff is now in its own thread: http://cocoontech.com/forums/topic/25070-access-point-placement-questions/