Added IMAP support for checking reported emails #1612

glennzw · 2019-10-07T11:57:35Z

This is quite a big pull request, with front end and back end changes. It was a tonne of work to be honest 🙂 This pull request allows you to set an IMAP server in Gophish that will be continuously monitored for Gophish campaigns that have been forwarded to it by users. If a campaign email is found, it's reported.

The logic is as follows:

Add IMAP details under Account Settings > Report Settings (see screenshot below)
If a user receives a suspicious email they can forward it to the address associated with the IMAP account
The IMAP Monitor checks for new email every n seconds (60 by default) and scans email for the rid=? tag, and reports it if found

Here's a story illustrating the process:

Here's a video of it in action:

https://youtu.be/D3rAYvAthdI

There are some advanced configuration options;

Set the monitor to delete reported campaign emails after successfully reporting them (so as an admin any emails in your specified inbox will only be non-Gophish campaigns and should be investigated).
Restrict checking of emails to a specified domain (e.g. set your company domain, widgets.com, so the monitor only checks emails from your staff).

There's decent error checking / testing too:

Some notes on implementation and configuration:

In config.json we can set the frequency of IMAP checking:

        "imap_freq": 60

The IMAP structure is defined in models\imap.go:

// IMAP contains the attributes needed to handle logging into an IMAP server to check
// for reported emails
type IMAP struct {
        UserId            int64     `json:"-" gorm:"column:user_id"`
        Enabled           bool      `json:"enabled"`
        Host              string    `json:"host"`
        Port              uint16    `json:"port,string,omitempty"`
        Username          string    `json:"username"`
        Password          string    `json:"password"`
        TLS               bool      `json:"tls"`
        Folder            string    `json:"folder"`
        RestrictDomain    string    `json:"restrict_domain"`
        DeleteCampaign    bool      `json:"delete_campaign"`
        LastLogin         time.Time `json:"last_login,omitempty"`
        LastLoginFriendly string    `json:"last_login_friendly,omitonempty"`
        ModifiedDate      time.Time `json:"modified_date"`
}

The actual IMAP account monitoring is done by controllers/imapmon.go. A separate worker is launched for each IMAP account (ie each Gophish user could have a different IMAP server set). This ensures against one account/server denying resources to others. IMAP connectivity is done with eazye. It uses outdated libraries, but was really easy to get up and running. I should ideally switch to go-imap at some point, but it'll be a bit more work (IMAP is a tricksy hobbit). Also need to add backing off logic, and maybe proper worker code.

Sticking with Gophish convention the management of IMAP settings is handled via an API, as you can see in controllers/api/imap.go. The endpoints are:

/api/imap --> Set IMAP settings
/api/imap/test --> Test IMAP supplied settings

The endpoints are accessible via JavaScript calls (implemented in static/js/src/app/gophish.js) . e.g: calling api.IMAP.get():

The IMAP monitor is launched in gophish.go along with the adminServer and phishServer:

	imapMonitor := controllers.NewImapMonitor(conf)

	go adminServer.Start()
	go phishServer.Start()
	go imapMonitor.Start()

There's probably a bit of work to be done on how I handled this.

Finally the settings.js front end code probably needs some tidying in terms of code reuse and naming conventions.

Overall I've done a lot of testing and it seems to work very well. Also, watch this space for my Chrome and Outlook email reporting plugins.

quelsan · 2019-10-07T12:20:27Z

Really nice and useful feature - haven't looked at the code yet, but I have been thinking about a feature such as this.

S0larflare · 2019-10-07T12:30:12Z

So first off, that looks really neat, I really like the entire workflow there to be honest. I'm not sure if you need a realtime count for last checked in the email settings, but thats just me 😄. Other than that, this is pretty fantastic and works better than how I have my reporting set up.

On another note, I've been butting my head against IMAP in Go for ages now and have never managed to get it to work properly. Any chance you'd be willing to field a few questions I have about parsing .msg attachments?

jordan-wright

Hey @glennzw!

Wow! You've done a stellar job with this PR! I really appreciate you taking the time to send this in. Very exciting stuff.

I like the approach and the way you've architected things. I think across the board you're spot on.

I've left some comments from an initial pass. It's possible there's more to review later, but I wanted to start giving feedback ASAP. Let me know your thoughts and we can work through this to get it in great shape and get it merged.

Thanks again so, so much for taking the time to send this over! Looking forward to adding this to Gophish!

config/config.go

controllers/api/imap.go

jordan-wright · 2019-10-08T01:48:21Z

controllers/api/imap.go

+			log.Error(err)
+		}
+		if len(ss) > 0 {
+			ss[0].LastLoginFriendly = humanize.Time(ss[0].LastLogin)


See the comment in the IMAP struct about the LastLoginFriendly attribute.

jordan-wright · 2019-10-08T01:58:44Z

controllers/imapmon.go

+
+	log "github.com/gophish/gophish/logger"
+
+	"github.com/glennzw/eazye"


I always get a bit nervous anytime we introduce a new dependency into Gophish and try to avoid it where possible.

In this case, I totally understand that this is your package so please know this is just a blanket policy that helps me sleep better at night 😛 It appears that the eazye functionality we're using is fairly low-ish footprint. What are your thoughts around doing some restructuring like this:

Create a new github.com/gophish/gophish/imap package to hold all of this code. This file specifically could be in monitor.go. We would then be able to change naming to be Monitor instead of ImapMonitor, etc.

We could create imap.go where we port over the needed functionality from eazye. To make things easier, we might be able to then adjust the parts that handle parsing the emails to use the email.NewReader function from my email library already used here.

Those are just some thoughts off the top of my head. I do feel that this functionality probably doesn't belong in controllers/ since that package is typically reserved for the web handlers.

Yeah totally agree, additional 3rd party packages make us all nervous!

So to be clear, github.com/glennzw/eazye is my fork of https://github.com/jprobinson/eazye with added functionality (so it's not really my package). I agree with your idea to create a github.com/gophish/gophish/imap package to avoid nasties creeping in.

I'd like to move away from eazye at some point (as it uses outdated/unmaintained packages) and use imap-go but this was the quickest way to get up and running.

I've created the gophish\imap package and moved functionality there. (I just copied in the single eazye.go file and added the ValidateIMAP() from the controller).

Also renamed ImapMonitor to just Monitor.

Next up: remove unused functionality in the new imap.go file and try use your email.NewReader to parse emails.

Commit: glennzw@7ddba13

controllers/imapmon.go

models/imap.go

static/js/dist/app/settings.min.js

jordan-wright · 2019-10-09T01:19:30Z

Just an update, here's how to create a new SQL migration. First, you need to install goose if you don't have it already:

go get bitbucket.org/liamstask/goose/cmd/goose

Note: We currently use liamstask/goose, but eventually I'd like to see about moving to github.com/pressly/goose since it's far more maintained.

Then, in the db/ folder, you can create a SQL migration for both SQLite and MySQL. The format is version_description, like this:

gophish/gophish/db$ goose -path db_sqlite3 -env production create 0.9.0_webhooks sql
goose: created /src/go/src/github.com/gophish/gophish/db/db_sqlite3/migrations/20191008201325_0.9.0_webhooks.sql

Then you can add the relevant SQL statements to the new migration file. These will execute when Gophish is run, since we compare the version in the database with the latest migration file. This means that you'll want to create a backup of your database when developing migrations so that you can easily roll back to a previous state.

glennzw · 2019-10-09T14:45:08Z

Hi @jordan-wright, thanks so much for the quick response and positive feedback! I'll address each of your comments individually but they all seem to make sense. I'll work on getting everything in line to make a merge! I'm sure more issues will crop up as you look closer at the code too. I'll ping you if I get stuck with goose too, thanks!

@S0larflare; I feel ya, IMAP is a pain. I can't give more advice than the example code on the go-imap repo for parsing attachments. I used eazye in the end as it's a super simple wrapper (but doesn't support attachments; I actually spent about a day trying to add that support!).

… const

S0larflare · 2019-10-12T09:51:07Z

Why is LastLoginFriendly “omitonempty” rather than “omitempty”?

Sent from my iPhone On 11 Oct 2019, at 21:05, Glenn Wilkinson <notifications@github.com> wrote: @glennzw commented on this pull request.

________________________________ In models/imap.go<#1612 (comment)>:

+

+// IMAP contains the attributes needed to handle logging into an IMAP server to check +// for reported emails +type IMAP struct { + UserId int64 `json:"-" gorm:"column:user_id"` + Enabled bool `json:"enabled"` + Host string `json:"host"` + Port uint16 `json:"port,string,omitempty"` + Username string `json:"username"` + Password string `json:"password"` + TLS bool `json:"tls"` + Folder string `json:"folder"` + RestrictDomain string `json:"restrict_domain"` + DeleteCampaign bool `json:"delete_campaign"` + LastLogin time.Time `json:"last_login,omitempty"` + LastLoginFriendly string `json:"last_login_friendly,omitonempty"` I initially tried to handle this purely in the frontend, but couldn't get moment.js to play nicely. It was a while ago now and I can't remember the specific issue but will dig into it again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1612?email_source=notifications&email_token=ACEGR3ONTNN3UHNAEWE22VLQODMAHA5CNFSM4I6DPTFKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCHXWFCY#discussion_r334155121>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACEGR3KP4LCSWZTQSR77VOLQODMAHANCNFSM4I6DPTFA>.

jordan-wright · 2019-10-13T00:21:38Z

I think the omitonempty is probably just a typo. Probably a quick fix @glennzw.

glennzw · 2019-10-23T14:52:48Z

I think the omitonempty is probably just a typo. Probably a quick fix @glennzw.

Yup, fixed!

Each user now has their own IMAP monitor, and can set their own polling frequency

glennzw · 2019-10-25T16:24:57Z

Thanks for your patience on getting these fixes in, I'm sure you're all very familiar with squeezing in open source projects in evenings and weekends 🙂

I've modified the IMAP polling logic (glennzw@8cd7c07) so that each user has their own IMAP monitoring Go routine, for which they can set their own polling frequency via the web UI advanced settings. The backend logic is:

Start an IMAP monitoring Go routine for each user
For each of the above, check if IMAP settings are present and enabled. If so, check then sleep for the poll frequency (default 60s)
If a user is deleted, stop their Go routine.
If a new user is created, a new Go routine is launched to monitor their IMAP settings

Took a while to settle on this logic, but I think it makes the most sense. I've run a bunch of tests adding and removing users, adding/enabling/disabling IMAP settings.

Open to hear your feedback.

jordan-wright · 2019-10-26T17:07:30Z

Hey @glennzw,

This is looking good! It seems like there's still some outstanding review items (moving things to github.com/gophish/gophish/imap, potentially removing the LastLoginFriendly, etc.) but let me know when you're ready for me to do another full review and I'd be happy to.

Keep up the great work! This is really quickly getting in great shape.

glennzw · 2019-10-28T14:24:26Z

Hi @jordan-wright, I've resolved all the reviews I was able to. Looking for comments on:

Created gophish\imap package Added IMAP support for checking reported emails #1612 (comment)
Reporting campaign via db or http query Added IMAP support for checking reported emails #1612 (comment)
Friendly last login Added IMAP support for checking reported emails #1612 (comment)

Thanks!

templates/settings.html

static/js/src/app/settings.js

imap/imap.go

imap/monitor.go

glennzw · 2020-01-07T16:45:19Z

Addressed the new issues, let me know if you find any more 🙂

imap/imap.go

models/imap.go

jordan-wright · 2020-01-08T03:05:44Z

Hey @glennzw!

This is still looking great. We're definitely in the home stretch. I've added a few more comments for your review, but things are really coming along nicely.

Thank you so much for all your great work on this!

glennzw · 2020-01-08T13:30:35Z

Addressed the new issues, some sloppy ones on my side there!

jordan-wright · 2020-01-15T05:11:14Z

Hey @glennzw!

Sorry for the delay- things have been quite busy over here. I should have time to review this today or tomorrow. I think the only outstanding item is removing the minified JS from the commits. I'm honestly not sure what the easiest way is to do that. I think there's a way to reset (or just checkout again) the files from master and then re-commit.

I'll do some digging on my end tomorrow if you haven't found an answer by then. I'm sure we can figure it out together 😄

Thanks so much for your patience!

glennzw · 2020-01-15T15:47:52Z

Hey @jordan-wright!

No worries at all, that's the beauty of OSS projects, we give when we have time :) I did a checkout against the .min.js files from the original branch I forked in my repo, hopefully that works 😬

git checkout 24fe998a3aa04e205900476a9601d481e94d8eea settings.min.js
git checkout 24fe998a3aa04e205900476a9601d481e94d8eea gophish.min.js
git checkout 24fe998a3aa04e205900476a9601d481e94d8eea vendor.min.js

controllers/api/imap.go

db/db_mysql/migrations/20191022184235_0.9.0_imap.sql

imap/imap.go

models/imap.go

jordan-wright · 2020-01-16T05:12:04Z

Hey @glennzw,

I left a few comments on the diff- things like a bug in the MySQL migration and some small nits. And it looks like all of the minified JS is gone except for vendor.min.js, so if we could remove that as well that'd be great.

With these things out of the way, I think we'll be in a position where we can merge this in 🎊

Co-Authored-By: Jordan Wright <jmwright798@gmail.com>

glennzw · 2020-01-16T12:26:49Z

Hi @jordan-wright,

Thanks for spotting those little mistakes, I really should have caught them!

Hmm for the vendor.min.js it looks like it matches the branch from which I originally forked (Fix multiple XSS issues):

commit 28ba68c9163f5ab02ba75814bac9122205f823f5
Author: Glenn Wilkinson <glenn.wilkinson@gmail.com>
Date:   Thu Aug 29 12:48:05 2019 +0100

    Added functionality to manage IMAP settings via UI

commit 24fe998a3aa04e205900476a9601d481e94d8eea (origin/master, origin/HEAD)
Author: David Maciejak <david.maciejak@gmail.com>
Date:   Sat Aug 24 10:07:15 2019 +0800

    Fix multiple XSS issues in User Management Page (#1547)
    
    If the user name is embedding some JS code, it will be executed on the client side. Note: gophish/static/js/dist/app/users.min.js will need to be regenerated too.

(I ran the command git checkout 24fe998a3aa04e205900476a9601d481e94d8eea vendor.min.js on my last checkin)

But it looks like the latest modification to vendor.min.js is from the Upgrade SweetAlert2 Dependency (#1583) commit, so perhaps if I check that file out it'll align better.

I ran git checkout 6222c5e180a97d0aa3ec345d0c7af06c380f3db5 vendor.min.js for my latest commit so let's see.

jordan-wright · 2020-01-17T02:14:17Z

Looks like that worked! The very last thing I noticed was that we'll probably want to update the filename of the SQL migrations. It's my understanding that our migrations library, goose, works based on the timestamp.

Since we added webhook support and those migrations have the timestamp 20191104103306, I think that folks who use the master branch of Gophish wouldn't see these IMAP migrations applied since they have the timestamp 20191022184541. The fix for this would be to update the date portion of those filenames, e.g. 20200116000000 or so.

Once that's done, things are good to merge! Sorry for all the back and forth. I really appreciate all your patience here!

glennzw · 2020-01-17T12:22:33Z

Hi @jordan-wright , whoop almost there! I renamed the two database files 👍

No worries on the back and forth, it's been a great learning experience for me and I've enjoyed it!

jordan-wright · 2020-01-18T17:57:50Z

This works for me! There might be some small tweaks in the future, but that's stuff we can figure out later. I think this is a stellar first pass.

Thank you so much for all the hard work and patience!

glennzw · 2020-01-18T18:05:49Z

Whooop, that's brilliant, I'm so happy to have contributed to the project! Thanks for all the guidance and code reviews! I hope it's a feature people find useful 😃

Also yes I'm sure there will be more tweaks and fixes once people start using it - I'll keep an eye on it!

wshi4440 · 2020-02-11T16:12:27Z

I downloaded 0.90 and tried this new feature. But I got this error "tls: first record does not look like a TLS handshake".
Did I miss any steps? Do I need to import server cert to the machine where gophish is running?

glennzw added 4 commits August 29, 2019 12:48

Added functionality to manage IMAP settings via UI

28ba68c

Added IMAP support for reporting of campaigns

2878762

Resolved merge conflict

e2e0e19

Updated IMAP settings for upgraded SweetAlert2

e2ee801

jordan-wright requested changes Oct 8, 2019

View reviewed changes

jordan-wright added the enhancement label Oct 8, 2019

jordan-wright self-assigned this Oct 8, 2019

glennzw added 2 commits October 11, 2019 18:29

Updated naming scheme from "Test IMAP" to "Validate IMAP"

7b19370

Fixed typos, comment formatting, and updated default INBOX value to a…

8768bf9

… const

Created new migration files via goose for IMAP support

26b0ed1

Updated IMAP polling logic.

8cd7c07

Each user now has their own IMAP monitor, and can set their own polling frequency

glennzw added 4 commits October 26, 2019 21:18

Renamed DeleteCampaign to DeleteReportedCampaignEmail

2dd82d3

Created gophihs/imap package and moved functionality there

7ddba13

Fixed IMAP API calls

baa51d6

Removed GetEnabledIMAPs function as no longer required

88bdf6d

jordan-wright reviewed Nov 5, 2019

View reviewed changes

templates/settings.html Outdated Show resolved Hide resolved

jordan-wright reviewed Nov 5, 2019

View reviewed changes

static/js/src/app/settings.js Outdated Show resolved Hide resolved

jordan-wright reviewed Nov 5, 2019

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Nov 5, 2019

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Nov 5, 2019

View reviewed changes

imap/monitor.go Show resolved Hide resolved

Updated MailboxInfo schemes

bab1145

jordan-wright reviewed Jan 8, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 8, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 8, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 8, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 8, 2020

View reviewed changes

models/imap.go Outdated Show resolved Hide resolved

Tidied up comments and naming conventions

7ecd0ec

Checked out original minified JS files

3e69829

jordan-wright reviewed Jan 16, 2020

View reviewed changes

controllers/api/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 16, 2020

View reviewed changes

db/db_mysql/migrations/20191022184235_0.9.0_imap.sql Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 16, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 16, 2020

View reviewed changes

imap/imap.go Outdated Show resolved Hide resolved

jordan-wright reviewed Jan 16, 2020

View reviewed changes

models/imap.go Show resolved Hide resolved

glennzw and others added 2 commits January 16, 2020 11:43

Update db/db_mysql/migrations/20191022184235_0.9.0_imap.sql

ef4b45f

Co-Authored-By: Jordan Wright <jmwright798@gmail.com>

Small fixes and reverted vendor.min.js

be8edac

Merge branch 'imap' of github.com:glennzw/gophish into imap

0f85995

Renamed database migration files with updated timestamp

2abafc4

jordan-wright merged commit 9de3274 into gophish:master Jan 18, 2020


		log "github.com/gophish/gophish/logger"

		"github.com/glennzw/eazye"

Added IMAP support for checking reported emails #1612

Added IMAP support for checking reported emails #1612

Uh oh!

Conversation

glennzw commented Oct 7, 2019

Uh oh!

quelsan commented Oct 7, 2019

Uh oh!

S0larflare commented Oct 7, 2019

Uh oh!

jordan-wright left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jordan-wright Oct 8, 2019

Choose a reason for hiding this comment

Uh oh!

jordan-wright Oct 8, 2019

Choose a reason for hiding this comment

Uh oh!

glennzw Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

glennzw Oct 28, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordan-wright commented Oct 9, 2019

Uh oh!

glennzw commented Oct 9, 2019

Uh oh!

S0larflare commented Oct 12, 2019 via email

Uh oh!

jordan-wright commented Oct 13, 2019

Uh oh!

glennzw commented Oct 23, 2019

Uh oh!

glennzw commented Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jordan-wright commented Oct 26, 2019

Uh oh!

glennzw commented Oct 28, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glennzw commented Jan 7, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordan-wright commented Jan 8, 2020

Uh oh!

glennzw commented Jan 8, 2020

Uh oh!

jordan-wright commented Jan 15, 2020

Uh oh!

glennzw commented Jan 15, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordan-wright commented Jan 16, 2020

Uh oh!

glennzw commented Jan 16, 2020

Uh oh!

jordan-wright commented Jan 17, 2020

glennzw commented Oct 25, 2019 •

edited

Loading