diff --git a/README.linux-compilation.md b/README.linux-compilation.md new file mode 100644 index 0000000..a732b89 --- /dev/null +++ b/README.linux-compilation.md @@ -0,0 +1,41 @@ +# Compiling instruction for _FastFinder_ on Linux + +_FastFinder_ was originally designed for Windows platform but it also work perfectly on Linux. Unlike other Go programs, if you want to compile or run it from source, you will need to install some libraries and compilation tools. Indeed, _FastFinder_ is strongly dependent of libyara, go-yara and CGO. Here's a little step by step guide: + +## Before installation + +Please ensure having: +* Go >= 1.17 +* GOPATH / GOOS / GOARCH correctly set +* administrator rights to insall + +## Compile YARA + +1/ download YARA latest release source tarball (https://github.com/VirusTotal/yara) +2/ Make sure you have `automake`, `libtool`, `make`, `gcc` and `pkg-config` installed in your system. +2/ unzip and compile yara like this: +``` +tar -zxf yara-.tar.gz +cd . +./bootstrap.sh +./configure +make +make install +``` +3/ Run the test cases to make sure that everything is fine: +``` +make check +``` + +## Configure CGO +CGO will link libyara and compile C instructions used by _Fastfinder_ (through go-yara project). Compiler and linker flags have to be set via the CGO_CFLAGS and CGO_LDFLAGS environment variables like this: +``` +export CGO_CFLAGS="-I/libyara/include" +export CGO_LDFLAGS="-L/libyara/.libs -lyara" +``` + +## You're ready to Go! +You can compile _FastFinder_ with the following command: +``` +go build -tags yara_static -a -ldflags '-s -w' . +``` diff --git a/README.md b/README.md index 5f7e5d8..d3e325c 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,38 @@ # _FastFinder_ - Incident Response - Fast suspicious file finder +[![Golang](https://img.shields.io/badge/Go-1.17-blue.svg)](https://golang.org) +![Linux](https://img.shields.io/badge/Supports-Linux-green.svg) +![windows](https://img.shields.io/badge/Supports-windows-green.svg) ## What is this project designed for? -_FastFinder_ is a lightweight tool made for threat hunting, live forensics and triage on Windows Platform. It is +_FastFinder_ is a lightweight tool made for threat hunting, live forensics and triage on both Windows and Linux Platforms. It is focused on enpoint enumeration and suspicious file finding based on various criterias: * file path / name +* md5 / sha1 / sha256 checksum * simple string content match * complex content condition(s) based on YARA ### Installation Compiled release of this software are available. If you want to compile from sources, it could be a little bit tricky because it strongly depends of -_go-yara_ and CGO compilation. Anyway, you'll find a detailed documentation [here](README.windows-compilation.md) +_go-yara_ and CGO compilation. Anyway, you'll find a detailed documentation [for windows](README.windows-compilation.md) and for [for linux](README.linux-compilation.md) ### Usage ``` -fastfinder [-h|--help] -c|--configuration "" [-b|--build - ""] [-o|--output ""] [-n|--nowindow] +usage: fastfinder [-h|--help] [-c|--configuration "configuration.yaml"] [-b|--build + "path_to_package_bin"] [-o|--output "log_file.log"] [-n|--nowindow] + [-p|--showprogress] [-v|--version] Incident Response - Fast suspicious file finder Arguments: -h --help Print help information - -c --configuration Fastfind configuration file + -c --configuration Fastfind configuration file. Default: configuration.yaml -b --build Output a standalone package with configuration and rules in a single binary -o --output Save fastfinder logs in the specified file -n --nowindow Hide fastfinder window + -p --showprogress Display I/O analysis progress + -v --version Display fastfinder version ``` Depending on where you are looking for files, _FastFinder_ could be used with admin OR simple user rights. @@ -57,8 +64,14 @@ output: * input content grep strings are always case SENSITIVE * backslashes haven't to be escaped on simple string pattern (see example) -## About this project and future versions +## About this project I initially created this project to automate the creation of fastfind on a wide computer network. It fulfills the needs I have today, nevertheless if you have complementary ideas, do not hesitate to ask for, I will see to implement them if they can be useful for everyone. On the other hand, pull request will be studied carefully. + +## Future releases +I don't plan to add any additional features right now. The next release will be focused on: +* Stability / performance improvements +* Unit testing / Code testing coverage / CI +* Build more examples based on live malwares tradecraft and threat actor campaigns \ No newline at end of file diff --git a/README.windows-compilation.md b/README.windows-compilation.md index 295c1e4..964b6cf 100644 --- a/README.windows-compilation.md +++ b/README.windows-compilation.md @@ -1,7 +1,6 @@ +# Compiling instruction for _FastFinder_ on Windows -# Installing _FastFinder_ on Windows - -_FastFinder_ is design for Windows platform but it's a little bit tricky because it's strongly dependant of go-yara and CGO. Here's a little step by step guide: +_FastFinder_ was originally designed for Windows platform but it's a little bit tricky to compile because it's strongly dependant of go-yara and CGO. Here's a little step by step guide: ## Before installation diff --git a/examples/example_configuration_linux.yaml b/examples/example_configuration_linux.yaml new file mode 100644 index 0000000..98b1225 --- /dev/null +++ b/examples/example_configuration_linux.yaml @@ -0,0 +1,19 @@ +input: + path: [] + content: + grep: [] + yara: + - './examples/example_rule_linux.yar' + checksum: + - 'bf1cde9c94c301cdc3b5486f2f3fe66b' + - '41ba1bd49cb22466e422098d184bd4267ef9529e' + - 'e875b1185577ff872fbaabde481cc196af03745c530403c8303f00fe35859bf7' +options: + contentMatchDependsOnPathMatch: false + findInHardDrives: true + findInRemovableDrives: false + findInNetworkDrives: false + findInCDRomDrives: false +output: + base64Files: true + filesCopyPath: '' diff --git a/examples/example_rule_linux.yar b/examples/example_rule_linux.yar new file mode 100644 index 0000000..a93f549 --- /dev/null +++ b/examples/example_rule_linux.yar @@ -0,0 +1,14 @@ +rule fastfinder_example{ + meta: + name = "fastfinder_example" + description = "Example of fastfinder yara match (on legitimate linux 'more' binary)" + reference = "https://github.com/codeyourweb/fastfinder" + strings: + $str1 = "GNU" + $str3 = "--More--" + $str4 = "file perusal filter for CRT viewing" + $str5 = "Press 'h' for instructions" + $op = { ba 05 00 00 00 31 ff 4? 8d 35 ?? ?? ?? ?? e8 ?? ?? ?? ?? 4? 89 ee 4? 89 c7 e8 ?? ?? ?? ?? ba 05 00 00 00 31 ff 4? 8d 35 ?? ?? ?? ?? e8 ?? ?? ?? ??} + condition: + all of them and uint16(0) == 0x457f +} \ No newline at end of file diff --git a/finder.go b/finder.go index a1e3653..3752bb0 100644 --- a/finder.go +++ b/finder.go @@ -14,9 +14,11 @@ import ( // PathsFinder try to match regular expressions in file paths slice func PathsFinder(files *[]string, patterns []*regexp2.Regexp) *[]string { + InitProgressbar(int64(len(*files))) var matchingFiles []string for _, expression := range patterns { for _, f := range *files { + ProgressBarStep() if match, _ := expression.MatchString(f); match { matchingFiles = append(matchingFiles, f) } @@ -29,7 +31,9 @@ func PathsFinder(files *[]string, patterns []*regexp2.Regexp) *[]string { // FindInFiles check for pattern or checksum match in files slice func FindInFiles(files *[]string, patterns []string, checksum []string) *[]string { var matchingFiles []string + InitProgressbar(int64(len(*files))) for _, f := range *files { + ProgressBarStep() b, err := ioutil.ReadFile(f) if err != nil { LogMessage(LOG_ERROR, "[ERROR]", "Unable to read file", f) diff --git a/go.mod b/go.mod index 74969e8..503e215 100644 --- a/go.mod +++ b/go.mod @@ -7,8 +7,19 @@ require ( github.com/gen2brain/go-unarr v0.1.2 github.com/h2non/filetype v1.1.3 github.com/hillu/go-yara/v4 v4.1.0 - golang.org/x/sys v0.0.0-20211124211545-fe61309f8881 + golang.org/x/sys v0.0.0-20211205182925-97ca703d548d gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b ) -require github.com/dlclark/regexp2 v1.4.0 +require ( + github.com/dlclark/regexp2 v1.4.0 + github.com/schollz/progressbar/v3 v3.8.3 +) + +require ( + github.com/mattn/go-runewidth v0.0.13 // indirect + github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect + github.com/rivo/uniseg v0.2.0 // indirect + golang.org/x/crypto v0.0.0-20211202192323-5770296d904e // indirect + golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect +) diff --git a/go.sum b/go.sum index f0f97b6..842aa2d 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,8 @@ github.com/akamensky/argparse v1.3.1 h1:kP6+OyvR0fuBH6UhbE6yh/nskrDEIQgEA1SUXDPjx4g= github.com/akamensky/argparse v1.3.1/go.mod h1:S5kwC7IuDcEr5VeXtGPRVZ5o/FdhcMlQz4IZQuw64xA= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dlclark/regexp2 v1.4.0 h1:F1rxgk7p4uKjwIQxBs9oAXe5CqrXlCduYEJvrF4u93E= github.com/dlclark/regexp2 v1.4.0/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc= github.com/gen2brain/go-unarr v0.1.2 h1:17kYZ2WMCVFrnmU4A+7BeFXblIOyE8weqggjay+kVIU= @@ -8,8 +11,40 @@ github.com/h2non/filetype v1.1.3 h1:FKkx9QbD7HR/zjK1Ia5XiBsq9zdLi5Kf3zGyFTAFkGg= github.com/h2non/filetype v1.1.3/go.mod h1:319b3zT68BvV+WRj7cwy856M2ehB3HqNOt6sy1HndBY= github.com/hillu/go-yara/v4 v4.1.0 h1:ZLT9ar+g5r1IgEp1QVYpdqYCgKMNm7DuZYUJpHZ3yUI= github.com/hillu/go-yara/v4 v4.1.0/go.mod h1:rkb/gSAoO8qcmj+pv6fDZN4tOa3N7R+qqGlEkzT4iys= -golang.org/x/sys v0.0.0-20211124211545-fe61309f8881 h1:TyHqChC80pFkXWraUUf6RuB5IqFdQieMLwwCJokV2pc= -golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +github.com/k0kubun/go-ansi v0.0.0-20180517002512-3bf9e2903213/go.mod h1:vNUNkEQ1e29fT/6vq2aBdFsgNPmy8qMdSay1npru+Sw= +github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= +github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU= +github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w= +github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ= +github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= +github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= +github.com/schollz/progressbar/v3 v3.8.3 h1:FnLGl3ewlDUP+YdSwveXBaXs053Mem/du+wr7XSYKl8= +github.com/schollz/progressbar/v3 v3.8.3/go.mod h1:pWnVCjSBZsT2X3nx9HfRdnCDrpbevliMeoEVhStwHko= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20211202192323-5770296d904e h1:MUP6MR3rJ7Gk9LEia0LP2ytiH6MuCfs7qYz+47jGdD8= +golang.org/x/crypto v0.0.0-20211202192323-5770296d904e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= +golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210910150752-751e447fb3d0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211205182925-97ca703d548d h1:FjkYO/PPp4Wi0EAUOVLxePm7qVW4r4ctbWpURyuOD0E= +golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo= diff --git a/logger.go b/logger.go index d2b12b3..f7cc2bc 100644 --- a/logger.go +++ b/logger.go @@ -43,7 +43,7 @@ func StdoutToLogFile(outLogPath string) { scanner := bufio.NewScanner(rd) for scanner.Scan() { stdoutLine := scanner.Text() - multiWriter.Write([]byte(stdoutLine + "\r\n")) + multiWriter.Write([]byte(stdoutLine + LineBreak)) } }() } @@ -68,7 +68,7 @@ func StderrToLogFile(outLogPath string) { scanner := bufio.NewScanner(rd) for scanner.Scan() { stdoutLine := scanner.Text() - multiWriter.Write([]byte(stdoutLine + "\r\n")) + multiWriter.Write([]byte(stdoutLine + LineBreak)) } }() } diff --git a/main.go b/main.go index b2ca5b3..0c7dd77 100644 --- a/main.go +++ b/main.go @@ -5,8 +5,12 @@ package main import ( + "fmt" "log" "os" + "runtime" + "sort" + "strings" "github.com/akamensky/argparse" "github.com/dlclark/regexp2" @@ -19,37 +23,51 @@ func main() { var err error if _, err = CreateMutex("fastfinder"); err != nil { - LogMessage(LOG_ERROR, "[ERROR]", "Only one instance or fastfinder can be launched") + LogMessage(LOG_ERROR, "[ERROR]", "Only one instance or fastfinder can be launched:", err.Error()) os.Exit(1) } // parse configuration file - parser := argparse.NewParser("fastfinder", "(v1.2) Incident Response - Fast suspicious file finder") - configPath := parser.String("c", "configuration", &argparse.Options{Required: true, Default: "configuration.yaml", Help: "Fastfind configuration file"}) - sfxPath := parser.String("b", "build", &argparse.Options{Required: false, Help: "Output a standalone package with configuration and rules in a single binary"}) - outLogPath := parser.String("o", "output", &argparse.Options{Required: false, Help: "Save fastfinder logs in the specified file"}) - hideWindow := parser.Flag("n", "nowindow", &argparse.Options{Required: false, Help: "Hide fastfinder window"}) + parser := argparse.NewParser("fastfinder", "Incident Response - Fast suspicious file finder") + pConfigPath := parser.String("c", "configuration", &argparse.Options{Required: false, Default: "configuration.yaml", Help: "Fastfind configuration file"}) + pSfxPath := parser.String("b", "build", &argparse.Options{Required: false, Help: "Output a standalone package with configuration and rules in a single binary"}) + pOutLogPath := parser.String("o", "output", &argparse.Options{Required: false, Help: "Save fastfinder logs in the specified file"}) + pHideWindow := parser.Flag("n", "nowindow", &argparse.Options{Required: false, Help: "Hide fastfinder window"}) + pShowProgress := parser.Flag("p", "showprogress", &argparse.Options{Required: false, Help: "Display I/O analysis progress"}) + pFinderVersion := parser.Flag("v", "version", &argparse.Options{Required: false, Help: "Display fastfinder version"}) err = parser.Parse(os.Args) if err != nil { log.Fatal(parser.Usage(err)) } + // version + if *pFinderVersion { + fmt.Println("fastfinder v1.3b") + if !Contains(os.Args, "-c") && !Contains(os.Args, "--configuration") { + os.Exit(0) + } + } + + // progressbar + EnableProgressbar(*pShowProgress) + + // configuration parsing var config Configuration - config.getConfiguration(*configPath) + config.getConfiguration(*pConfigPath) if config.Output.FilesCopyPath != "" { config.Output.FilesCopyPath = "./" } // window hidden - if *hideWindow { + if *pHideWindow { HideConsoleWindow() } // init file logging - if len(*outLogPath) > 0 { - StdoutToLogFile(*outLogPath) - StderrToLogFile(*outLogPath) + if len(*pOutLogPath) > 0 { + StdoutToLogFile(*pOutLogPath) + StderrToLogFile(*pOutLogPath) } // check for input configuration @@ -59,9 +77,13 @@ func main() { } // sfx building option - if len(*sfxPath) > 0 { - BuildSFX(config, *sfxPath, *outLogPath, *hideWindow) - LogMessage(LOG_INFO, "[INFO]", "package generated successfully at", *sfxPath) + if len(*pSfxPath) > 0 { + if runtime.GOOS != "windows" { + LogMessage(LOG_ERROR, "[ERROR]", "Standalone package can be built only on Windows") + os.Exit(1) + } + BuildSFX(config, *pSfxPath, *pOutLogPath, *pHideWindow) + LogMessage(LOG_INFO, "[INFO]", "package generated successfully at", *pSfxPath) os.Exit(0) } @@ -84,18 +106,35 @@ func main() { // drives enumeration LogMessage(LOG_INFO, "[INIT]", "Enumerating drives") var basePaths []string - drives := EnumLogicalDrives() + drives, excludedPaths := EnumLogicalDrives() if len(drives) == 0 { LogMessage(LOG_ERROR, "[ERROR]", "Unable to find drives") os.Exit(1) } + for _, drive := range drives { if (drive.Type == DRIVE_REMOVABLE && config.Options.FindInRemovableDrives) || (drive.Type == DRIVE_FIXED && config.Options.FindInHardDrives) || (drive.Type == DRIVE_REMOTE && config.Options.FindInNetworkDrives) || (drive.Type == DRIVE_CDROM && config.Options.FindInCDRomDrives) { - basePaths = append(basePaths, drive.Name+":\\") + if runtime.GOOS == "windows" || len(basePaths) == 0 { + basePaths = append(basePaths, drive.Name) + } else { + alreadyParsed := false + for _, p := range basePaths { + if len(drive.Name) > len(p) && !strings.HasPrefix(drive.Name, p) { + alreadyParsed = true + } + } + if !alreadyParsed { + basePaths = append(basePaths, drive.Name) + } + } + } else { + if runtime.GOOS != "windows" { + excludedPaths = append(excludedPaths, drive.Name) + } } } @@ -103,23 +142,43 @@ func main() { LogMessage(LOG_ERROR, "[ERROR]", "No drive corresponding to your configuration drive type") os.Exit(1) } else { - LogMessage(LOG_INFO, "[INIT]", "Looking for the following drives", basePaths) + LogMessage(LOG_INFO, "[INIT]", "Looking for the following drives:") + for _, p := range basePaths { + LogMessage(LOG_INFO, " |", p) + } + } + + if len(excludedPaths) > 0 { + LogMessage(LOG_INFO, "[INFO]", "Excluding the following paths:") + for _, p := range excludedPaths { + LogMessage(LOG_INFO, " |", p) + } } - LogMessage(LOG_INFO, "[INIT]", "Looking for the following paths patterns:") - for _, p := range config.Input.Path { - LogMessage(LOG_INFO, " |", p) + if len(config.Input.Path) > 0 { + LogMessage(LOG_INFO, "[INIT]", "Looking for the following paths patterns:") + for _, p := range config.Input.Path { + LogMessage(LOG_INFO, " |", p) + } + } + + if runtime.GOOS != "windows" { + sort.Slice(basePaths, func(i, j int) bool { + return len(basePaths[i]) > len(basePaths[j]) + }) } // start main routine - LogMessage(LOG_INFO, "[INFO]", "Enumerating files") for _, basePath := range basePaths { - LogMessage(LOG_INFO, "[INFO]", "Looking for files in", basePath) + LogMessage(LOG_INFO, "[INFO]", "Enumerating files in", basePath) var matchContent *[]string var matchPattern *[]string // files listing - files := ListFilesRecursively(basePath) + files := ListFilesRecursively(basePath, excludedPaths) + if runtime.GOOS != "windows" { + excludedPaths = append(excludedPaths, basePath) + } // match file path if len(config.Input.Path) > 0 { @@ -154,14 +213,18 @@ func main() { if len(config.Input.Content.Yara) > 0 { LogMessage(LOG_INFO, "[INFO]", "Checking for yara matchs in", basePath) if config.Options.ContentMatchDependsOnPathMatch { + InitProgressbar(int64(len(*matchPattern))) for _, file := range *matchPattern { + ProgressBarStep() if FileAnalyzeYaraMatch(file, rules) && !Contains(*matchContent, file) { LogMessage(LOG_INFO, "[ALERT]", "File match on", file) *matchContent = append(*matchContent, file) } } } else { + InitProgressbar(int64(len(*files))) for _, file := range *files { + ProgressBarStep() if FileAnalyzeYaraMatch(file, rules) && !Contains(*matchContent, file) { LogMessage(LOG_INFO, "[ALERT]", "File match on", file) *matchContent = append(*matchContent, file) @@ -176,7 +239,7 @@ func main() { } // handle false condition on ContentMatchDependsOnPathMatch options - if !config.Options.ContentMatchDependsOnPathMatch { + if len(config.Input.Path) > 0 && !config.Options.ContentMatchDependsOnPathMatch { for _, p := range *matchPattern { if !Contains(*matchContent, p) { *matchContent = append(*matchContent, p) @@ -185,9 +248,15 @@ func main() { } // copy matching files - LogMessage(LOG_INFO, "[INFO]", "Copy all matching files") - for _, f := range *matchContent { - FileCopy(f, config.Output.FilesCopyPath, config.Output.Base64Files) + if len(*matchContent) > 0 { + LogMessage(LOG_INFO, "[INFO]", "Copy all matching files") + InitProgressbar(int64(len(*matchPattern))) + for _, f := range *matchContent { + ProgressBarStep() + FileCopy(f, config.Output.FilesCopyPath, config.Output.Base64Files) + } + } else { + LogMessage(LOG_INFO, "[INFO]", "No match found") } } } diff --git a/progressbar.go b/progressbar.go new file mode 100644 index 0000000..b37e922 --- /dev/null +++ b/progressbar.go @@ -0,0 +1,22 @@ +package main + +import "github.com/schollz/progressbar/v3" + +var progressbarEnabled bool +var bar *progressbar.ProgressBar + +func EnableProgressbar(enable bool) { + progressbarEnabled = enable +} + +func InitProgressbar(value int64) { + if progressbarEnabled { + bar = progressbar.Default(value) + } +} + +func ProgressBarStep() { + if progressbarEnabled { + bar.Add(1) + } +} diff --git a/utils_common.go b/utils_common.go index 0a6973c..e4e5c14 100644 --- a/utils_common.go +++ b/utils_common.go @@ -44,7 +44,7 @@ func GetEnvironmentVariables() (environmentVariables []Env) { } // ListFilesRecursively returns a list of files in the specified path and its subdirectories -func ListFilesRecursively(path string) *[]string { +func ListFilesRecursively(path string, excludedPaths []string) *[]string { var files []string err := filepath.Walk(path, func(path string, f os.FileInfo, err error) error { @@ -54,6 +54,13 @@ func ListFilesRecursively(path string) *[]string { } if !f.IsDir() { + for _, excludedPath := range excludedPaths { + if len(excludedPath) > 1 && strings.HasPrefix(path, excludedPath) && len(path) > len(excludedPath) { + LogMessage(LOG_INFO, "[INFO]", "Skipping dir", path) + return filepath.SkipDir + } + } + files = append(files, path) } return nil diff --git a/utils_linux.go b/utils_linux.go index 3c099b1..e1be9b7 100644 --- a/utils_linux.go +++ b/utils_linux.go @@ -3,32 +3,200 @@ package main import ( - "encoding/base64" + "bufio" + "bytes" + "fmt" "io" - "log" + "io/ioutil" "os" - "path/filepath" + "os/exec" + "regexp" + "strconv" "strings" "syscall" - "unsafe" + "time" ) -type Env struct { - Name string - Value string +const LineBreak = "\n" + +type disks map[string]map[string]string +type cmdRunner struct{} + +func New() *cmdRunner { + return &cmdRunner{} +} + +func (c *cmdRunner) Run(cmd string, args []string) (io.Reader, error) { + command := exec.Command(cmd, args...) + resCh := make(chan []byte) + errCh := make(chan error) + go func() { + out, err := command.CombinedOutput() + if err != nil { + errCh <- err + } + resCh <- out + }() + timer := time.After(2 * time.Second) + select { + case err := <-errCh: + return nil, err + case res := <-resCh: + return bytes.NewReader(res), nil + case <-timer: + return nil, fmt.Errorf("time out (cmd:%v args:%v)", cmd, args) + } } // HideConsoleWindow hide the process console window func HideConsoleWindow() { - LogMessage(LOG_INFO, "[COMPAT]", "Hide console not implented on linux. You should consider run this program as a task") + LogMessage(LOG_INFO, "[COMPAT]", "Hide console option not implented on linux. You should consider run this program as a task") } // CreateMutex creates a named mutex to avoid multiple instance run func CreateMutex(name string) (uintptr, error) { - return 0, nil + lockFile := "fastfinder.lock" + currentPid := os.Getpid() + + lockContent, err := ioutil.ReadFile(lockFile) + if err == nil { + if len(lockContent) > 0 && string(lockContent) != fmt.Sprintf("%d", currentPid) { + lockProcessId, _ := strconv.Atoi(string(lockContent)) + process, err := os.FindProcess(lockProcessId) + if err == nil { + pSignal := process.Signal(syscall.Signal(0)) + if pSignal == nil { + return uintptr(currentPid), fmt.Errorf("another instance of fastfinder is running") + } + } + } + } + + f, err := os.OpenFile(lockFile, os.O_CREATE|os.O_RDWR, 0664) + if err != nil { + return 0, fmt.Errorf("cannot instanciate mutex") + } + defer f.Close() + f.Write([]byte(fmt.Sprintf("%d", currentPid))) + + return uintptr(currentPid), nil } // EnumLogicalDrives returns a list of all logical drives letters on the system. -func EnumLogicalDrives() (drivesInfo []DriveInfo) { - return drivesInfo +func EnumLogicalDrives() (drivesInfo []DriveInfo, excludedPaths []string) { + excludedPaths = []string{"/dev", "/lost+found", "/proc", "/media/floppy"} + disks, err := Lsblk() + if err != nil { + LogMessage(LOG_ERROR, "[COMPAT]", "Error getting disks info: %v - try to parse from /", err) + } else { + // Get fixed / removable / cdrom drives + for _, disk := range disks { + if disk["mountpoint"] != "" { + switch disk["type"] { + case "part": + if IsUSBStorage("/dev/" + disk["name"]) { + drivesInfo = append(drivesInfo, DriveInfo{Name: disk["mountpoint"], Type: DRIVE_REMOVABLE}) + } else { + drivesInfo = append(drivesInfo, DriveInfo{Name: disk["mountpoint"], Type: DRIVE_FIXED}) + } + case "rom": + drivesInfo = append(drivesInfo, DriveInfo{Name: disk["mountpoint"], Type: DRIVE_CDROM}) + default: + excludedPaths = append(excludedPaths, disk["mountpoint"]) + + } + } + } + } + + // Get network drives + nDrives, err := FindInNetworkDrives() + if err != nil { + LogMessage(LOG_ERROR, "[COMPAT]", "Error getting network drives: %v", err) + } + + for _, driveName := range nDrives { + drivesInfo = append(drivesInfo, DriveInfo{Name: driveName, Type: DRIVE_REMOTE}) + } + + return drivesInfo, excludedPaths +} + +// FindInNetworkDrives uses df -aT and returns a list of all valid fuse mount +func FindInNetworkDrives() (mounts []string, err error) { + out, err := exec.Command("df", "-aT").Output() + if err != nil { + return mounts, err + } + + outlines := strings.Split(string(out), "\n") + + for _, line := range outlines { + parsedLine := strings.Fields(line) + if len(line) > 5 && + strings.HasPrefix(parsedLine[len(parsedLine)-1], "/") && + (strings.Contains(parsedLine[0], "fuse") || strings.Contains(parsedLine[1], "fuse")) { + a, _ := strconv.ParseInt(parsedLine[3], 10, 64) + b, _ := strconv.ParseInt(parsedLine[4], 10, 64) + if a > 0 && b > 0 { + mounts = append(mounts, parsedLine[len(parsedLine)-1]) + } + } + } + + return mounts, nil +} + +// Lsblk returns a map of all disks and their properties +func Lsblk() (disks, error) { + var cmdrun = cmdRunner{} + rr, err := cmdrun.Run("lsblk", []string{"-P", "-b", "-o", "NAME,TYPE,MOUNTPOINT"}) + if err != nil { + return nil, err + } + disks := parser_lsblk(rr) + return disks, nil +} + +func parser_lsblk(r io.Reader) map[string]map[string]string { + var lsblk = make(disks) + re := regexp.MustCompile("([A-Z]+)=(?:\"(.*?)\")") + scan := bufio.NewScanner(r) + for scan.Scan() { + var disk_name = "" + disk := make(map[string]string) + raw := scan.Text() + sr := re.FindAllStringSubmatch(raw, -1) + for i, k := range sr { + k[1] = strings.ToLower(k[1]) + if i == 0 { + disk_name = k[2] + } + + if Contains([]string{"name", "type", "mountpoint"}, k[1]) { + disk[k[1]] = k[2] + } + } + lsblk[disk_name] = disk + } + return lsblk +} + +// IsUSBStorage returns true if the given device is a USB storage based on udevadm linux command return +func IsUSBStorage(device string) bool { + deviceVerifier := "ID_USB_DRIVER=usb-storage" + cmd := "udevadm" + args := []string{"info", "-q", "property", "-n", device} + out, err := exec.Command(cmd, args...).Output() + + if err != nil { + LogMessage(LOG_ERROR, "[ERROR]", "Error checking device %s: %s", device, err) + return false + } + + if strings.Contains(string(out), deviceVerifier) { + return true + } + + return false } diff --git a/utils_windows.go b/utils_windows.go index 673ae06..2d57ce3 100644 --- a/utils_windows.go +++ b/utils_windows.go @@ -9,6 +9,8 @@ import ( "golang.org/x/sys/windows" ) +const LineBreak = "\r\n" + var ( modKernel32 = windows.NewLazySystemDLL("kernel32.dll") modUser32 = windows.NewLazySystemDLL("user32.dll") @@ -46,20 +48,20 @@ func CreateMutex(name string) (uintptr, error) { } // EnumLogicalDrives returns a list of all logical drives letters on the system. -func EnumLogicalDrives() (drivesInfo []DriveInfo) { +func EnumLogicalDrives() (drivesInfo []DriveInfo, excludedPaths []string) { var drives []string if ret, _, callErr := procGetLogicalDrives.Call(); callErr != windows.ERROR_SUCCESS { - return []DriveInfo{} + return []DriveInfo{}, []string{} } else { drives = bitsToDrives(uint32(ret)) } for _, drive := range drives { var driveInfo DriveInfo - driveInfo.Name = drive + driveInfo.Name = drive + ":\\" drivePtr, err := syscall.UTF16PtrFromString(drive + ":") if err != nil { - return drivesInfo + return drivesInfo, []string{} } if ret, _, callErr := procGetDriveTypeW.Call(uintptr(unsafe.Pointer(drivePtr))); callErr != windows.ERROR_SUCCESS { @@ -71,7 +73,7 @@ func EnumLogicalDrives() (drivesInfo []DriveInfo) { drivesInfo = append(drivesInfo, driveInfo) } - return drivesInfo + return drivesInfo, []string{} } // map drive DWORD returned by EnumLogicalDrives to drive letters diff --git a/yaraprocessing.go b/yaraprocessing.go index f9c9f3e..09e398f 100644 --- a/yaraprocessing.go +++ b/yaraprocessing.go @@ -13,41 +13,38 @@ import ( ) // PerformYaraScan use provided YARA rules and search for match in the given byte slice -func PerformYaraScan(data *[]byte, rules *yara.Rules) yara.MatchRules { +func PerformYaraScan(data *[]byte, rules *yara.Rules) (yara.MatchRules, error) { result, err := yaraScan(*data, rules) if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", err) + return nil, err } - return result + return result, nil } // PerformArchiveYaraScan try to decompress archive and YARA scan every file in it -func PerformArchiveYaraScan(path string, rules *yara.Rules) (matchs yara.MatchRules) { +func PerformArchiveYaraScan(path string, rules *yara.Rules) (matchs yara.MatchRules, err error) { var buffer [][]byte a, err := unarr.NewArchive(path) if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", err) - return matchs + return nil, err } defer a.Close() list, err := a.List() if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", err) - return matchs + return nil, err } for _, f := range list { err := a.EntryFor(f) if err != nil { - return matchs + return nil, err } data, err := a.ReadAll() if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", err) - return matchs + return nil, err } buffer = append(buffer, data) @@ -55,10 +52,10 @@ func PerformArchiveYaraScan(path string, rules *yara.Rules) (matchs yara.MatchRu matchs, err = yaraScan(bytes.Join(buffer, []byte{}), rules) if err != nil { - return matchs + return nil, err } - return matchs + return matchs, nil } // LoadYaraRules compile yara rules from specified paths and return a pointer to the yara compiler @@ -109,34 +106,45 @@ func FileAnalyzeYaraMatch(path string, rules *yara.Rules) bool { if _, err = os.Stat(path); err != nil { LogMessage(LOG_ERROR, "[ERROR]", path, err) - } else { - // read file content - content, err = os.ReadFile(path) - if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", path, err) - } + return false + } - filetype, err := filetype.Match(content) - if err != nil { - LogMessage(LOG_ERROR, "[ERROR]", path, err) - } + // read file content + content, err = os.ReadFile(path) + if err != nil { + LogMessage(LOG_ERROR, "[ERROR]", path, err) + return false + } - // cleaning memory if file size is greater than 512Mb - if len(content) > 1024*1024*512 { - defer debug.FreeOSMemory() - } + filetype, err := filetype.Match(content) + if err != nil { + LogMessage(LOG_ERROR, "[ERROR]", path, err) + return false + } + + // cleaning memory if file size is greater than 512Mb + if len(content) > 1024*1024*512 { + defer debug.FreeOSMemory() + } + + // cancel analysis if file size is greater than 2Gb + if len(content) > 1024*1024*2048 { + LogMessage(LOG_ERROR, "File size is greater than 2Gb, skipping", path) + return false + } - // cancel analysis if file size is greater than 2Gb - if len(content) > 1024*1024*2048 { - LogMessage(LOG_ERROR, "File size is greater than 2Gb, skipping", path) + // archive or other file format scan + if Contains([]string{"application/x-tar", "application/x-7z-compressed", "application/zip", "application/vnd.rar"}, filetype.MIME.Value) { + result, err = PerformArchiveYaraScan(path, rules) + if err != nil { + LogMessage(LOG_ERROR, "[ERROR]", "Error performing yara scan on", path, err) return false } - - // archive or other file format scan - if Contains([]string{"application/x-tar", "application/x-7z-compressed", "application/zip", "application/vnd.rar"}, filetype.MIME.Value) { - result = PerformArchiveYaraScan(path, rules) - } else { - result = PerformYaraScan(&content, rules) + } else { + result, err = PerformYaraScan(&content, rules) + if err != nil { + LogMessage(LOG_ERROR, "[ERROR]", "Error performing yara scan on", path, err) + return false } }