_block:xyz_ and _blocker_ to better manage command queue dependencies

[codingo]

One of the problems with trying to automate a number of tasks is dependancies. The idea I have to work around this is _block:<name> and _blocker_.

This will work in the following manner:

Argument	Description
_blocker_	All commands before this must be completed before any other items can be added into the queue
_block:_	All commands in a block are run and queued sequentially

Take the following runfile, as an example:

_block:file-creation_
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
_block:file-creation_
_blocker_
nmap _target_ -oN _target_/output/scans/_target_-nmap

In this example, because the items in the _block_ are always loaded sequentially they shouldn't ever run out of order. Additionally, because we have a blocker before nmap, this item won't run until the block is finished.

Alternatively, we could have a command list like the following:

_block:nmap_
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
nmap _target_ -oN _target_/output/scans/_target_-nmap
_block:nmap_
nikto --host _target_

In this example, the block would run the same as before, but assuming the thread count is high enough then nikto would begin to run immediately, passing results back to the terminal (whilst nmap and file creation happened in the background).

[codingo]

Blocks can be nested since everything in a block is sequential anyway, nesting will make no difference. That said, a warning should be shown if a block is not closed. This shouldn't need to be an exception, but the user should be made aware since everything after the block would then run sequentially.

[codingo]

In case it's not clear from the above as well, it's important that the user can add multiple blocks to a single job. The idea of this is that the threader will optimize by taking jobs off the command list file, but by having blocks you can have your sequential tasks within the same file (instead of making shell scripts that run outside of the command queue).

[iamOgunyinka]

So I have an example I'd like clarity on. Suppose there are 8 tasks scheduled to run:

echo hello
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
copy foo bar/
pushd /usr/home/codingo/
popd

and should be run on 5 threads/workers. The default behavior of your implementation is to pick 5 tasks first(not sure if the order of appearance matters?) and on completion, run the other 3. I think that's OK since the tasks seem to be independent of one another.

Now, enter the _block:<name>_ block(let's call it TaskBlock) and the sequence-before segment _blocker_. Given the 8 tasks above, if a _blocker_ is indicated before copy foo bar/ like so,

echo hello
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
_blocker_
copy foo bar/
pushd /usr/home/codingo/
popd

• does that mean all operations immediately after the _blocker_ segment (starting with copy foo bar/) must wait until the tasks above the _blocker_ is completed?
• When the tasks above the _blocker_ segment is completed, does the execution of the tasks follow the sequential order or they can be multi-threaded again before the next blocker is encountered?
• What is the behavior of the tasks in a TaskBlock? Guaranteed sequential or multi-threaded?

[codingo]

Gotcha, so the intended functionality is that once a _blocker_ is hit, queuing of those tasks will be halted until the blocker is resolved. Once it's resolved commands may queue again and do not need to be completed sequentially. Items before a _blocker_ do not need to run sequentially, only items within a _block.

Items within a taskblock are guaranteed to be sequential.

[codingo]

Currently being tested in #55

[codingo]

Closed with #60, bounties to be awarded to #59 and #60