No clear documentation for creating a aws-cli plugin

[RichardBronosky]

I've done code spelunking and found interesting ideas in:

aws-cli/awscli/plugin.py

Lines 25 to 27 in c0d866a

	:type plugin_mapping: dict
	:param plugin_mapping: A dict of plugin name to import path,
	e.g. ``{"plugingName": "package.modulefoo"}``.

aws-cli/awscli/customizations/s3/s3.py

Lines 23 to 26 in 5ce60be

	This function is require to use the plugin. It calls the functions
	required to add all neccessary commands and parameters to the CLI.
	This function is necessary to install the plugin using a configuration
	file

aws-cli/awscli/customizations/commands.py

Lines 47 to 49 in 37aac25

	# This is the name of your command, so if you want to
	# create an 'aws mycommand ...' command, the NAME would be
	# 'mycommand'

and finally

aws-cli/awscli/clidriver.py

Line 55 in c0d866a

load_plugins(session.full_config.get('plugins', {}),

which lead me to try:

$ aws configure set plugins '{"mycommand": "awsplugins.mycommand"}'
$ cat ~/.aws/config
[default]
region = us-east-1
output = json
plugins = {"mycommand": "awsplugins.mycommand"}
$ aws help | grep mycommand
$ aws mycommand
usage: aws [options] <command> <subcommand> [parameters]
aws: error: argument command: Invalid choice, valid choices are:

autoscaling                              | cloudformation
cloudfront                               | cloudhsm
cloudsearch                              | cloudsearchdomain
cloudtrail                               | cloudwatch
cognito-identity                         | cognito-sync
datapipeline                             | directconnect
dynamodb                                 | ec2
ecs                                      | elasticache
elasticbeanstalk                         | elastictranscoder
elb                                      | emr
glacier                                  | iam
importexport                             | kinesis
kms                                      | lambda
logs                                     | opsworks
rds                                      | redshift
route53                                  | route53domains
sdb                                      | ses
sns                                      | sqs
ssm                                      | storagegateway
sts                                      | support
swf                                      | s3api
s3                                       | configure
deploy                                   | configservice
help

Are there any examples of a working aws-cli plugin external to the aws-cli package? Is it supposed to be possible?

[jamesls]

The big caveat is that the plugin API is subject to change which is the main reason it's not documented yet. We do have on our backlog plans to flesh out the APIs and get proper documentation around plugins. But here's the main points if you want to try it out:

In the ~/.aws/config file you'll need to add a plugins section that has the name of the package to import:

[plugins]
foo = myplugin

Then you'll need to make it so that the right hand side, in this case myplugin is importable, (i.e import myplugin works).

You then need to define an entry point, awscli_initialize that takes a single argument.

At this point, you can look at some of the built in customizations to get an idea for how these work. Just about all the code in awscli/customizations uses this interface (though instead of awscli_initialize, they typically use their own function names). For example, the cloudtrail customization has a good example of how to add a new command. The only difference is instead of initialize, you'd need to call the function awscli_initialize.

Feel free to experiment and give us feedback. Just keep in mind there's no backwards compatibility guarantee for the plugin API (yet) and it will likely change in the future. Let me know if you have any other questions.

[jamesls]

Like there was actually a regression in a recent release for the "plugins" section being set via the configure set command (#1263). Once this is fixed, you can also configure plugins via: aws configure set plugins.name modulename

[RichardBronosky]

Thanks, @jamesls! That helps a lot. I've run into a challenge that I hope you can help me figure out. I've packaged my work up into https://github.com/RichardBronosky/aws-cli-plugins

The good news is that my awscli_initialize is being called! Unfortunately, my call to cli.register seems to do nothing. I've tried passing an inject_commands function like I've seen in https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudtrail.py#L35 and tried passing the add_command method that my class inherits from BasicCommand like I've seen in https://github.com/aws/aws-cli/blob/master/awscli/customizations/configure.py#L29-L31

I'm trying to get my head around https://github.com/boto/botocore/blob/develop/botocore/hooks.py#L297 but it seems like it's going to require a complete understanding of botocore. I think my problem may have something to do with my use of register vs. register_first or register_last. Or maybe the event_name. I did try calling-command.helloworld as seen in https://github.com/aws/aws-cli/blob/master/awscli/customizations/cloudsearchdomain.py#L22 with no effect.

Can you point me in the right direction to get my command registered?

[RichardBronosky]

RichardBronosky/aws-cli-plugins@d45711e
Not sure if this is the right thing to do, but I found that calling cli.register with 'building-command-table.main' gets all the way through inject_commands and HelloWorld.__ini__

It seems that I have some confusion about where "helloworld" and "say-hello" commands and subcommands are separated. A little more Trial & Exception should get me there.

I'm seeing a lot of results here https://github.com/aws/aws-cli/search?q=building-command-table.main but, if you can confirm to me that 'building-command-table.main' is the right approach, that'd help.

[jamesls]

Yes, building-command-table.main is the event you'll need to add a command to the main command table (i.e the commands available after aws). After that, to add subcommands to a command (the commands available after aws <firstcommand>), then the event name will be building-command-table.yourcommandname.

As for the way the events works, the main difference between this system and other publish/subscribe event systems is that the CLI events are hierarchical. It still works with exact event names as well. So for example, if I register for an event, when that specific event is emitted, my handler will be called:

>>> import botocore.session
>>> s = botocore.session.get_session()
>>> def myhandler(**kwargs):
...   print("Called with kwargs:", kwargs)
...
>>>
>>> s.register('foo.bar', myhandler)
>>>
>>> s.emit('foo.bar', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None)]

However, I can also just register for foo, and my handler will be called when foo is emitted, or when foo.bar is emitted, or foo.bar.anything.else:

>>> s.register('foo', myhandler)
>>>
>>> s.emit('foo.bar', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None), (<function myhandler at 0x108e55050>, None)]

>>> s.emit('foo.bar.bar.anything', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar.bar.anything', 'arg1': 'foo', 'arg2': 'bar'})
('Called with kwargs:', {'event_name': 'foo.bar.bar.anything', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None), (<function myhandler at 0x108e55050>, None)]

>>> s.emit('foo.anything.else', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.anything.else', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None)]

The main reason do this is because event names in the CLI/botocore have names such as before-call.s3.ListObjects, which is emitted before an s3.list_objects() call. So if I want a specific handler just for this operation I can register a handler with before-call.s3.ListObjects. But if I wanted to register a handler that was called before any operation (say for logging/debugging/auditing purposes), I could just register a handler for before-call, and the handler would be called before any API call, regardless of the service operation. I could also register a handler just for any S3 call by registering for before-call.s3.

Hope that helps let me know if you have any other questions.

[jcmcken]

Are there any plans to make the plugin system a public, documented API? It seems like this issue was closed without answering that question.

We use a custom IAM role federation service that ties to our corporate services for authentication/authorization. Currently, users have to develop separate tools to acquire and manage credentials which is somewhat awkward. Implementing a common tool for interacting with that middleware as a plugin within awscli would make the experience much better, but we're concerned that the plugin system isn't stable or supported.

[nitrocode]

@jcmcken check out https://github.com/RichardBronosky/aws-cli-plugins or https://github.com/nitrocode/aws-cli-plugins . If you find a way to add a subcommand without showing the help dialogue, let me know 😄