How to Create a Smart-Proxy Plugin » History » Revision 25
Revision 24 (Ewoud Kohl van Wijngaarden, 02/15/2019 02:47 PM) → Revision 25/30 (Ewoud Kohl van Wijngaarden, 02/15/2019 02:50 PM)
h1. How to Create a Smart-Proxy Plugin
This guide outlines main components of a plugin, but assumes some degree of familiarity with ruby gems, bundler, rack, and Sinatra. You'll find links to useful documentation in each of the sections below.
{{toc}}
h2. Plugin Organization
Smart-Proxy plugins are normal ruby gems, please follow documentation at http://guides.rubygems.org/make-your-own-gem/ for guidance on gem creation and packaging. It is strongly recommended to follow smart_proxy_<your plugin name here> naming convention for your plugin.
We have some templates for creating your plugin:
* "smart_proxy_example plugin":https://github.com/theforeman/smart_proxy_example is a minimal example plugin that can be used as a skeleton
* "smart_proxy_dns_plugin_template":https://github.com/theforeman/smart_proxy_dns_plugin_template is a template for creating new DNS provider plugins
Also, "smart_proxy_pulp plugin":https://github.com/theforeman/smart-proxy-pulp is an example for a fully functional, yet easy to understand Smart-Proxy plugin.
h2. Making your plugin official
Once you're ready to release the first version, please see [[How_to_Create_a_Plugin#Making-your-plugin-official]] for info on making your plugin part of the Foreman project.
h2. Plugin definition
A plugin definition is used to define plugin's name, version, location of rackup configuration, and other parameters. At a minimum, Plugin Descriptor must define plugin name and version. Note the base class of the descriptor is ::Proxy::Plugin:
<pre><code class="ruby"> class='ruby'>
module Proxy::Example
class Plugin < ::Proxy::Plugin
plugin :example, "0.0.1"
http_rackup_path File.expand_path("http_config.ru", File.expand_path("../", __FILE__))
https_rackup_path File.expand_path("https_config.ru", File.expand_path("../", __FILE__))
default_settings :hello_greeting => 'Hello there!', :important_path => '/must/exist'
load_classes ::Proxy::Example::ClassLoader
load_programmable_settings "::Proxy::Example::ProgrammableSettings"
load_dependency_injection_wirings "::Proxy::Example::DIConfiguration"
load_validators :my_validator => Proxy::Example::CustomValidators::MyValidator
validate_readable :optional_path, :important_path
validate :a_setting, :my_validator => true, :if => lambda {|settings| !settings[:a_setting].nil?}
validate :another_setting, :my_other_validator => true
start_services :service_a, :service_b
end
end
</code></pre>
Here we defined a plugin called "example", with version "0.0.1", that is going to listen on both http and https ports.
h3. Full list of descriptor parameters
Following is the full list of parameters that can be defined by the Plugin Descriptor, and the version of the Smart Proxy that they were added in.
General smart proxy configuration parameters:
* plugin :example, "1.2.3": *required*. Sets plugin name to "example" and version to "0.0.1".
* http_rackup_path "path/to/http_config.ru": _optional_, _1.6+_. Sets path to http rackup configuration. If omitted, the plugin is not going to listen on the http port. Please see below for information on rackup configuration.
* https_rackup_path "path/to/https_config.ru": _optional_, _1.6+_. Sets path to https rackup configuration. If omitted, the plugin is not going to listen on the https port. Please see below for information on rackup configuration.
Loading and dependencies:
* requires :another_plugin, '~> 1.2.0': _optional_, _1.6+_. Specifies plugin dependencies, where ":another_plugin" is another plugin name, and '~> 1.2.0' is version specification (pls. see http://guides.rubygems.org/patterns/#pessimistic_version_constraint for details on version specification).
* bundler_group :my_plugin_group: _optional_, _1.6+_. Sets the name of the bundler group for plugin dependencies. If omitted the plugin name is used.
* after_activation { do_something }: _optional_, _1.6+_. Supplied block is going to be executed after the plugin has been loaded and enabled. Note that the block is going to be executed in the context of the Plugin Descriptor class.
* load_classes: _1.12+_. must be a class, class name, or a block. Specified class must implement "load_classes" instance method that loads module's dependencies.
* load_dependency_injection_wirings: can be a class, class name, or a block. _1.12+_. The class must implement load_dependency_injection_wirings(di_container, settings_hash) instance method.
* start_services: _1.12+_. list of dependency injection wiring labels. Services that perform work independently (asynchroniously) of http requests should implement #start method.
Settings related:
* default_settings :first => 'my first setting', :another => 'my other setting': _optional_. _1.6+_. Defines default values for plugin parameters. These parameters can be overridden in plugin settings file. Setting any of the parameters in default_settings to nil will trigger a validation error.
* load_programmable_settings: can be a class, class name, or a block. _1.12+_. Specified class must implement load_programmable_settings(settings_hash) instance method that returns new or updated settings.
* load_validators: a hash of validator name (a symbol) to validator class mappings. _1.12+_.
* validate: validate :setting_one, :setting_two, ..., :setting_n, :validator_name => { :validator_param_one => 'value one', ...,}, :if => lambda {|settings| ... }, alternatively use :validator_name => true if validator has no parameters. If predicate is specified, the validator will be called only if the lambda evaluates to true. Predicate's lambda expects module's settings passed as a parameter. (_1.12+_)
* validate_readable :optional_path, :important_path: _optional_, _1.10+_. Verifies that settings listed here contain paths to files that exist and are readable. Optional settings (not listed under default_settings) will be skipped if left uninitialized.
* validate_presence :setting_one, ..., :setting_n: _optional_, _1.10+_. Verifies that settings listed are not equal to nil. Executed automatically for each of the default settings.
h3. Provider definition
Some plugins are *providers* for an existing plugin or module in the Smart Proxy, e.g. a DNS provider.
These are registered almost identically, but use Proxy::Provider instead of Proxy::Plugin. No rackup_paths are used for providers, since they don't add any new REST API, they only add functionality to an existing module.
<pre><code class="ruby"> <pre>
module Proxy::Dns::PluginTemplate
class Plugin < ::Proxy::Provider
plugin :dns_plugin_template, ::Proxy::Dns::PluginTemplate::VERSION
requires :dns, '>= 1.11'
after_activation do
require 'smart_proxy_dns_plugin_template/dns_plugin_template_main'
require 'smart_proxy_dns_plugin_template/dns_plugin_template_dependencies'
end
end
end
</code></pre> </pre>
Additionally, each provider must specify which class implements interface expected by the main plugin. This is done by declaring an association for module's dependency injection container.
<pre><code class="ruby"> <pre>
require 'dns_common/dependency_injection/dependencies'
class Proxy::Dns::DependencyInjection::Dependencies
dependency :dns_provider, Proxy::Dns::PluginTemplate::Record
end
</code></pre> </pre>
h2. Plugin Initialization
The initialization process can be thought of as consisting of two phases: loading and validation of settings and runtime initialization -- selection of classes, their parameters, and how they will be instantiated.
During the first phase of the process, all modules are gathered into groups consisting of the main plugin and one or more providers. If any of the members of the group fail at any time during initialization, the rest of the modules in the group will be failed as well. Initialization starts with all loaded and enabled plugin (main modules) classes being collected, then for each:
* configuration file is loaded
* dependencies are loaded (also see load_classes)
* configuration-related code executed, and configuration updated (also see load_runtime_configuration)
* validators executed (also see load_validators)
* provider names resolved to provider classes
At this point the steps above are repeated for all providers, one module group at a time. During the second phase, for each of the modules:
* dependency injection wirings are resolved (also see load_dependency_injection_wirings)
* services started (also see start_services)
* module's versions are checked against other modules stated requirements (also see)
h2. How to Load Dependencies
_This technique requires Smart Proxy 1.12 or higher._
The class loader must implement load_classes instance method:
<pre><code class="ruby"> <pre>
class ::Proxy::Example::ClassLoader
def load_classes
require 'example/class_a'
require 'example/class_b'
end
end
</code></pre> </pre>
alternatively, a block can be used to load dependencies:
<pre><code class="ruby"> <pre>
module Proxy::Example
class Plugin < ::Proxy::Plugin
...
load_classes do
require 'example/class_a'
require 'example/class_b'
end
...
end
end
</code></pre> </pre>
h2. How to Programmatically Update Settings
_This technique requires Smart Proxy 1.12 or higher._
The class must implement load_programmable_settings(settings_hash) instance method that returns new or updated settings:
<pre><code class="ruby"> <pre>
class ::Proxy::Example::RuntimeConfiguration
def load_programmable_settings(settings_hash)
settings_hash[:a_setting] = "Hello, world"
settings_hash
end
end
</code></pre> </pre>
h2. How to Define Dependency Injection Wirings
_This technique requires Smart Proxy 1.12 or higher._
The class must implement load_dependency_injection_wirings instance method that has dependency injection container and settings hash as its parameters:
<pre><code class="ruby"> <pre>
class ::Proxy::Example::DIConfiguration
def load_dependency_injection_wirings(container_instance, settings)
container_instance.dependency :depedency_a, ::Proxy::Example::ClassA
container_instance.dependency :dependency_b, ::Proxy::Example::ClassB
container_instance.singleton_dependency :service_a, lambda {|container| ::Proxy::Example::ServiceA.new(settings[:example_setting], container_instance.get_dependency(:dependency_a))}
end
end
</code></pre> </pre>
When Proxy::DependencyInjection::Container#dependency is used to define a dependency, a new instance of a class will be returned, or lambda executed every time the dependency is requested.
If only a single instance of a class is ever required, use Proxy::DependencyInjection::Container#singleton_dependency: the class will be instantiated first time the dependency is requested, and then reused on all subsequent requests.
h2. How to Create Custom Validators
_This technique requires Smart Proxy 1.12 or higher._
A validator must use ::Proxy::PluginValidators::Base as its base class and implement validate!(settings) instance method that accepts a hash containing module settings. validate! should raise an exception if the check it's making fails.
<pre><code class="ruby"> <pre>
class Proxy::Example::CustomValidators
class MyValidator < ::Proxy::PluginValidators::Base
def validate!(settings)
raise ::Proxy::Error::ConfigurationError("Unsupported greeting") if settings[@setting_name] != "Hello, world"
end
end
end
</code></pre>
</pre>
h2. API
Modular Sinatra app is used to define plugin API. Note the base class Sinatra::Base and inclusion of ::Proxy::Helpers:
<pre><code class="ruby">
module Proxy::Example
class Api < Sinatra::Base
helpers ::Proxy::Helpers
get "/hello" do
Proxy::Example::Plugin.settings.hello_greeting
end
end
</code></pre>
Here we return a string defined in 'hello_greeting' parameter (see Plugin Descriptor above and settings file below) when a client performs a GET /hello. Please refer to "Sinatra documentation":http://www.sinatrarb.com/intro.html on details about routing, template rendering, available helpers, etc.
h2. Rackup Configuration
During startup Smart-Proxy assembles web applications listening on http and https ports using rackup files of enabled plugins. Plugin rackup files define mounting points of plugin API:
<pre><code class="ruby">
require 'example_plugin/example_api'
map "/example" do
run Proxy::Example::Api
end
</code></pre>
The example above should be sufficient for the majority of plugins. Please refer to "Sinatra+Rack":http://www.sinatrarb.com/intro.html documentation for additional information.
h2. Plugin Settings
On startup Smart-Proxy will load and parse plugin configuration files located in its settings.d/ directory. Each plugin config file is named after the plugin and is a yaml-encoded collection of key-value pairs and used to override default values of plugin parameters.
<pre>
---
:enabled: true
:hello_greeting: "O hai!"
</pre>
This settings file enables the plugin (by default all plugins are disabled), and overrides :hello_greeting parameter. Plugin settings can be accessed through .settings method of the Plugin class, for example: ExamplePlugin.settings.hello_greeting. Global Smart-Proxy parameters are accessible through Proxy::SETTINGS, for example Proxy::SETTINGS.foreman_url returns Foreman url configured for this Smart-Proxy.
h2. Bundler Configuration
Smart-Proxy relies on bundler to load its dependencies and plugins. We recommend to create a dedicated bundler config file for your plugin, and name it after the plugin. For example:
<pre><code class="ruby">
gem 'smart_proxy_example'
group :example do
gem 'json'
end
</code></pre>
You'll need to create a dedicated bundler group for additional dependencies of your plugin. By default the group shares the name with the plugin, but you can override it using bundler_group parameter in Plugin Descriptor. Please refer to [[How_to_Install_a_Smart-Proxy_Plugin]] for additional details on "from source" plugin installations.
h2. Adding a DNS provider
_Requires Smart Proxy 1.15 or higher (1.14 has a different interface.)_
When extending the 'dns' smart proxy module, the plugin needs to create a new Proxy::Dns::Record class with @do_create@ and @do_remove@ methods for adding and removing of DNS records.
The easiest way to do this is using the "Smart Proxy DNS plugin template":https://github.com/theforeman/smart_proxy_dns_plugin_template which can get you up and running with a new DNS provider plugin in minutes.
DNS Provider classes are instantiated by DNS module's dependency injection container.
<pre><code class="ruby"> <pre>
plugin :dns_plugin_template, ::Proxy::Dns::PluginTemplate::VERSION
</code></pre> </pre>
And then in the main file of the plugin:
<pre><code class="ruby"> <pre>
require 'dns_common/dns_common'
module Proxy::Dns::PluginTemplate
class Record < ::Proxy::Dns::Record
include Proxy::Log
attr_reader :example_setting, :optional_path, :required_setting, :required_path
def initialize(required_setting, example_setting, required_path, optional_path, dns_ttl)
@required_setting = required_setting # never nil
@example_setting = example_setting # can be nil
@required_path = required_path # file exists and is readable
@optional_path = optional_path # nil, or file exists and is readable
# Common settings can be defined by the main plugin, it's ok to use them locally.
# Please note that providers must not rely on settings defined by other providers or plugins they are not related to.
super('localhost', dns_ttl)
end
def do_create(name, value, type)
# FIXME: There is no trailing dot. Your backend might require it.
if false
name += '.'
value += '.' if ['PTR', 'CNAME'].include?(type)
end
# FIXME: Create a record with the correct name, value and type
raise Proxy::Dns::Error.new("Failed to point #{name} to #{value} with type #{type}")
end
def do_remove(name, type)
# FIXME: There is no trailing dot. Your backend might require it.
if false
name += '.'
end
# FIXME: Remove a record with the correct name and type
raise Proxy::Dns::Error.new("Failed to remove #{name} of type #{type}")
end
end
end
</code></pre> </pre>
DNS provider support was first added in version 1.10, but the interface was updated between 1.10 and 1.11. Later in 1.15 it was further modified. Please see the history of this page for 1.14-compatible recommendations and the 1.14-stable branch of the example DNS plugin instead of master.
h2. Adding a DHCP provider
_Requires Smart Proxy 1.11 or higher._
When creating a new 'dhcp' provider smart proxy module, the plugin needs to create a new Proxy::DHCP::Server class that implements @load_subnets@, @load_subnet_data@, @find_subnet@, @subnets@, @all_hosts@, @unused_ip@, @find_record@, @add_record@, and @del_record@ methods.
Provider classes are instantiated by DHCP module's dependency injection container.
<pre><code class="ruby"> <pre>
plugin :example_dhcp_provider, ::ExampleDhcpPlugin::Provider::VERSION
</code></pre> </pre>
And then in the main file of the plugin:
<pre><code class="ruby"> <pre>
require 'dhcp_common/server'
module ::ExampleDhcpPlugin
class Provider < ::Proxy::DHCP::Server
include Proxy::Log
include Proxy::Util
def initialize
super('localhost')
end
def load_subnets
# loads subnet data into memory
end
def find_subnet(network_address)
# returns Proxy::DHCP::Subnet that has network_address or nil if none was found
end
def load_subnet_data(a_subnet)
# loads lease- and host-records for a Proxy::DHCP::Subnet
end
def subnets
# returns all available subnets (instances of Proxy::DHCP::Subnet)
end
def all_hosts(network_address)
# returns all reservations in a subnet with network_address
end
def unused_ip(network_address, mac_address, from_ip_address, to_ip_address)
# returns first available ip address in a subnet with network_address, for a host with mac_address, in the range of ip addresses: from_ip_address, to_ip_address
end
def find_record(network_address, ip_or_mac_address)
# returns a Proxy::DHCP::Record from a subnet with network_address that has ip- or mac-address specified in ip_or_mac_address, or nil of none was found
end
def add_record(params)
# creates a record
end
def del_record(network_address,a_record)
# removes a Proxy::DHCP::Record from a subnet with network_address
end
end
end
</code></pre> </pre>
DHCP provider support was first added in version 1.11.
h2. Testing
Make sure that your Gemfile includes the "smart-proxy" gem as a development dependency:
<pre><code class="ruby">
group :development do
gem 'smart_proxy', :git => "https://github.com/theforeman/smart-proxy.git"
end
</code></pre>
Ensure that your plugin has a Rakefile, for example:
<pre><code class="ruby">
require 'rake'
require 'rake/testtask'
desc 'Default: run unit tests.'
task :default => :test
desc 'Test Pulp Plugin'
Rake::TestTask.new(:test) do |t|
t.libs << '.'
t.libs << 'lib'
t.libs << 'test'
t.test_files = FileList['test/**/*_test.rb']
t.verbose = true
end
</code></pre>
Load 'smart_proxy_for_testing' in your tests:
<pre><code class="ruby">
$: << File.join(File.dirname(__FILE__), '..', 'lib')
require 'smart_proxy_for_testing'
require 'test/unit'
require 'webmock/test_unit'
require 'mocha/test_unit'
require "rack/test"
require 'smart_proxy_pulp_plugin/pulp_plugin'
require 'smart_proxy_pulp_plugin/pulp_api'
class PulpApiTest < Test::Unit::TestCase
include Rack::Test::Methods
def app
PulpProxy::Api.new
end
def test_returns_pulp_status_on_200
stub_request(:get, "#{::PulpProxy::Plugin.settings.pulp_url.to_s}/api/v2/status/").to_return(:body => "{\"api_version\":\"2\"}")
get '/status'
assert last_response.ok?, "Last response was not ok: #{last_response.body}"
assert_equal("{\"api_version\":\"2\"}", last_response.body)
end
end
</code></pre>
To execute all tests <code><pre>bundle exec rake test</code></pre>. To save time during development it is possible to execute tests in a single file: <code><pre>bundle exec rake test TEST=path/to/test/file</pre></code>
Please refer to "Sinatra documention":http://www.sinatrarb.com/testing.html for detailed information on testing of Sinatra applications.
Once you have tests, see "Jenkins":https://projects.theforeman.org/projects/foreman/wiki/Jenkins#Smart-proxy-plugin-testing "Jenkins":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#Smart-proxy-plugin-testing for info on setting up tests under Jenkins.