Welcome to mesoshttp’s documentation!¶
This is a Python library to create a Mesos scheduler.
It removes the need to use/install Mesos python bindings and makes use of the HTTP API (native bindings are not updated anymore with new features)
MesosClient is the main entrypoint that connects to the Mesos master. You can then register for callbacks on events. On subscribe, you get a driver instance which you can use to send messages to master on current framework (kill, etc.)
MesosClient must be executed in a separate thread as it keeps a loop pooling connection with the master.
Submitted tasks should be in JSON format (according to mesos.proto).
See sample/test.py for example.
Callbacks will “block” the mesos message treatment, so they should be short, or messages should be forwarded to a queue in an other thread/process where longer tasks will handle messages.
Contents:
MesosClient¶
MesosClient reference¶
- class
mesoshttp.client.
MesosClient
(mesos_urls, frameworkId=None, frameworkName='Mesos HTTP framework', frameworkUser='root', frameworkHostname='', frameworkWebUI='', max_reconnect=3)¶Entrypoint class to connect framework to Mesos master
Instance should be started in a separate thread as MesosClient.register will start a blocking loop with a long connection.
- class
SchedulerDriver
(mesos_url, frameworkId, streamId, requests_auth=None, verify=True)¶Handler to communicate with scheduler
MesosClient.SchedulerDriver instance is available after the SUBSCRIBED event with the subscribed event.
__init__
(mesos_url, frameworkId, streamId, requests_auth=None, verify=True)¶Create a driver instance related to created framework
kill
(agent_id, task_id)¶Kill specified task
Parameters:
- agent_id (str) – slave agent_id
- task_id (str) – task identifier
message
(agent_id, executor_id, message)¶Send message to an executor
Parameters:
- agent_id (str) – slave identifier
- executor_id (str) – executor identifier
- message (str) – message to send, raw bytes encoded as Base64
reconcile
(tasks)¶Reconcile tasks
Parameters: tasks (list) – list of dict { “agent_id”: xx, “task_id”: yy }
request
(requests)¶Send a REQUEST message
Parameters: requests (list) – list of resources request [{‘agent_id’: : XX, ‘resources’: {}}]
revive
()¶Send REVIVE request
shutdown
(agent_id, executor_id)¶Shutdown an executor
Parameters:
- agent_id (str) – slave identifier
- executor_id (str) – executor identifier
tearDown
()¶Undeclare framework
_MesosClient__zk_detect
(zk_url, prefix='/mesos')¶Try to get master url info from zookeeper
Parameters:
- zk_url (str) – ip/port to reach zookeeper
- prefix (str) – prefix to search for in zookeeper
__init__
(mesos_urls, frameworkId=None, frameworkName='Mesos HTTP framework', frameworkUser='root', frameworkHostname='', frameworkWebUI='', max_reconnect=3)¶Create a frameworkId
Parameters:
- mesos_urls (list) – list of mesos http endpoints
- frameworkId (str) – identifier of the framework, if None, will declare a new framework
- frameworkName (str) – name of the framework
- frameworkUser (str) – user to use (will run tasks as user), defaults to root
- frameworkHostname (str) – Hostname of the framework
- frameworkWebUI (str) – URL of the framework WebUI
- max_reconnect (int defaults to 3) – number of reconnection retries when connection fails
__weakref__
¶list of weak references to the object (if defined)
add_capability
(capability)¶Adds a framwork capability
Parameters: capability (str) – caapbility name
combine_offers
(offers, operations, options=None)¶Accept offers with task operations
Parameters:
- offers (list) – offers to be accepted
- operations (list of json TaskInfo) – JSON TaskInfo instances to accept
- options (JSON filters instances) – optional filters
This method does not check if the operations are valid JSON TaskInfo instances and if conform with the offers.
disconnect_framework
()¶Stops framework but does not teardown (unregister) the framework
This will enable framework reconnection and will not kill running jobs
get_driver
()¶Get driver instance to dialog with master
Returns: MesosClient.SchedulerDriver
get_master_info
()¶Get Mesos master information, return None if not connected
Returns: json formatted info about connected Mesos master
on
(eventName, callback)¶Register callback for an event.
Multiple callbacks can be registered for the same event
Parameters:
- eventName (str) – name fo the event to register to (MesosClient.SUBSCRIBED, etc.)
- callback (def) – function to call on event
register
()¶Register framework, return False if could not connect, else will open a permanent HTTP connection.
Creates an infinite loop on a permanent connection to Mesos master to receive messages. On message, callbacks will be called.
set_checkpoint
(do_checkpoint)¶Sets framework checkpoint value
Parameters: do_checkpoint (bool) – de/activate checkpoint in framework
set_credentials
(principal, secret)¶Set credentials to authenticate with Mesos master
Parameters:
- principal (str) – login to use
- secret (str) – password
set_failover_timeout
(timeout)¶Sets failover timeout value
Parameters: timeout (int) – define framework failover timeout, in seconds
set_role
(role_name)¶Set Mesos role to use by framework
Parameters: role_name (str) – Mesos role name
set_service_account
(service_secret, verify=False)¶Set credentials to authenticate with DCOS and Mesos Master
Parameters:
- service_secret (dict) – Optional DCOS Service account secret. Supersedes principal / secret.
- verify (bool) – validate HTTPS fronted Mesos API using CA root trusts, defaults to False
tearDown
()¶Unregister and stop scheduler
Offers¶
Offers reference¶
- class
mesoshttp.offers.
Offer
(mesos_url, frameworkId, streamId, mesosOffer, requests_auth=None, verify=True)¶Wrapper class for Mesos offers
__init__
(mesos_url, frameworkId, streamId, mesosOffer, requests_auth=None, verify=True)¶Initialize self. See help(type(self)) for accurate signature.
accept
(operations, options=None)¶Accept offer with task operations
Parameters:
- operations (list of json TaskInfo) – JSON TaskInfo instances to accept in current offer
- options (dict) – Optional offer additional params (filters, …)
decline
(options=None)¶Decline offer
Parameters: options (dict) – Optional offer additional params (filters, …)
get_offer
()¶Get offer info received from Mesos
Returns: dict
Update¶
Update reference¶
- class
mesoshttp.update.
Update
(mesos_url, frameworkId, streamId, mesosUpdate, requests_auth=None, verify=True)¶This class manages Update message from Mesos master
__init__
(mesos_url, frameworkId, streamId, mesosUpdate, requests_auth=None, verify=True)¶Initialize self. See help(type(self)) for accurate signature.
ack
()¶Acknowledge an update message