|Doug Hellmann||May 11, 2012 8:55 am|
|Loic Dachary||May 11, 2012 12:40 pm|
|Loic Dachary||May 11, 2012 12:55 pm|
|Doug Hellmann||May 11, 2012 1:18 pm|
|Doug Hellmann||May 11, 2012 1:21 pm|
|Loic Dachary||May 14, 2012 5:22 am|
|Doug Hellmann||May 14, 2012 7:15 am|
|Loic Dachary||May 14, 2012 10:03 am|
|Doug Hellmann||May 14, 2012 1:27 pm|
|Julien Danjou||May 15, 2012 3:04 am|
|Loic Dachary||May 15, 2012 4:52 am|
|Julien Danjou||May 15, 2012 5:21 am|
|Doug Hellmann||May 15, 2012 7:26 am|
|Nick Barcet||May 16, 2012 12:31 am|
|Loic Dachary||May 16, 2012 1:09 am|
|Julien Danjou||May 16, 2012 2:19 am|
|Doug Hellmann||May 16, 2012 8:05 am|
|Subject:||Re: [Openstack] [metering] public API design|
|From:||Doug Hellmann (doug...@dreamhost.com)|
|Date:||May 11, 2012 1:18:34 pm|
On Fri, May 11, 2012 at 3:40 PM, Loic Dachary <lo...@enovance.com> wrote:
On 05/11/2012 05:55 PM, Doug Hellmann wrote:
During yesterday's meeting we discussed the API proposal at http://wiki.openstack.org/EfficientMetering#API and came up with a few missing items and other points for discussion. We should try to work out those details on the list before the meeting next week.
The original proposal included these API calls:
GET list components GET list components meters (argument : name of the component) GET list [user_id|project_id|source] GET list of meter_type GET list of events per [user_id|project_id|source] ( allow to specify user_id or project_id or both ) GET sum of (meter_volume, meter_duration) for meter_type and [user_id|project_id|source]
They can be broken down into three groups:
- Discover the sorts of things the server can provide:
- list the components providing metering data - list the meters for a given component - list the known users - list the known projects - list the known sources - list the types of meters known
- Fetch raw event data, without aggregation:
- per user - per project - per source - per user and project
- Produce aggregate views of the data:
- sum "volume" field for meter type over a time period - per user - per project - per source - sum "duration" field for meter type over a period of time - per user - per project - per source
I updated the wiki with these three groups, it is a good starting point for discussion. http://wiki.openstack.org/EfficientMetering?action=diff&rev2=75&rev1=74
We agreed that all of the queries related to metering data would need to have parameters to set the start and end times with the start time inclusive and the end time exclusive (i.e., start <= timestamp < end).
Callers also are likely to want to filter the raw data queries based on the meter type, so we need to add that as an optional argument there. The aggregate values have the meter type as a required argument (because it does not make sense to aggregate data from separate meters together).
There are a few other queries missing from that list. The items below are based on the meeting notes and my own thoughts from yesterday, please add to the list if I have left anything out (or suggest why we should not implement something suggested here, of course):
- list discrete events that may not have a duration (instance creation, IP allocation, etc.) - list raw event data for a resource ("what happened with a specific instance?") - aggregate event data per meter type for a resource over a period of time ("what costs are related to this instance?") - sum volume for meter type over a time period for a specific resource ("how much total bandwidth was used by a VIF?") - sum duration for meter type over a time period for a specific resource ("how long did an instance run?") - metadata for resources (such as location of instances) - aggregating averages in addition to sums
I've added this list as a note in the API chapter. http://wiki.openstack.org/EfficientMetering?action=diff&rev2=76&rev1=75
Some of these items may be better handled in the consumer of this API (the system that actually bills the user). I am curious to know how others are handling auditing or other billing detail reporting for users.
I will be in a meeting with the participants of the http://www.opencloudware.org/ project early next week and I'll ask them. Maybe Daniel Dyer could share a practical experience with us.
In order to support the discrete events, we need to capture them. Perhaps those could be implemented as additional counter types. Thoughts?
Are there resources that should be billed (henced metered) if their lifespan is shorter than the polling / publishing event period ? I mean that if someone creates a VM and immediately destroys it, does it really matter. While writing this statement, I though of a way to abuse a system that does not account for discrete events like "created a VM" but only account for "this VM has run for at least X minutes" :-) One could write a software to launch short lived VM at specific times and get computing power for free.
Exactly. We may bill the customer for the first X minutes of use for a VM just for creating it, and then do the math necessary to avoid double billing them during the first billing cycle.
Do discrete event need to be separate ? Back to the example above, someone
creates a VM : that does not create a record in the ceilometer storage. The VM is destroyed a few seconds later : it creates an event, as if it was polled / published to measure the uptime of the VM. The c1 meter would show the VM has been up for 15 seconds since it was created. If the c1 meter is generated from an exist event captured from http://wiki.openstack.org/SystemUsageData that occurs every 5 minutes, the uptime will be 5 minutes more than the previous one.
VMs may not be the best example for discussing discrete events since so much of what happens there is related to time. A better example is creating images. We may charge the user a one-time fee for uploading an image to Glance, and then charge an ongoing fee for the storage consumed. So the one-time fee is a discrete event based on creating the image.
While thinking about this use case I also had a simple idea to optimize the storage of events and added it to the wiki http://wiki.openstack.org/EfficientMetering?action=diff&rev2=77&rev1=76
-- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ lo...@enovance.com ☎ +33 1 49 70 99 82