Computing Metric Results Via the API

How to Format Request

Requests should be post requests to https://api.ai-gov-sys-devportal.com/gateway/metrics

Body

Requests should be sent in JSON format. For any new parameters (ie. parameters not already saved in the database), you should include them in the new_parameter_values subdictionary. The currently supported parameters are as follows:

Field	Data Type	Description
`df`	String (df.to_json())	Pandas DataFrame holding input data and data predicted by a model
`y_true`	String (y_true.to_json())	Pandas DataFrame holding ground truth values. It should have the same `protected_attribute` and `labels` column as `df`
`protected_attribute`	String	Column in `df` signifying the sensitive trait (eg. class, sex, religion etc.)
`privileged_group`	String	Column in `df` signifying a group with a historically positive label for their protected attribute (eg. male, abled)
`labels`	String	Column in `df` that is the output of the model (eg. isHired?)
`positive_label`	String, Boolean	Value in `labels` that is classified as positive
`scores`	String	Column in `df` that contains the scores for each row
`kwargs`	Dictionary	Extra Values (planned to be used for users to specify custom metrics)
`epsilon`	Number	Trivially small addend used in division operation to avoid division by 0 errors. Default value is 1e-10.
`beta`	Number	Beta value used in F-Beta metrics explained below. Default value is 1 meaning all F-Beta scores default to F1 scores.

Any existing parameter values (ie. values already stored in the database) should be put into the existing_parameter_values subdictionary of the request. Each entry should be formatted as 'parameter_name': 'parameter_value_id'. An example is shown below:

{
  "existing_parameter_values":{
      "df": "9293b41d-b098-4bfe-980f-e2b70a1a47bf"
  }
}

In the above example, it is the responsibility of the user to provide the remaining new_parameter_values for their metric to run correctly.

In addition to the parameters, both new and pre-existing, a metric function name must also be provided as follows:{

{"func_name": "name of fairness metric"}

A full example of a request is provided below with the expected output of (0.5, 0.3636363636363636):

{
   "new_parameter_values":{
      "df":"{\"name\":{\"0\":\"Bob\",\"1\":\"Charlie\",\"2\":\"Dave\",\"3\":\"Dave\",\"4\":\"Dave\",\"5\":\"Dave\",\"6\":\"Dave\",\"7\":\"Dave\",\"8\":\"Dave\",\"9\":\"Dave\",\"10\":\"Dave\",\"11\":\"Alice\",\"12\":\"Alice\",\"13\":\"Alice\",\"14\":\"Alice\",\"15\":\"Alice\",\"16\":\"Alice\",\"17\":\"Eve\",\"18\":\"Eve\",\"19\":\"Eve\",\"20\":\"Eve\"},\"age\":{\"0\":32,\"1\":28,\"2\":22,\"3\":22,\"4\":22,\"5\":22,\"6\":22,\"7\":22,\"8\":22,\"9\":22,\"10\":22,\"11\":25,\"12\":25,\"13\":25,\"14\":25,\"15\":25,\"16\":25,\"17\":19,\"18\":19,\"19\":19,\"20\":19},\"gender\":{\"0\":\"male\",\"1\":\"male\",\"2\":\"male\",\"3\":\"male\",\"4\":\"male\",\"5\":\"male\",\"6\":\"male\",\"7\":\"male\",\"8\":\"male\",\"9\":\"male\",\"10\":\"male\",\"11\":\"female\",\"12\":\"female\",\"13\":\"female\",\"14\":\"female\",\"15\":\"female\",\"16\":\"female\",\"17\":\"female\",\"18\":\"female\",\"19\":\"female\",\"20\":\"female\"},\"hired\":{\"0\":true,\"1\":true,\"2\":true,\"3\":true,\"4\":true,\"5\":false,\"6\":false,\"7\":false,\"8\":false,\"9\":false,\"10\":false,\"11\":true,\"12\":true,\"13\":false,\"14\":false,\"15\":false,\"16\":false,\"17\":false,\"18\":true,\"19\":true,\"20\":true}}",
      "privileged_group":"male",
      "labels":"hired",
      "positive_label":true,
      "kwargs":{
         "beta":1
      },
      "y_true":"{\"gender\":{\"0\":\"male\",\"1\":\"male\",\"2\":\"male\",\"3\":\"male\",\"4\":\"male\",\"5\":\"male\",\"6\":\"male\",\"7\":\"male\",\"8\":\"male\",\"9\":\"male\",\"10\":\"male\",\"11\":\"female\",\"12\":\"female\",\"13\":\"female\",\"14\":\"female\",\"15\":\"female\",\"16\":\"female\",\"17\":\"female\",\"18\":\"female\",\"19\":\"female\",\"20\":\"female\"},\"hired\":{\"0\":true,\"1\":true,\"2\":false,\"3\":false,\"4\":false,\"5\":false,\"6\":false,\"7\":false,\"8\":false,\"9\":false,\"10\":true,\"11\":true,\"12\":true,\"13\":true,\"14\":true,\"15\":true,\"16\":true,\"17\":false,\"18\":false,\"19\":false,\"20\":false}}",
      "protected_attribute":"gender"
   },
   "existing_parameter_values":{
      
   },
   "func_name":"Predictive Equality"
}

Authorization

Two forms of authorization are required. Firstly, in the headers of the request, an x-api-key-id and x-api-key-secret must be included. Both of these values can be found on the ai-gov-sys-devportal under Organization > Settings > API Keys.
Secondly, your bearer token must be included in the request authorization.

Programmatic Example

# data
import json
import pandas as pd
import requests
from metrics import *

# Example DataFrame
data = [
    {'name': 'Bob', 'age': 32, 'gender': 'male', 'hired': True},
    {'name': 'Charlie', 'age': 28, 'gender': 'male', 'hired': True}, # 2 true positives
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': True}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': True}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': True}, # 3 false positives
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, 
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, # 5 true negatives
    {'name': 'Dave', 'age': 22, 'gender': 'male', 'hired': False}, # 1 false negative

    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': True},
    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': True}, # 2 true positive
    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': False},
    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': False},
    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': False},
    {'name': 'Alice', 'age': 25, 'gender': 'female', 'hired': False}, # 4 false negative
    {'name': 'Eve', 'age': 19, 'gender': 'female', 'hired': False}, # 1 true negative
    {'name': 'Eve', 'age': 19, 'gender': 'female', 'hired': True},
    {'name': 'Eve', 'age': 19, 'gender': 'female', 'hired': True},
    {'name': 'Eve', 'age': 19, 'gender': 'female', 'hired': True}, # 3 false positive
]

data_true = [
    {'gender': 'male', 'hired': True},
    {'gender': 'male', 'hired': True}, # 2 true positives
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False}, # 3 false positives
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False},
    {'gender': 'male', 'hired': False}, # 5 true negatives
    {'gender': 'male', 'hired': True}, # 1 false negative

    {'gender': 'female', 'hired': True},
    {'gender': 'female', 'hired': True}, # 2 true positives
    {'gender': 'female', 'hired': True},
    {'gender': 'female', 'hired': True},
    {'gender': 'female', 'hired': True},
    {'gender': 'female', 'hired': True}, # 4 false negative
    {'gender': 'female', 'hired': False}, # 1 true negative
    {'gender': 'female', 'hired': False},
    {'gender': 'female', 'hired': False},
    {'gender': 'female', 'hired': False}, # 3 false positives
]
df = pd.DataFrame(data)
y_true = pd.DataFrame(data_true)

# Example usage of the parity_difference function
protected_attribute = 'gender'
privileged_group = 'male'
labels = 'hired'
positive_label = True

# request
url = ...
headers = {
  'Content-Type': 'application/json',
  'Authorization': 'Bearer [token]',
  'x-api-key-id': [key_id],
  'x-api-key-secret': [key_secret],
}
payload = json.dumps({
  "new_parameter_values": {
    "df": df.to_json(),
    "privileged_group": privileged_group,
    "labels": labels,
    "positive_label": positive_label,
    "kwargs": {
      "beta": 2
    },
    "y_true": y_true.to_json(),
    "protected_attribute": protected_attribute
  },
  "existing_parameter_values": {},
  "func_name": "F-Beta Score"
})

response = requests.request("POST", url, headers=headers, data=payload)

## results should be equivalent
print(response.text) # includes result as well as parameter value IDs
print(fb_score(df, labels, positive_label, y_true, beta=2)

Return Response

Assuming a correct metric calculation and successful API interaction, the response should be a json dictionary with the result as well as the ID's of all parameter values used in the calculation. This is especially useful for when running many metrics with the same parameters.

Interacting with Test Cases and Runs

Metrics provided by Fairo can be used in test cases and runs programmatically as well as through the ui.
Let's start with the same headers as before.

headers = {
  'Content-Type': 'application/json',
  'Authorization': 'Bearer [token]',
  'x-api-key-id': [key_id],
  'x-api-key-secret': [key_secret],
}

Creating the Test Case Category

First, create a test case category for the new test case.

# create test case category 
payload = {
    "name": "basic test cases"
}
url = base_api_url + "/test_case_categories"
response = requests.request("POST", url, headers=headers, data=json.dumps(payload))
test_case_category_id = json.loads(response.text)['id']

Getting other object ID's

Next, get the required parameters needed for your test case. In this example, we are using the Overall Accuracy metric meaning we will need the parameters df, y_true, and labels.

# get required parameters
url = base_api_url + "/parameters?fairo_data=true"
response = requests.request("GET", url, headers=headers)
parameters_list = json.loads(response.text)['results']
parameter_ids_list = []
for parameter in parameters_list:
    if parameter['name'] == 'df' or parameter['name'] == 'y_true' or parameter['name'] == 'labels':
        parameter_ids_list.append(parameter['id'])

Let's also get the id for Overall Accuracy.

# get metric id
url = base_api_url + "/metrics?fairo_data=true"
response = requests.request("GET", url, headers=headers)
metrics_list = json.loads(response.text)['results']
for metric in metrics_list:
    if (metric['name'] == 'Overall Accuracy'):
        metric_id = metric['id']

Creating the Test Case

Now we can create the test case with the given metric id and parameter ids.

# create test case
payload = {
    "name": "accuracy test case",
    "description": "model must have above minimum accuracy",
    "steps": "compute accuracy and compare",
    "category": test_case_category_id,
    "associated_metric": metric_id,
    "required_parameters": parameter_ids_list,
    "operator": ">",
    "test_value": ".5"
}
url = base_api_url + "/test_cases"
response = requests.request("POST", url, headers=headers, data=json.dumps(payload))
test_case_id = json.loads(response.text)['id']

This example test case requires that the accuracy of the model is greater than .5, a rather underwhelming benchmark. Note the inclusion of test_case_category_id, metric_id, and parameter_ids_list from above.

Creating the Test Run

Lastly, we can create the test run which takes many test cases to perform. In this example only one test case is provided, that being the one created above.

# create test run
payload = {
    "name": "accuracy test run",
    "description": "running accuracy test",
    "required_test_cases": [test_case_id]
}
url = base_api_url + "/test_runs"
response = requests.request("POST", url, headers=headers, data=json.dumps(payload))

Again, note the use of test_case_id.