¶ Generating an API token
Step-by-step instructions on how to get Go-API access token
¶ Requirements
- An API tool: For this walkthrough, we will be using Postman, a popular service with a free account option. There are many other services and the basics steps outlined here should also work with them as well.
- GO Login Credentials: In order to generate a token, you will need to have a GO account with Red Cross Red Crescent permissions.
¶ What are Authentication Tokens?
Information in GO is segregated by audience. Some pieces of information are fine for the general public to access, while other pieces are limited to only members of the Red Cross Red Crescent Movement. When you create a user profile on GO as a member of a national society, your account is given access to that more private level of information, and when we access that data via the API, the system needs a way to verify that we are permitted to see it. That’s where tokens come in.
A token is passed to the server along with whatever your search request is, as a way to tell it that we’re allowed to see what we’re requesting. Think of it like a secret handshake to get into the members-only club.
¶ How Do I Get An Authentication Token?
There are 4 key elements for interacting with an API:
- The endpoint
- The method
- The headers
- The data (or body)
¶ Endpoint
The endpoint here is :
Here https://goadmin.ifrc.org is the base URL which is telling the request to ping the GO server. The /get_auth_token is accessing the token generator.
¶ Method
We can generate a token with a special API call. There are a number of methods of interacting with APIs depending on what we are trying to do with the data on the server. GETrequests are the most commonly used methods in APIs and websites. The GET method is sued to retrieve data from a server at a specified resource. But in some scenarios, including the generation of a token, we need to use the POST method. POST request are used to send data to the API server to create or update a resource.
¶ Headers
Headers are additional pieces of information that are sent along with the requests to the API server. They can be used to control the behavior of the server in handling requests. We can add the following headers to our API requests:
Key (Header Name): Content-Type Value(Header Value): application/json
Header tells the server what media type it is dealing with, and then that the data should be formatted as JSON. Other media types include:
- audio
- font
- example
- image
- message
- model
- multipart
- text
- video
¶ Generating authorization token using curl method
- Open the Command Prompt on your Windows machine.
- Use the
curlcommand followed by the URL to which you want to send the POST request. To include data in the request body, you can use the-doption followed by the data. - If you need to include headers in the request, you can use the
-Hoption followed by the header.
curl -X POST -H "Content-Type: application/json" -d "{\"username\": \"your GO username\", \"password\": \"your GO password\"}" https://goadmin.ifrc.org/get_auth_token
- In this command, the JSON payload (
{"username": "your GO username", "password": "your GO password"}) is enclosed within double quotes (\"). This ensures that any special characters within the payload, such as!,@,#, are properly escaped. - Replace the
{\"username\": \"your GO username\", \"password\": \"your GO password\"}with the JSON data you want to include in the request body. In this case your GO username and password. - Press Enter to execute the command. You should see the response returned by the server in the command prompt. Using
curlfrom the command line gives you flexibility in making various types of HTTP requests, including POST requests, with different headers and data payloads.
¶ Generating authorization token in Postman
Postman can be downloaded on Windows, MacOS and Linux operating systems from there official website https://www.postman.com/downloads/. API token can be generated by following steps:
- Log into Postman. At the top left of the window click
New. - In the window that pops up, click on
Collection. This will create a new collection in which API request can be stored.
- On the left side of the window, the collection will appear. Under the name of your collection click
Add a request. - In the main window that appears, click on GET and from the drop down window select
POST. - In the field labelled as Enter URL or paste text, type
https://goadmin.ifrc.org/get_auth_token.
- Now click on
Body, and then selectrawfrom the menu below. The right most option should showJSONas this is the format in which we will have to input the go username and password. IfJSONis not visible, then click on the rightmost option and from the dropdown menu selectJSONoption. - Then enter
{"username": “your-username”, "password": “your-password”}. In the field“your-username”enter your go username or email id and in the field “your-password” enter the go password. Remember to keep the quotes around all the elements.
- In the headers tab ensure that the key
Content-Typehas the Valueapplication/jsonand the keys labelled asContent-Type,Content-LengthandHosthave the checkmark.
- After these steps, click on
Sendoption in the top right corner. If you have entered everything correctly, then you should get the following response from the server under theBody.
{
"token": "your authorization token will appear here",
"username": "arun.gandhi",
"first": "Arun ",
"last": "Gandhi",
"expires": "2023-08-29T12:29:56.844Z",
"id": 9634
}
¶ Using Authorizing Token
The authorization token can be used to authenticate when getting data from any of the GO API endpoints. Authorization token enables to access the data that is not public. Data can still be accessed without using authorization token, however only the public data will be visible. Here we show how can you access the data corresponding to events in regions or countries.
¶ Using authorization token in Power BI/ Excel
The API can be accessed by building a URL query in Power BI/ Excel. We start with the base URL to access the database and then set the parameters to select just the data that we need. In this example we will be accessing the Events and then set the parameters regions__in (integer) to 2 (Asia Pacific) and then set the countries__in (integer) parameter to 84 (country id) so as to access the events occurring in India. The regions__in parameter can take values from 0 to 4 (integer) where 0 corresponds to “Africa”, 1 corresponds to “Americas”, 2 corresponds to “Asia Pacific”, 3 corresponds to “Europe” and 4 corresponds to “Middle East & North Africa”.
So our URL for this particular search would like this:
https://goadmin.ifrc.org/api/v2/event/?regions__in=2&countries__in=84
- Here
https://goadmin.ifrc.org/api/tells PowerBI (or any other service/tool referencing the API) to connect with the GO server. All of your API calls to GO will start with this as a base URL. - The
v2/event/refers to the latest version of the API. There are multiple data tables within the database and event tells GO to search for the table that stores events data ( as opposed to , say appeals, or the deployments. See https://goadmin.ifrc.org/api-docs/swagger-ui/ to see a full list of the available API endpoints. - The
?regions__in=2&countries__in=84are the 2 parameters that have been used to filter out the data for a particular region and country. The parameters are telling us to return the data corresponding to the Asia Pacific region and India. countries__in takes an integer and it corresponds to the country id. The country id of India is 84.
¶ Getting your data in PowerBI/ Excel
To get your data in PowerBI/ Excel using the authorization token proceed with the following steps:
- Open PowerBI/ Excel and select Get Data from the top ribbon, and then select Web.
- Select the Advanced option from the window that appears.
- Under the URL parts option, add the base URL in the first box and the URL corresponding to the parameters in the second box.
- Once you have added both the URL parts, you will be able to see the complete URL that will be used for querying the data under the URL preview option.
- To use the authorization token you will have to enter the authorization token under the HTTP request header parameters option. In the box on the left side type Authorization and in the right box type Bearer followed by your authorization token and then click OK.
- If the authorization token is valid this will open a new window called Power Query editor where you can organize the data that is returned in preparation for visualization.
- Upon clicking OK the Power query editor opens and a bunch of transformations are already applied by Power Query. It is a good idea to give your query a name before you start bringing a lot of queries. So on the left pane with query selected, look at the right side of the window and rename it to something like India Events.
- The data comes into PowerBI in JSON format, which makes transferring the data easy, but can be difficult to easily interpret. Our India Events has returned 50 lines of data and we are accessing the data from the first page of the API (see the section named
Paginationto access data from all the pages in the API endpoint). - We can expand the other columns like results.countries. The results.countries is a placeholder for the data for that particular country. Click on the button right to the column name and select Expand to New Rows. Similarly you can expand any other columns which were not expanded by the Power Query.
- Click again on the button on the right side of the column name and a dialog box will appear showing the data fields available. You can select any number of fields and then click ok.
- Now we are able to see the data which this query is actually providing us. Click on Close and Apply at the top of your Power Query editor window. Now our data is ready for building visuals.
¶ Using authorization token in python
We can also use python for extracting data from any GO-API endpoint. We use the python code below to obtain data from the Event API endpoint. Since the API is paginated, the python code below shows how to get the data from the first page.
import requests
import pandas as pd
api_url = "https://goadmin.ifrc.org/api/v2/event/"
auth_token = "Your authorization token"
First we import the necessary python packages and specify the API URL and authorization token.
requests: Used for sending HTTP requests and handling responses, commonly used for web scraping, API interaction, and downloading web content.pandas: Provides data structures and data analysis tools for working with structured data.
def fetch_data(limit=50, offset=0, regions__in=None, countries__in=None):
params = {"limit": limit, "offset": offset}
if regions__in or countries__in:
params["regions__in"] = regions__in
params["countries__in"] = countries__in
headers = {"Authorization": f"Token {auth_token}"}
response = requests.get(api_url, params=params, headers=headers)
if response.status_code == 200:
return response.json()
else:
print(f"Failed to fetch data. Status code: {response.status_code}")
return None
# Fetch data from the first page
first_page_data = fetch_data(limit= 50, offset=0, regions__in="2")
if first_page_data:
# Process the data from the first page
results = first_page_data.get('results', [])
# Do whatever you need to do with the results
df = pd.DataFrame(results) # convert the JSON object into pandas dataframe We define a fetch_data function to obtain data from the API. Inside the fetch_data function, we specify the authorization token in the headers dictionary.
In order to get the data from any other page, we can change the offset value. Offset value of 0 suggest that the data is coming from the first page. Incrementing the offset by 50 will allow the code to access the data from subsequent pages. Limit tells how many records to display. For example if we want data from the second API page we need to set the offset as 100. In general the fallowing formula can be used to set the value of Offset when accessing data from any API page.
Offset = (Page_num) - 1 * 50
¶ Pagination
In order to handle large datasets our API supports Pagination. Pagination allows you to retrieve the subsets of data, making it easier to manage and process. It also has filtering and sorting capabilities to search the endpoints.
¶ Pagination Query parameters
Pagination is deactivated by default and must be specified in each API request. There are 4 available query parameters that you may use namely, offset, limit, next and previous.
limit (optional):
Description: It determines the number of records that will be retrieved.Data type: IntegerExample: limit=10Note: The default value if 50. The maximum value is 1000
offset (optional):
Description: It determines how many records will be skipped before being included in the returned results. It specifies the starting point from where the data should be fetched.Data type: IntegerExample: offset=50Note: if not provided, the default is 0
For example if there are a total of 100 records, by setting offset to 50 and limit to 20, we are returning a total of 20 records from the second set of records. It means that a total of 20 records will be returned from the second page of the API endpoint.
next (optional):
Description: Provides the URL to fetch the next page of the dataData type: String (URL)Example:https://goadmin.ifrc.org/api/v2/personnel/?limit=50&offset=150Note: The parameter is present only if the next page is available
previous (optional):
Description: Provides the URL to fetch the previous page of the dataData type: String (URL)Example:https://goadmin.ifrc.org/api/v2/personnel/?limit=50&offset=50Note: The parameter is present only if there is a previous page is available.
¶ Pagination best practices
It is very important to verify whether the correct number of records have been retrieved from the database. Once you have used the Pagination query parameters to retrieve all the records from the database, you no longer need to make additional API requests.
To determine if additional API requests are needed, you should compare the value of limit parameter with the total number of obtained records. If the number of records are less than the limit parameter's value, then no additional requests are required.
If the number of records is equal to the limit parameter's value, then you may need to send additional API requests. It is also important to check the value of the offset parameter. An offset value of 0 indicates that the data is fetched from the first first page.
If you see an empty array [] in the body of the response, it means that there are no records for that request and you don't need to send additional API request.
¶ Example Usage
Request
curl -X 'GET' \
'https://goadmin.ifrc.org/api/v2/event/?imit=100&offset=0&l®ions__in=2&countries__in=84' \
-H 'accept: application/json'
Response
Response body
Download
{
"count": 78,
"next": null,
"previous": null,
"results": [
{
"dtype": {
"id": 4,
"name": "Cyclone",
"summary": "",
"translation_module_original_language": "en"
},
"countries": [
{
"iso": "IN",
"iso3": "IND",
"id": 84,
"record_type": 1,
"record_type_display": "Country",
"region": 2,
"independent": true,
"is_deprecated": false,
"fdrs": "DIN001",
"average_household_size": null,
"society_name": "Indian Red Cross Society",
"name": "India",
"translation_module_original_language": "en"
}
(...)
In the example, the API response we fetch the data from the Event API endpoint. The data is obtained based on the previous and next parameters. The countries__in and regions__in have been used to filter the data. limit==100 means that we are obtaining 100 records and offset=0 means that data is fetched from the first till the last page. In the response we obtain only 78 records because these records have been filtered out for Nepal using the countries__in and regions__in parameters.
¶ Pagination in Power BI/ Excel
Whenever you are using PowerBI or any other service by default it will be accessing and showing you the data from the first page of the API endpoint which contains 50 records by default. If you want to access all the data coming stored on any particular API endpoint, this section shows you how you can do that in PowerBI/ Power Querry.
Using the steps below you can retrieve the data from the API using pagination along with authorization token and parameters. In this example we will be accessing the Events data and will use regions__in and countries__in as parameters.
- In order to access the data from all the API pages for a particular API end point, instead of selecting Get Data from the top ribbon, select Transform data. This will open the Power query editor.
- We first define a Parameters table which will contain the Authorization token,
regions__inandcountries__inparameters that will be used in querying the data from the events API endpoint. - Click on Enter Data under the Home ribbon. This will open a window where a table can be created by entering the Authorization token and the two other parameters.
- Lets leave the
regions__inandcountries__inparameters as blank. - Upon clicking OK, a new query will be created under the Queries window. Lets rename this query to Parameters. The query can be renamed by changing its name under the Properties situated on the right side of the editor window.
- Under the Home ribbon, from New Source, select Blank Query. Click on Advance editor, this will open the advance query editing dialog box. Advance editor uses the m language to edit the query. In the advance editor window, paste the following code:
let
// Get the parameter values
ParametersTable = Parameters,
Token = Table.SelectRows(ParametersTable, each [Column1] = "Token"){0}[Column2],
region = Table.SelectRows(ParametersTable, each [Column1] = "regions__in"){0}[Column2],
country = Table.SelectRows(ParametersTable, each [Column1] = "countries__in"){0}[Column2],
BaseUrl = "https://goadmin.ifrc.org",
RelativePath = "api/v2/event/",
PageSize = 50, // Number of items per page
// Construct the query based on parameter values
queryOptions =
[
countries__in = if country <> null then Text.From(country) else "",
regions__in = if region <> null then Text.From(region) else "",
limit = Text.From(PageSize),
offset = "0" // Set initial offset to 0 as a string
],
GetPage = (offset) =>
let
Url = BaseUrl,
Headers = if Token <> null then [#"Accept-Language"="en", #"Authorization"="Token " & Token] else [#"Accept-Language"="en"],
Response = Json.Document(Web.Contents(Url, [Headers=Headers, RelativePath = RelativePath, Query=queryOptions & [offset = Text.From(offset)]])),
Results = Response[results]
in
Results,
CombineAllPages = List.Generate(
() => [Offset = 0],
each let page = GetPage([Offset]) in page <> null and List.Count(page) > 0,
each [Offset = [Offset] + PageSize],
each GetPage([Offset])
),
#"Converted to Table" = Table.FromList(CombineAllPages, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1")
in
#"Expanded Column1"
- In the code above the Token, region and country variables derive their values from the Parameters table that was constructed in the previous steps.
- We also define the BaseUrl which tells the PowerBI to connect with the Events API endpoint on the GO server. To see the full details of the API endpoints visit
https://goadmin.ifrc.org/api-docs/swagger-ui/. - The number of records on every page are stored in the PageSize variable. Since each API page contains 50 records, set this variable to 50.
- The
queryOptionsvariable in this code is constructing the filter query parameter that will be appended to the API URL to filter the results. It is checking if theregionandcountryparameter values are null or not, and constructing different filter values based on that. The filter allows the query to the API to be customized based on the input parameters, to filter the results returned from the API. - The
GetPagefunction is responsible for constructing the full API URL for a specific page, making the API request based on the parameters used, parsing the response and returning just the array of results for that page. - The
CombineAllPagesfunction is responsible for looping through all available pages of results from the API and combining them into a single list. It loops through all the available pages by incrementing the page counter. It callsGetPageto fetch each API page, stops when theGetPagereturn 0 and accumulates all the results into a single list. - After the code has been pasted in the advance editor window, click on done. A series of steps will be added in the APPLIED STEPS on the right side of the query editor window.
- Since the Events API endpoint consists of 102 pages at present, we can see 102 rows in the data obtained. Having a list with record does not tell us much. Converting it to a table will make it easier to read so click on the Convert to Table in the Transform section of the ribbon at the top of the window. In the window that pops up, select None and Show as errors for the two options.
- Right now our table has an index column and Column1 which is just a place holder for the data for that particular result. Next to the name of the column click on the button and select Expand to new rows. Now we can see all the records obtained from every page. Again click on the button next to the name of the column and we are able to see what data this query is providing us. We select the fields which we would like to see in our data and click ok.
Note: We can also specify the values of
regions__inandcountries__inparameters in the Parameters Table. This will further filter the data from all the API pages. For example if we specify theregions__in = 2andcountries__in = 84, then we will obtain 77 records.
¶ Pagination in Python
Here we show how can you retrieve data from the GO API using python. We will be retrieving the data from all the pages of the Event API endpoint and will also filter the data using the regions__in and countries__in parameters. First we import the necessary python packages that will be used to retrieve the data.
import requests
import multiprocessing
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from functools import partial
import pandas as pd
The following packages have been imported :
requests: Used for sending HTTP requests and handling responses, commonly used for web scraping, API interaction, and downloading web content.concurrent.futures.ThreadPoolExecutor: Part of theconcurrent.futuresmodule, used for managing a pool of worker threads to execute functions concurrently.functools.partial: Allows you to create partial functions by fixing a certain number of arguments of a function and generating a new function.pandas: Provides data structures and data analysis tools for working with structured data.
Next we specify the API URL and Authorization token. The Page_limit is set to 50 because each page contains 50 records.
api_url = "https://goadmin.ifrc.org/api/v2/event/"
auth_token = "Your Authorization token"
page_limit = 50 Next we define a function called fetchpage which will be used to retrieve the data from the API pages. In this function we also specify our filtering parameters namely regions__in and countries__in which with the help of which we will be filtering out data. We also specify a header dictionary in which we provide the authorization token.
def fetch_page(api_url,page_num, limit=50, offset=0, regions__in=None, countries__in=None):
params = {"limit": limit, "offset": offset + limit * (page_num - 1)}
if regions__in or countries__in:
params["regions__in"] = regions__in
params["countries__in"] = countries__in
headers = {"Authorization": f"Token {auth_token}"}
response = requests.get(f"{api_url}", params=params, headers=headers)
return response.json()
Note: The data from the GO API can be requested in 4 supported languages namely English, French, Arabic and Spanish. If you want the API response in the any of the 4 languages, you can specify another header in the fetch_page function as shown below:
headers = { "Authorization": f"Token {auth_token}", "Accept-Language": "two letter language code" # Add the Accept-Language header with the desired language }
Since the API endpoint contains over 100 pages, we will be parallelizing the task. We first calculate the total number of API pages and will be utilizing the concurrent.futures module to parallelize the execution of a function called fetch_page across multiple threads.
max_workers = multiprocessing.cpu_count()
response = requests.get(f"{api_url}", headers={"Authorization": f"Token {auth_token}"})
if response.status_code == 200:
total_records = response.json()["count"]
else:
print(f"Failed to fetch total records count. Status code: {response.status_code}")
total_pages = (total_records + page_limit - 1) // page_limit
print("total_pages: ", total_pages, "\ntotal cpu count: ", max_workers)
with ThreadPoolExecutor(max_workers=max_workers * 4) as executor:
page_nums = range(1, total_pages + 1)
partial_fetch_page = partial(fetch_page, api_url,limit=50, offset=0)
results = executor.map(
partial_fetch_page, page_nums
) # regions__in = "2", countries__in = "84"
final_results = []
for result in results:
item = result
final_results.extend(item["results"])
The partial function is used to fix some arguments of fetch_page while leaving others as parameters to be filled later. In this case, it sets default values for limit, offset, regions__in, and countries__in.
executor.mapmethod is used to apply thepartial_fetch_pagefunction to each page number inpage_nums. This means that the fetch_page function will be executed concurrently for each page number using the thread pool. The map function returns an iterator of results.
In summary, the code is using a thread pool to fetch data from multiple pages concurrently, and it collects the results into a list called final_results.
The code below converts the retrieved data in to pandas data frame. The expand is a function which has been used to flatten out the data in the dtype, countries and appeals JSON objects. The same function can also be used to expand/ flatten out other JSON objects in the data and later the user can filter out the columns and prepare the data for visualization. Lastly the data has been exhorted to a csv format.
df = pd.DataFrame(final_results)
def expand(data_frame, column=str):
result = data_frame.explode(column)
result = result.reset_index(drop=True)
return result
# expandingh the countries and appeals columns as
# the data is stored as list
result = expand(df, "countries")
result = expand(result, "appeals")
# Get the index of the columns "countries" , "dtype" and "appeals"
countries_index = result.columns.get_loc("countries")
appeals_index = result.columns.get_loc("appeals")
dtype_index = result.columns.get_loc("dtype")
# Concatenate and expand columns
df_expanded = pd.concat(
[
pd.json_normalize(result["dtype"], sep="_").add_prefix("dtype."),
pd.json_normalize(result["countries"], sep="_").add_prefix("countries."),
result.iloc[
:, countries_index + 1 : appeals_index
], # columns between "countries" and "appeals"
pd.json_normalize(result["appeals"], sep="_").add_prefix("appeals."),
result.iloc[:, appeals_index + 1 :], # columns after "appeals"
],
axis=1,
)
#storing the data in csv format
df_expanded.to_csv("your/path/to/save/the/file/df_expanded.csv")
¶ Dataset refresh errors and Publishing Power BI report using GO API data
Sometimes loading data using M code (and in Power Query) can cause problems in refreshing the Power BI report after it has been published on PowerBI.com. The data refreshing works fine on Power BI desktop but problems can arrive during automatic refresh of a published Power BI report. The problem is whenever we try to retrieve data from an API Power BI does some static analysis on the code to determine the data sources of the dataset and whether the supplied dataset are correct. In some cases the definition of a data source depends on the parameters from a custom M function, and static analysis can fail sometimes and therefore data might not refresh on PowerBI.com. Upon trying to refresh data, we could see a rather unhelpful error message which can look like the following:
This problem usually occurs when data source is not correctly defined in the Web.Contents function. To address these problems, it is important to use Relative Path and Query options with Web.Contents function. Therefore, the following expression should be used whenever constructing a Power Query using M code while retrieving data from an API source.
Web.Contents(Url, [Headers=Headers, RelativePath = Relative path, Query=queryOptions & [offset = Text.From(offset)]])
Here URL is the base API URL on which static analyses by Power BI is performed before data refresh. For e.g. in the case of dataset coming from GO API, the URL would be https://goadmin.ifrc.org. The relative path would be /api/v2/event/. The Query would could be any search parameter that can be used for filtering down the data from the GO API, for e.g. queryOptions = [countries__in = “IND”]. This example shows the correct method of utilizing the Web.Contents function using relative path and Query options.
So when a static analyses is done, Power BI evaluated the base API URL which is https://goadmin.ifrc.org. When the data is refreshed, the search term in the query option and relative path would override the search term in the base URL.
¶ Getting API response in a supported Language
The data from any GO API endpoint can be requested in 4 supported languages. This can be accomplished by setting an Accept_language request header in the request.
At the moment the GO API supports response in 4 languages and the value of the header should be a two letter language code you want to get the response in.
- en for English
- fr for French
- es for Spanish
- ar for Arabic
¶ Getting API response in a supported language in PowerBI/ Excel
To get the data in PowerBI/ Excel follow the steps as shown on the Using Authorization token page. For example if you want to get the API response in French you will have to enter the Accept Language header under the HTTP request header parameters option. Along with the Authorization token, you have to type Accept-Language in the box on the left hand side and in the right box type the two letter keyword (here “fr”) for the language in which you want to get the response.
If the headers are entered correctly and the authorization token is valid then a new window called Power Query editor will open where the data can be organized/ transformed for visualization. Now we are able to see the data which this query is actually providing us. Since the chosen response is in French language you can see that the name of the countries will start appearing in French. Click on Close and Apply at the top of your Power Query editor window. Now our data is ready for building visuals.
To generate the API response in a supported language in python see the
Paginationsection and modify the python code accordingly.