Yodlee and Rails Implementation, Part 5: Retrieving Data

In previous posts, we've handled user accounts as well as rendering the bank login forms. We are finally ready to submit a user's bank login information back to Yodlee and retrieve transaction data.

Creating the Account Model

Let's start by setting up an Account model. Each user can have many accounts (Citibank, American Express etc) and we will store the top-level information for each account in the Account model.

rails g model Account user:references bank:references yodlee_id:integer status_code:integer last_refresh:datetime

Each account belongs to both a user and a bank. Yodlee will give each account an ID number, which Yodlee calls "itemId" or "memItemId" and which we store in the database yodlee_id field. The status_code will store the status code returned by Yodlee on the last attempted transaction pull, along with the time as last_refresh.

Very Important: Before migrating this model, open the migration file and add to the status_code column a default value of 801. We will discuss status codes in more detail shortly, but 801 is the Yodlee status code for an account that has never been "refreshed"; that is, an account whose transactions have never been scraped yet. Your migration should therefore look like this:

class CreateAccounts < ActiveRecord::Migration
  def change
    create_table :accounts do |t|
      t.references :user, index: true
      t.references :bank, index: true
      t.integer :yodlee_id
      t.integer :status_code, :default => 801
      t.datetime :last_refresh
      t.text :last_mfa

      t.timestamps
    end
  end
end

Keep Using the Pattern

Following the pattern we have used many times now, we are going to make a Yodlee::Account class that collaborates with our ActiveRecord Account model.

First, let's update our ActiveRecord Account model with a yodlee method as we've done before:

app/models/account.rb
class Account < ActiveRecord::Base

  # Associations
  belongs_to :user
  belongs_to :bank

  # Methods
  def yodlee
    @yodlee ||= Yodlee::Account.new(self)
  end

end

Now, let's create a Yodlee::Account class. I've filled it in with a few small convenience methods.

app/models/yodlee/account.rb
module Yodlee
  class Account < Base

    attr_reader :account

    def initialize account
      @account = account
    end

    def bank
      account.bank
    end

    def user
      @user ||= account.user
    end

    def item_id
      account.yodlee_id
    end

    def token
      user.yodlee.token
    end

  end
end

Quick Refresher

For the next part to make sense, we need to do a quick refresher on how our code works thus far.

A. For each bank's login form, Yodlee provides us with a JSON object, along the lines of:

{
   "conjunctionOp":{
      "conjuctionOp":1
   },
   "componentList":[
      {
         "valueIdentifier":"LOGIN",
         "valueMask":"LOGIN_FIELD",
         "fieldType":{
            "typeName":"TEXT"
         },
         "size":20,
         "maxlength":40,
         "name":"LOGIN",
         "displayName":"User ID",
         "isEditable":true,
         "isOptional":false,
         "isEscaped":false,
         "helpText":"41348",
         "isOptionalMFA":false,
         "isMFA":false
      },
      {
         "valueIdentifier":"PASSWORD",
         "valueMask":"LOGIN_FIELD",
         "fieldType":{
            "typeName":"IF_PASSWORD"
         },
         "size":20,
         "maxlength":40,
         "name":"PASSWORD",
         "displayName":"Password",
         "isEditable":true,
         "isOptional":false,
         "isEscaped":false,
         "helpText":"41347",
         "isOptionalMFA":false,
         "isMFA":false
      }
   ],
   "defaultHelpText":"4"
}

B. Our Yodlee::Bank login_requirements method converts this to HTML:

bank.yodlee.form(wrapper: 'account')
# => "<div class='field'> <label>User ID *</label> <input class='string required' id='account[LOGIN]' name='account[LOGIN]' size='20' type='text' maxlength='40' value='' /> </div> <input class='string required' id='account[PASSWORD]' name='account[PASSWORD]' size='20' type='password' maxlength='40' value='' />"

C. This form HTML will be presented to the user in a view. (I'm not discussing views at all in this series; only the back-end models. I'm confident any Rails developer with a little experience can handle the views and controllers.) The user will fill it in and it gets posted back to a Rails controller. Following Rails convention, that means we will receive a params hash, with params[:account] returning a { :username => 'foo', :password => 'bar' } hash.

D. Now that we realize that our data input is going to consist of a hash like { :username => 'foo', :password => 'bar' }, let's create a method in our Yodlee::Account class that can accept that hash and transform it into the input needed by Yodlee to connect to that bank account and scrape it.

Parsing a User's Login

Going forward, I am going to use American Express as the example in this article, as I happen to have an American Express account. We will assume that I have an American Express account with the username "foo" and password "bar". (It is worth noting that Yodlee has a test bank you can use called DAG. Read more about it here.)

The API call we use to add a user's bank account is addItemForContentService1. Take a moment to visit that link and inspect the inputs Yodlee requires.

The first three inputs required by Yodlee are the app's cobrand token, the user's token, and the ID of the bank that this account is for (eg American Express). That makes sense.

Next, for each field in the bank's login form, Yodlee requires us to send back all the data it gave us for that field, such as displayName, helpText, maxLength and so on, along with the user's input for that field (eg "foo" for username).

So, being that we know that our Rails controller will provide us with the user's form input with a hash like { :username => 'foo', :password => 'bar' }, we need to create a method that takes the Yodlee login form data, interpolates the user input { :username => 'foo', :password => 'bar' } hash into it, and then submits the whole thing back to Yodlee.

To do that, we add the following two methods to the Yodlee::Account class.

app/models/yodlee/account.rb
def parse_creds creds
  all_fields = bank.yodlee.login_requirements.componentList
  creds.each_with_index.inject({}) do |sum, (cred, index)|
    key, value = cred
    field = all_fields.find { |f| f.valueIdentifier == key }

    sum[:"credentialFields[#{index}].fieldType.typeName"] = field.fieldType.typeName
    sum[:"credentialFields[#{index}].value"] = value

    %w( displayName helpText maxlength name size valueIdentifier valueMask isEditable isOptional isEscaped isMFA isOptionalMFA ).each do |attr|
      sum[:"credentialFields[#{index}].#{attr}"] = field.send(attr)
    end

    sum
  end
end

def create creds
  response = query({
    :endpoint => '/jsonsdk/ItemManagement/addItemForContentService1',
    :method => :POST,
    :params => {
      :cobSessionToken => cobrand_token,
      :userSessionToken => token,
      :contentServiceId => bank.content_service_id,
      :shareCredentialsWithinSite => true,
      :startRefreshItemOnAddition => false,
      :'credentialFields.enclosedType' => 'com.yodlee.common.FieldInfoSingle'
    }.merge(parse_creds(creds))
  })

  if response
    account.update_attributes!(yodlee_id: response.primitiveObj)
    refresh
    ping
  end
end

The second method above, create, is the important one, and the only one we will use directly. The controller will pass it the credentials hash { :username => 'foo', :password => 'bar' }. It then posts those credentials to the addItemForContentService1 API. How? Look carefully at the params hash in the create method. It ends with .merge(parse_creds(creds)) - in other words, it takes { :username => 'foo', :password => 'bar' }, passes it to the parse_creds method to convert it to the format Yodlee expects, and mixes the result into the params posted to the addItemForContentService1 API.

Finally, the create method does three things after it submits the user credentials to Yodlee:

  1. Yodlee will return an ID number as the response for the addItemForContentService1 API. This ID number represents the account, and we save it in the yodlee_id field
  2. Calls the refresh method (coming up next)
  3. Calls call the ping method (coming up next)

Refresh and Ping

Every time we want to pull fresh transaction data from this account, we need to tell Yodlee to do a "Refresh". This gets Yodlee's scrapers to re-login to the user's bank account and scrape the newest data. There is a specific API for triggering a Refresh.

Now the refresh process can take a few seconds and we have to wait until it is done to fetch any transaction data. That is where the ping method comes in. It keeps pinging the isItemRefreshing API to find out if the account has finished refreshing, and once it has, it checks for its latest data.

app/models/yodlee/account.rb
def refresh
  query({
    :endpoint => '/jsonsdk/Refresh/startRefresh7',
    :method => :POST,
    :params => {
      :cobSessionToken => cobrand_token,
      :userSessionToken => token,
      :itemId => item_id,
      :'refreshParameters.refreshMode.refreshMode' => 'NORMAL',
      :'refreshParameters.refreshMode.refreshModeId' => 2,
      :'refreshParameters.refreshPriority' => 1
    }
  })
end

def is_refreshing?
  response = query({
    :endpoint => '/jsonsdk/Refresh/isItemRefreshing',
    :method => :POST,
    :params => {
      :cobSessionToken => cobrand_token,
      :userSessionToken => token,
      :memItemId => item_id
    }
  })

  response.primitiveObj
end

def ping
  sleep(2)
  if is_refreshing?
    ping
  else
    get_last_refresh_info
  end
end

def get_last_refresh_info
  response = query({
    :endpoint => '/jsonsdk/Refresh/getRefreshInfo1',
    :method => :POST,
    :params => {
      :cobSessionToken => cobrand_token,
      :userSessionToken => token,
      :'itemIds[0]' => item_id
    }
  })

  if response && response = response.find { |a| a.itemId == item_id }
    account.status_code = response.statusCode
    account.last_refresh = Time.at(response.lastUpdateAttemptTime)
    account.save
  end
end

Have a look at the ping method. It waits two seconds, then checks if the account is still refreshing. If it is, it calls itself again, triggering another two second wait, and so on. When the account finishes refreshing, the ping method calls the get_last_refresh_info method.

get_last_refresh_info uses the getRefreshInfo1 API to get the newest data on the account. The only two data points we are concerned with are the account's last refresh time, which we save so that when you go to production you can always check when the account data was fetched last, and the account's newest status code. Remember that our status code was initially 801, which means "never refreshed". If the account credentials were submitted accurately, that status code should now be "0", which means that the refresh/add-account was successful . If something has gone wrong, the status code will be something else. You can see a full list of status codes here.

Note: For development purposes, we use sleep(2) in the ping method to create the delay we need. In production you should certainly not do this, as this would lock up your Rails app. If an account took 30 seconds for the initial refresh to finish, no other users would be able to use your app during those 30 seconds. In a production environment you should use a background process instead, like Delayed Job, Resque, or Sidekiq.

WHEW!

Baby, was that a lot of code! Let's give it a whirl. I am going to use my American Express account, but obviously you should use whatever bank you have, or use the Yodlee test bank DAG.

In the console:

reload!
user = User.last
bank = Bank.find_by_content_service_id(12)

# First, create a new account
account = Account.create!(user: user, bank: bank)

# Then submit the login credentials. You must use the correct field names for each bank.
# Run bank.login_credentials to see that American Express names it's fields LOGIN and PASSWORD
account.yodlee.create({ "LOGIN" => "foo", "PASSWORD" => "bar" })

If all goes well, this should now trigger a whole bunch of API calls. First, you will see an API call fetching the login form data (courtesy of parse_creds), then the call to addItemForContentService1, a couple to isItemRefreshing, and finally to getRefreshInfo. If all has went well, you should now have a status code of 0.

account.reload.status_code # => 0

Success at last!

Yodlee has scraped our data. So how do we retrieve it?

Retrieving Transactions

Yodlee provides a number of different API calls to retrieve transaction data and it's worth taking time to explore them all to find which works best for you.

As an example though, we will use executeUserSearchRequest. Add this method to the Yodlee::Account class.

app/models/yodlee/account.rb
def transaction_data
  query({
    :endpoint => '/jsonsdk/TransactionSearchService/executeUserSearchRequest',
    :method => :POST,
    :params => {
      :cobSessionToken => cobrand_token,
      :userSessionToken => token,
      :'transactionSearchRequest.containerType' => 'All',
      :'transactionSearchRequest.higherFetchLimit' => 500,
      :'transactionSearchRequest.lowerFetchLimit' => 1,
      :'transactionSearchRequest.resultRange.endNumber' => 100,
      :'transactionSearchRequest.resultRange.startNumber' => 1,
      :'transactionSearchRequest.searchClients.clientId' => 1,
      :'transactionSearchRequest.searchClients.clientName' => 'DataSearchService',
      :'transactionSearchRequest.userInput' => nil,
      :'transactionSearchRequest.ignoreUserInput' => true,
      :'transactionSearchRequest.searchFilter.currencyCode' => 'USD',
      :'transactionSearchRequest.searchFilter.postDateRange.fromDate' => 1.year.ago.strftime('%m-%d-%Y'),
      :'transactionSearchRequest.searchFilter.postDateRange.toDate' => Time.zone.now.strftime('%m-%d-%Y'),
      :'transactionSearchRequest.searchFilter.transactionSplitType' => 'ALL_TRANSACTION'
    }
  })
end

Let's head back to the console and retrieve that data!

reload! 
account = Account.last
account.yodlee.transaction_data

Aha! Look at all that data! Delicious :)

Wrapping Up

As always, compare your code to my gist to make sure it's right:

The Final Wrap Up

So there you have it - detailed transaction data. Inspect the data in the log we created earlier so you can see what type of data is returned and what is available.

Before we end the series, I'd like to touch on two things.

Always Use Conditionals

When writing code to commit transaction data to your database, always use conditionals: always write if statements - if the field is present, commit it. Why? As Yodlee deals with tons of banks, the format of the data returned isn't always consistent. For example, just because fetching transactions from American Express can include a "running balance" field on each transaction doesn't guarantee that a Citibank account data dump will have a "running balance" field. Adding conditionals will ensure your users don't hit 500 errors.

MFA Banks

At this point, I need to discuss MFA: Multi-Factor Authentication.

Multi-factor authentication is the term given when a bank requires a user to supply more than just a username and password. For some banks, that means that the user must answer security questions. For others, the user may need to supply a security token, identify an image, or solve a captcha.

Throughout this series, we have only dealt with the API flow for banks that do not use multi-factor authentication. But many banks do have multi-factor authentication. Submitting account credentials to Yodlee for MFA banks uses a completely different API flow: addItemForContentService1, startRefresh7, getMFAResponse, putMFARequest, and getRefreshInfo1.

You can learn more about MFA on the Yodlee developer portal here and see a flowchart of the MFA API flow here.

So Long, Folks

And.... we're done. If you've made it till here and found this tutorial useful, please shout in the comments. I'd love to hear from you.