Yodlee and Rails Implementation, Part 2: The Banks

In the previous post of this series, we learned about setting up our app to connect with Yodlee.

Let's build on that to start interacting with specific banks.

Content Service Thingies

Throughout the Yodlee documentation, Yodlee refers to banks as Content Services. That's because Yodlee doesn't just deal with banks. It can also aggregate data from stock services, investment services, and mileage point programs. It therefore uses the term Content Service as an all-encompassing label for all of these.

In this series, we will be focusing on banks and credit card services exclusively, as most startups using Yodlee are using it for banks. The API calls are identical though for the other types of content services.

Our Bank Cache

To start, we are going to fetch from Yodlee their entire list of content services, and cache a copy within our local database. Let's start by creating a bank model. Although Yodlee provides many fields of data for each bank, in our database we are only going to cache the fundamentals fields.

rails g model Bank content_service_id:integer content_service_display_name site_id:integer site_display_name mfa home_url container

Let me explain what these seven fields represent.

Yodlee keeps a separate record for each type of content service. For example, Citibank Checking, Citibank Credit Cards, and Citibank Home Loans are all separate content services, even though they are all under the Citibank umbrella. Each has its own row in the Yodlee content service database table, and each has its own ID. We save these fields as content_service_id and content_service_display_name.

Now, although Citibank Checking is a separate content service from Citibank Credit Cards, they both belong to the same umbrella site: Citibank. That is the next two fields we save, site_id and site_display_name. So think of the site_id as the overall conglomerate umbrella, Citibank, and the content_service_id as the specific service, like Citibank Checking.

Next, we use the mfa field to record whether this bank is an MFA (Multi-Factor Authentication) bank. We will discuss MFA in detail later on; for now, all you need to know is that it is important to record whether each bank is MFA or not.

The next field, home_url, is the URL of this content service, for example, http://www.citibank.com.

And the last field, container, saves what type of content service it is: a bank, a credit card, a loan bank, stocks, and so on.

Creating the Importer

Create a new file in your models/yodlee folder, and name it importer.rb

app/models/yodlee/importer.rb
module Yodlee
  class Importer < Base

    def content_services *type
      # types allowed: :all, OR specific type(s), :bank, :credits, :miles, :loans etc
      all_content_services.each do |bank|

        container = bank.containerInfo.containerName.to_sym
        next unless Array(type).include?(container) || type == :all

        mfa = bank.key?('mfaType') ? bank.mfaType.typeName : 'none'

        row = ::Bank.where(content_service_id: bank.contentServiceId).first_or_create
        row.update_attributes!(
          :content_service_id           => bank.contentServiceId,
          :home_url                     => bank.homeUrl,
          :content_service_display_name => bank.contentServiceDisplayName,
          :site_id                      => bank.siteId,
          :site_display_name            => bank.siteDisplayName,
          :container                    => container,
          :mfa                          => mfa
        )
      end
    end


    private

    def all_content_services
      query({
        :endpoint => '/jsonsdk/ContentServiceTraversal/getAllContentServices',
        :method => :POST,
        :params => {
          :cobSessionToken => cobrand_token,
          :notrim => true
        }
      })
    end

  end
end

This class is rather light. The private method, all_content_services, contains the API call to fetch the entire list of content services from Yodlee.

The public method, content_services, is the method that actually does the database persisting. Because you probably don't want to save every type of content service, it accepts a type parameter telling it which content services types to store, and which to skip. When you run content_services, it will grab the full content services data dump from all_content_services, loop through each service, commit it to the database if it is of the type you want, and skip it if it isn't.

Let's try importing banks and credit card services. In your Rails console:

reload!
Yodlee::Importer.new.content_services :bank, :credits

Tip: Running this method takes a long time. It can easily take 4-5 minutes for the API call to return the huge load of data. Yodlee is returning data on tens of thousands of banks, so the lag is understandable. This lag makes it appear as if the API call is hung. It isn't. Now is a good time to get up and make some coffee.

Note: Use this getAllContentServices API call sparingly. Running it is a heavy load both on your system and on Yodlee's, which is why we cache the data we receive from it.

After the method completes running, you should have about 5,300 rows in your Bank table.

Logging for Easy Debugging

It's time to take a step and back and re-read some of the code we used above. Have another look at the all_content_services method. This method, which is just an API call, relies on another API call to work successfully: the Cobrand Login API.

Most of the calls we do with Yodlee will be like this, where each call triggers other, lower-level APIs first. This makes debugging difficult, especially in production, as on occasion something goes wrong and you don't know which API call returned an unexpected response which then triggered a wave of issues down the chain.

Because of this, I always create a Log model where I simply store the full request and response of every call sent to Yodlee. This table is really easy to create and has saved me countless hours when debugging.

Let's create the model.

rails g model Log endpoint method params:text response_code:integer response:text

Remember how in our previous blog post we refactored our API calls to all defer to a single method, query, in our Base class? Well now that is going to pay off. Because we can add a drop of code to that query method, and all our API calls are instantly committed to the log.

First, we'll add a private method to the Base class which commits the request and response data.

app/models/yodlee/base.rb
def log_query opts
  Log.create!(
    :method        => opts[:method],
    :endpoint      => opts[:endpoint],
    :params        => opts[:params].to_json,
    :response      => opts[:response].to_json,
    :response_code => opts[:code]
  )
end

Now update the query method, adding one line to call the log_query method and log everything:

app/models/yodlee/base.rb
def query opts
  method   = opts[:method].to_s.downcase
  response = self.class.send(method, opts[:endpoint], query: opts[:params])
  data     = response.parsed_response

  # ADDING THIS LINE:
  log_query(opts.merge({response: data, code: response.code}))

  if response.success?
    if [ TrueClass, FalseClass, Fixnum ].include?(data.class)
      data
    else
      convert_to_mash(data)
    end
  else
    nil
  end
end

If we now re-run any method that calls Yodlee, it will log the request and response. In your console:

reload!
Yodlee::Base.new.cobrand_token
Log.count # should return 1
Log.last # should return the last request and response

Tip: While this logging mechanism works great for your development environment and for learning Yodlee, in production you may want to configure the Log model to use a separate database. Logs are by nature very write-heavy and you probably don't want them cluttering your real database. You could also use a key-value database like Redis for this.

Make It Practical

At this point, I really, really, REALLY recommend you take 10 minutes to make a simple frontend interface for viewing the log data we are storing. In the next tutorials, the JSON responses from Yodlee will become increasingly complex, and it will be crucial for you to have a way to view the response data easily. (Using the Rails console doesn't work well for this. There is just too much data returned.)

Here is an example of a quick bootstrap interface you could make:

index

show

Wrap Up

Now is a good time to check your code against my gists, to make sure you have everything you should:

Coming Up Next

In our next blog post, we discuss fetching login forms for banks and how to convert them into HTML.