The New Yodlee REST/JSON API and Rails

Yodlee is a service used by banks, financial institutions, and apps to aggregate financial data, including credit cards, bank accounts, stock portfolios, and loans. It is the magic that powers over 300 apps, including powerhouses like Xero, Credit Karma, and tuition.io.

On October 21st, Yodlee released a new REST/JSON API to replace its current SOAP/XML API. This is big step forward, as JSON APIs are much easier to handle than XML APIs. Here are is a breakdown of the changes.

No More SOAP Fussiness

SOAP/XML APIs are kind of a pain. They require you to submit the fields within the XML block in an exact order. JSON on the other hand is inherently an unordered object, so you can submit fields in any order and it works. In addition, SOAP APIs require you to add namespaces, and Yodlee's SOAP API in particular requires additional namespaces and special attributes on specific tags, which is tricky to get right. The JSON API dispenses with all this for a faster, easier integration.

Speed

XML responses require a gem like Nokogiri for parsing, which is much slower than parsing JSON. Similarly, manipulating a JSON object is much easier than manipulating a XML one: adding a value to a JSON object takes a second. Adding a value to a XML block requires you to create a Nokogiri node, add it to the tree, and then start shifting it around using traversal methods.

Slimmer Requests

Some of the API requests have been slimmed down considerably and require you to submit less input than the SOAP API. Having to submit less means fewer things that can go wrong. No complaints there.

The responses, on the other hand, are basically identical to the SOAP API responses (except that they're in JSON notation).

Shiny New Portal

Yodlee has also unveiled a shiny new portal to complement the new API release. The portal can be accessed at http://developer.yodlee.com. The new portal comes with improved documentation and guides, and even includes a (very basic) sample Rails app.

Instant Sandbox Accounts

Signing up at the new portal gives you instant access to an evaluation sandbox account. This is a huge deal. In the past, paying clients had access to the sandbox, which created a chicken-and-the-egg problem for developers: they wanted to learn Yodlee and market that skill to clients, but couldn't get easy sandbox access without already having a paying Yodlee client. (I believe access was available to developers who reached out to Yodlee, though.) The new open sandbox approach now lets everyone learn Yodlee. Each sandbox account comes with five fake users to interact with and expires after thirty days.

What I Would Change

Yodlee has clearly put a lot of work into this release, and its API is now easier to use. The API itself hasn't changed very much. You are making the same requests and receiving the same responses; it's just easier to make those requests. Personally, I was hoping to see some new features. As a developer who has integrated Yodlee for clients multiple times, I feel that Yodlee could greatly benefit by adding HTML Forms and Webhooks to the API.

HTML Forms

Yodlee deals with tens of thousand of banks, each of which has its own login form. Just consider the amount of variation that exists: Each bank can have any combination of regular fields, password fields, radio buttons, checkboxes, select dropdowns, multi-field fields (like a Social Security field which is made up of three sub-fields) and combinations of combinations, like a Date of Birth field, which is usually a field with three sub-field dropdowns (for day, month, and year). If that is not enough, each field can be required or optional, has a default size, a maxlength, a label value... you get the picture.

To deal with the enormous variety, Yodlee returns each bank's login form as a large, nested JSON object. (This JSON object is exactly the same as the XML object the SOAP API returns.) If you want to get an idea of how complex a form can get, consider this one:

{
  "conjunctionOp": {
    "conjuctionOp": 1
  },
  "componentList": [
    {
      "valueIdentifier": "LOGIN",
      "valueMask": "LOGIN_FIELD",
      "size": 20,
      "maxlength": 40,
      "name": "LOGIN",
      "displayName": "Username",
      "isEditable": true,
      "isOptional": false,
      "isEscaped": false,
      "helpText": "43666",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "valueIdentifier": "PASSWORD",
      "valueMask": "LOGIN_FIELD",
      "fieldType": {
        "typeName": "IF_PASSWORD"
      },
      "size": 20,
      "maxlength": 40,
      "name": "PASSWORD",
      "displayName": "Password",
      "isEditable": true,
      "isOptional": false,
      "isEscaped": false,
      "helpText": "43665",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "validValues": [
        "what city was your high school",
        "name of the first company you worked for",
        "your father's middle name",
        "your maternal grandmother's first name",
        "name of your High School"
      ],
      "displayValidValues": [
        "In what city was your high school? (full name of city only)",
        "What is the name of the first company you worked for?",
        "What is your father's middle name?",
        "What is your maternal grandmother's first name?",
        "What was the name of your High School?"
      ],
      "valueIdentifier": "OP_OPTIONS1",
      "valueMask": "LOGIN_FIELD",
      "fieldType": {
        "typeName": "OPTIONS"
      },
      "size": 20,
      "maxlength": 40,
      "name": "OP_OPTIONS1",
      "displayName": "Question 1",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43664",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "valueIdentifier": "OP_LOGIN1",
      "valueMask": "LOGIN_FIELD",
      "size": 20,
      "maxlength": 40,
      "name": "OP_LOGIN1",
      "displayName": "Answer 1",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43663",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "validValues": [
        "what city was your father born",
        "first name of the best man at your wedding",
        "first name of your first manager",
        "name of your first girlfriend/boyfriend",
        "your high school mascot"
      ],
      "displayValidValues": [
        "In what city was your father born? (Enter full name of city only)",
        "What is the first name of the best man at your wedding?",
        "What was the first name of your first manager?",
        "What was the name of your first girlfriend/boyfriend?",
        "What was your high school mascot?"
      ],
      "valueIdentifier": "OP_OPTIONS2",
      "valueMask": "LOGIN_FIELD",
      "fieldType": {
        "typeName": "OPTIONS"
      },
      "size": 20,
      "maxlength": 40,
      "name": "OP_OPTIONS2",
      "displayName": "Question 2",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43662",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "valueIdentifier": "OP_LOGIN2",
      "valueMask": "LOGIN_FIELD",
      "size": 20,
      "maxlength": 40,
      "name": "OP_LOGIN2",
      "displayName": "Answer 2",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43661",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "validValues": [
        "what city did you meet your spouse for the first time",
        "what city was your mother born",
        "what city were you born",
        "your mother's middle name",
        "your favorite restaurant in college"
      ],
      "displayValidValues": [
        "In what city did you meet your spouse for the first time?",
        "In what city was your mother born? (Enter full name of city only)",
        "In what city were you born? (Enter full name of city only)",
        "What is your mother's middle name?",
        "What was your favorite restaurant in college?"
      ],
      "valueIdentifier": "OP_OPTIONS3",
      "valueMask": "LOGIN_FIELD",
      "fieldType": {
        "typeName": "OPTIONS"
      },
      "size": 20,
      "maxlength": 40,
      "name": "OP_OPTIONS3",
      "displayName": "Question 3",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43660",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "valueIdentifier": "OP_LOGIN3",
      "valueMask": "LOGIN_FIELD",
      "size": 20,
      "maxlength": 40,
      "name": "OP_LOGIN3",
      "displayName": "Answer 3",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43659",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "validValues": [
        "what did you want to be when you grew up",
        "first name of your oldest nephew",
        "your paternal grandfather's first name",
        "your paternal grandmother's first name",
        "name of the town your grandmother lived in"
      ],
      "displayValidValues": [
        "As a child, what did you want to be when you grew up?",
        "What is the first name of your oldest nephew?",
        "What is your paternal grandfather's first name?",
        "What is your paternal grandmother's first name?",
        "What was the name of the town your grandmother lived in? (Enter full name of town only)"
      ],
      "valueIdentifier": "OP_OPTIONS4",
      "valueMask": "LOGIN_FIELD",
      "fieldType": {
        "typeName": "OPTIONS"
      },
      "size": 20,
      "maxlength": 40,
      "name": "OP_OPTIONS4",
      "displayName": "Question 4",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43658",
      "isOptionalMFA": false,
      "isMFA": false
    },
    {
      "valueIdentifier": "OP_LOGIN4",
      "valueMask": "LOGIN_FIELD",
      "size": 20,
      "maxlength": 40,
      "name": "OP_LOGIN4",
      "displayName": "Answer 4",
      "isEditable": true,
      "isOptional": true,
      "isEscaped": false,
      "helpText": "43657",
      "isOptionalMFA": false,
      "isMFA": false
    }
  ],
  "defaultHelpText": "8373"
}

Yeah, exactly.

The correct way to handle this within your app is to build a smart form parser that can take in the JSON object and convert it, piece by piece, to HTML. I'll share my form parser code in a future blog post, but an inexperienced developer would be understandably intimidated.

This is why a killer API feature would be for Yodlee to pre-convert the form to HTML, and add it as a string within the JSON response. Then you could simply do something like <form><%= @response.componentList.toHTML %></form> and be done with it. Plug and play.

Webhooks

In the current API system, when a form is submitted back to Yodlee (e.g., when you submit an API request saying that user X has given his login credentials - username y and password z - please login to his bank account and scrape his data) you need to manually ping Yodlee every few seconds to see where the request is holding. If Yodlee replies that the bank login was successful or not, you relay that to the user. But if Yodlee replies that its scrapers are still working on the login, you then queue up another ping, and keep pinging until a response is ready.

This workflow is messy to deal with gracefully.

Many new APIs deal with these time-related problems through webhooks: When you submit the form back to Yodlee, you would supply a URL for Yodlee to ping YOU when it is ready with a response. The workflow would be so much simpler: you submit the form back to Yodlee, and then patiently wait till Yodlee pings you and tells you what the login outcome was.

Getting Started

OK, so enough with the breakdown! You can started with the new API by signing up for an account at the Developer Portal. If you wanted to try Yodlee before but were turned off by the complexity of using SOAP, now is a great time to give it a shot. Let us know your experience in the comments below.

Postscript Plug

At Firehawk Creative we've probably got more experience with stellar Rails-Yodlee integrations than anyone else on the planet. We'd love to help you get your Yodlee app off the ground. Ping us at info@firehawkcreative.com