R Email Parsing and Email Signature Extraction

If you use R and need to parse emails then this guide is for you. R is an amazing statistics package and many companies use it to do heavy data analysis. A tool like SigParser is critical for enabling data scientists to be productive when working with emails.

We’ll cover:

• Why parsing emails is hard
• R Code Example for parsing emails with SigParser

Why parsing emails is hard

Splitting Email Chains

If you want to split emails on the headers with R, that is tough. No two email clients seem to produce the same header format and over time the email clients change the way they format headers.

Signature and Contact Detail Detection

Parsing email signatures with R is also difficult. Many think they can just use a couple regex expressions and they'll be done. But the more you start working the problem the harder it becomes. Here are some of the major things you'd need to handle.

• No signature is formatted the same
• Phone numbers can have many different formats
• Need to attribute the type of phone number (Fax vs Mobile vs Work phones).
• The phone type indicator has lots of variations. For example, Mobile vs M: vs Cell vs C: and many others
• Titles can be incredibly difficult to capture without getting too much wrong information.
• Locations are tough. Very few people put full addresses. Often they'll only put the city and state but no country. Even street addresses are massively different by country.

Then there is identifying where the email signature is in the email which is really hard. We use a machine learning algorithm with lots of labeled validation emails. We've been labeling our test set for years across many organizations.

This is why we expose a simple email parsing API for use with R

All of the above is why we suggest using our email parsing service to parse emails from R. As we improve you'll automatically get any improvements without the need to redeploy any of your code. If an email client like Gmail starts using a new reply header format, we'll have a fix deployed with days and you won't have to do anything.

The SigParser Email Parsing API

The SigParser Email Parsing API is a serverless, stateless email parsing API which is easy to call from R. It can extract contacts and split emails into sections. It can find phone numbers, titles, addresses and attribute them to the correct contact. It even takes care of deduping contacts for you if the same email address appears in the email.

Stateless means we store none of the email contents. It is a processing only service. We only store some high level statistics like length of the email or how long it took to process.

Example: Parse an Email with R

library(httr)
library(jsonlite)

# Set API URL to parse a message into pieces or get the cleaned message body or forwards
url <- 'https://ipaas.sigparser.com/api/Parse/Email/Message/JSON';
# OR to parse an email signature
url <- 'https://ipaas.sigparser.com/api/Parse/Email/Contact/JSON';

# Set payload.
# It is also possible to assign a raw JSON string to body
# without the use of toJSON().
body <- toJSON(list("from_address" = "jsmith@example.com",
                    "from_name" = "John Smith",
                    "plainbody" = "This is an email.",
                    "htmlbody" = NA),
               auto_unbox = TRUE)

# Set header.
# Attention: This is not a list, but a function for POST().
# The API_KEY is the Email Parsing API Key obtained from
# https://app.sigparser.com/Account/App#/api
header <- add_headers("content-type" = "application/json",
                      "x-api-key" = "API_KEY",
                      "cache-control" = "no-cache")

# Response from POST with parsed metadata.
response_raw <- POST(url = url, body = body, header)

# Print status code.
print(response_raw$status_code)

# Fully parsed JSON response.
response_parsed <- parse_json(response_raw)

# Print emails.
response_parsed$emails

With response_parsed you have a fully parsed R object that can be navigated through just like a normal object.

Here is the structure of the JSON you’ll get back.

{
    "error": "string",
    "contacts": [
      {
        "firstName": "string",
        "lastName": "string",
        "emailAddress": "string",
        "emailAddressDomain": "string",
        "emailAddressDomainWithoutTLD": "string",
        "phoneNumber": "string",
        "mobilePhone": "string",
        "voipPhone": "string",
        "officePhone": "string",
        "fax": "string",
        "address": "string",
        "title": "string",
        "twitterUrl": "string",
        "twitterHandle": "string",
        "linkedInUrl": "string",
        "linkedInHandle": "string",
        "companyName": "string",
        "website": "string"
      }
    ],
    "isSpammyLookingEmailMessage": true,
    "isSpammyLookingSender": true,
    "isSpam": true,
    "from_LastName": "string",
    "from_FirstName": "string",
    "from_Fax": "string",
    "from_Phone": "string",
    "from_Address": "string",
    "from_Title": "string",
    "from_MobilePhone": "string",
    "from_OfficePhone": "string",
    "from_LinkedInUrl": "string",
    "from_TwitterUrl": "string",
    "from_TwitterHandle": "string",
    "from_EmailAddress": "string",
    "emails": [
      {
        "from_EmailAddress": "string",
        "from_Name": "string",
        "textBody": "string",
        "htmlLines": [
          "string"
        ],
        "date": "2019-05-05T22:27:56.124Z",
        "didParseCorrectly": true,
        "to": [
          {
            "name": "string",
            "emailAddress": "string"
          }
        ],
        "cc": [
          {
            "name": "string",
            "emailAddress": "string"
          }
        ],
        "htmlBody": "string",
        "spammyLookingEmail": true,
        "subject": "string",
        "cleanedBodyHtml": "string",
        "cleanedBodyPlain": "string"
      }
    ],
    "from_LinkedInHandle": "string",
    "duration": 0,
    "cleanedemailbody": "string",
    "cleanedemailbody_ishtml": true,
    "cleanedemailbody_plain": "string",
    "from_CompanyName": "string",
    "from_Website": "string",
    "from_EmailAddressDomain": "string",
    "from_EmailAddressDomainWithoutTLD": "string"
  }