We developed our “Pulse” Business Intelligence Software for PwC. It is now a global service, maintained and supported by BSL. Each day Pulse imports thousands of documents from many commercial content providers and website feeds, delivering personal briefings (called News Alerts) to PwC subscribers throughout the PwC organisation. We recently created a new importer for Forrester using their API.
About Forrester
Forrester is a worldwide provider of research, consultancy and technology advice. Forrester helps business and technology leaders accelerate their organisations’ growth, empowering them to put the customer at the centre of everything they do. They give business and technology leaders the confidence to introduce bold strategies, shaping and guiding how to navigate today’s unprecedented rate of change in business and technology.
Pulse contents
Pulse imports thousands of documents from hundreds of sources each day. During import, our importers parse the information from the supplier or website, transforming the contents (which may contain XML, JSON, plain text, RSS or HTML) into a standard representation, completely independent from the source format. Pulse sources include several premium commercial providers, such as LexisNexis, Factiva and Forrester. Each uses its own proprietary standards and formats, all of which we support.
Pulse briefings and “news alerts”
We collate and index all imported content in real-time and deliver hundreds of personal briefings to thousands of subscribers throughout the world, using complex queries that define the exact contents subscribers wish to see. Each user can subscribe to any number of briefings, each containing contents from many sources and titles. Pulse provides instant access to these briefings from desktop and mobile clients.
Importing Forrester articles
The method by which new articles are made available in Pulse depends on the vendor. For example, at Forrester, until recently, documents were made available within Pulse by placing files on an FTP server, which Pulse automatically imports each day. However, Forrester is removing support for FTP delivery. Therefore, earlier this year, we were asked by PwC to update Pulse to support the new Forrester API.
Proposed solution
Pulse supports multiple methods for content import, such as fetching from an FTP server, retrieving files via an API over HTTPS, reading from a folder on the file system, or manually adding content using a custom editor. Since Pulse already supports API-based document retrieval, Forrester’s request required few infrastructure changes within Pulse. All we had to do was add Forrester’s specific API to the list of APIs available to Pulse. Then it was just a matter of adjusting Pulse’s configuration to retrieve the Forrester documents using the API rather than downloading them from the FTP server.
Development work
We make several changes when adding support for a new API. Firstly, we must add the API to the Administrator interface so that Pulse can use it. In the interface, users can specify how often Pulse will call the API to retrieve new documents. Some APIs require multiple API endpoints to be executed in sequence to retrieve documents. In such a case, we must associate all these endpoints with the implementation of the specific API in Pulse.
In most cases, we need to authenticate calls to execute the endpoints. We send the authentication details via an Authentication request header, an access token, or a valid session cookie accompanying each call. If applicable, we add pagination, executing the endpoints repeatedly until we’ve retrieved all available documents.
The Forrester API
The Forrester API uses three API endpoints to retrieve documents. The first endpoint triggers a search at Forrester. The second endpoint checks the status of the query, returning answers such as “we are still searching”, “something went wrong during the search”, or “the search was successful, we have found [n] documents”. Finally, we use the third endpoint to retrieve subsets from the found documents.
The Forrester API implementation within Pulse includes logic to start a new search, repeatedly query the status until the search is complete, and then retrieve sets of matching documents until all are retrieved.
BSL expertise
Developing complex global services such as Pulse need talented, experienced developers familiar with Java, database optimisation, content parsing, and network security/ authentication. BSL works exclusively with in-house expertise, meaning we can deliver software like the Forrester Importer in just a few weeks. On schedule and within budget. If you need a skilled development partner here in the Netherlands, we’d like to help. So please give us a call, and we’ll set up an online meeting to discuss your project.
