A fundamental challenge exists in today’s NLP-based text analytics solutions; the volume of dynamic data is exploding and there are not enough subject matter experts available to map out the relationships between concepts.

Many aspects of advanced text mining capabilities rely on the existence or creation of ontologies, semantic rules, and templates to:

• Resolve ambiguity to determine the right meaning
• Determine synonym relationships
• Identify and categorize entities, actions and events

Creating deep ontologies and semantic rules using traditional text analytic approaches (e.g. CRM, genomics, security intelligence) is a manual process. At times this is the preferred approach, because the relationships need to be mapped for a specific policy  (e.g. practice systems for hospitals or answers to a support question).

The challenge for enterprises today is the majority of their data is unstructured and dynamically changing, making it difficult to surface insights and take action upon it in real-time.  The cost of manually building an ontology and the lack of skilled subject matter experts make it unlikely that the enterprise ever adequately resources this activity.   The rapid growth of dynamic and real-time data feeds across multiple internal and external sources, vital to business intelligence, exacerbates the difficulty of extracting true meaning from unstructured text data.

Atigeo’s xPatterns platform automates the creation and dynamic maintenance of semantic ontologies. xPatterns uses a neural network to model associations between concepts.  It represents “IsAssociatedWith” relationships for domains, derived simply from reading and reviewing large bodies of unstructured text information about the domain that can be streamed to the system in real-time. The technology can be leveraged to determine indirect semantic relationships between queried concepts, and to facilitate understanding of the relevance of a specific document to a specific concept. It can dynamically extract key concepts from any viewed document and surface documents that are more closely related to it.  This allows users to traverse and assimilate bodies of content in ways similar to how the human mind organizes information.  This can allow a user to quickly learn about a new topic and make fact based decisions.

Moreover, it is also possible semantically index a user’s past queries and documents viewed to maintain unstructured semantic profile that can help personalize views, refine relevance or even proactively notify the user when related content gets added to the system.

xPatterns is a cloud based platform that can be easily integrated with existing enterprise applications and collect data from internal and external structured and unstructured sources.   It can be used to augment existing business intelligence and text analytics solutions.  The platform has been used to enhance search and discovery and speed assimilation of complex information.  It can monitor trends in social media, match supplier capabilities to open RFPs, translate physician notes to ICD 10 codes and spot patterns of non-compliance or fraud, waste and abuse.

A picture or a demo is worth more than millions of random words.  Go http://www.atigeo.com/videos/   today so see some of the ways we are connecting related concepts and delivering insights on large dynamic unstructured data sets.

Did you know that more than 20 million people “Like” Starbucks on Facebook? Or that Starbucks increasingly leans on its social media channels for marketing campaigns, outreach efforts and even product development? Ideas pour in to the Coffee Experience conglomerate from its social media fans all the time, including simple practical ones like putting handbag hooks in women’s bathrooms. So ladies, the next time you have a more pleasant, sanitary experience at Starbucks, you’ll know who to thank…that’s right, Facebook.

From the get-go, the main emergent theme of the recent Gartner Customer 360 Summit was Social CRM. It’s not uncommon for companies to now have huge multitudes (in the millions, even) of Friends on Facebook or Followers on Twitter. Those kinds of numbers change the game in terms of the definition, significance, and leverage points of today’s Customer Relationship Management. The boundary between company and customer is becoming more permeable, and the immediacy of access and response is becoming near instantaneous. What stirs the stew even more is that the number of touch points or data channels between company and customer is exploding—as many as 60 according to one company—with the concomitant need to build a unified profile of a customer.

As with other internet data, a great majority of the data exchange in social CRM is in the form of unstructured text—be it Facebook updates, tweets, or customer support logs (emails or transcriptions of customer support audio interactions). What was evident at the 360 Summit was that just about every company presentation—whether from vendors or consumers of technology solutions—touched on analytics as a main component of their social CRM systems. Companies need to have an understanding (in real-time) of their customer’s complaints/requests/wishes based on an ocean of unstructured and structured data and a multiplicity of channels. As one analyst put it, “it’s a good time to be in the business of data analytics.”

Today’s analytics solutions are both much improved and have much room for improvement, and by all accounts, no single vendor solution or solution path has emerged as a clear market winner or technology choice. The conference chair, Gartner CRM analyst Gene Alvarez, addressed this in his opening “Road to 2015” remarks. He pointed out that the key market drivers are: Social, Mobile, Explosion of Data and Explosion of Channels; emphasizing that today’s CRM leaders have access to lots of data but no way to put it all together.

There is no doubt that the analytics and semantic markets are ripe with opportunity. And revenue! The annual online ad market alone is estimated at $140 billion, and single-digit upticks in click rates are enormously value-laden. What’s exciting is that Atigeo’s Intelligence Platform, xPatterns, built on a foundation of knowledge auto-discovery (unencumbered by the need for hierarchical taxonomies and OWL ontologies) is well-poistioned to bring together all the needed elements of an encompassing solution: Concepts, Content, Context, and People.

That’s not to trivialize the complexity or magnitude of the problem: driving data to analysis to action is a multifaceted problem, especially for large enterprises, and technology solutions are only part of the mix in positioning a company for a successful Customer 360-type engagement. But what’s undeniable is that the Voice of the Customer is increasingly loud (if not always clear–and clarity is precisely the point here). And companies are and will continue to invest heavily in setting up their social CRM strategy to be responsive to that voice. It all adds up to enormous opportunity for Atigeo.

It was the 1980s.  Every teenager was saying it or singing it: “I want my MTV.” Cable TV broke the broadcast model and opened the door for niche content providers such as MTV, ESPN, and more.  Today, the internet is breaking the broadcast and cable model by providing even more choice in the form of Netflix, YouTube, iTunes, and more.  If you listen closely you can almost hear the teens of today singing “I want my ME TV.”

I recently attended a Telco 2.0 Executive Brainstorm and Developer Forum in Palo Alto, CA. It was clear from the discussions that the market for Smart TV is ripe for disruptive technologies enabling a new era of relevance and personalization – the pillars of “ME TV.” Key players are falling into line around a common three screen vision, where content is connected, personalized, and available across devices. There are three experiences of this new media environment that emerged from our discussions. First is the traditional experience of watching content on the TV, American Idol or Rattle and Hum for example. The second is a social experience, such as tweeting about the show, posting Facebook updates, or interacting with a customized app for the show. The social aspect could enable viewers to interact with each other via video and have a shared media experience. The third experience involves consumption of related metadata or content, such as relevant wiki articles, ecommerce, or news stories. Metadata content can also include maps or travel information related to a particular scene in a show or movie, which provides the opportunity for deeper interaction, context, and connection.

Of the three experiences (consuming traditional TV content, social connection, and interaction with metadata) the latter two require a relevance or recommendation engine. One of the challenges with existing relevance and recommendation engines is that current capabilities only present similar TV shows or movies to viewers based on, “people who liked this also liked that.” To date, solutions have not been generic enough to be able to arrange and create relevant social experiences and metadata experiences based on the TV content being consumed across devices.

For me, a big “aha” moment that surfaced from the executive brainstorming session at Telco 2.0 is that the market is beginning to mature and our xPatterns solution can play a pivotal role in the smart TV ecosystem. xPatterns can drive relevance based on TV content and what’s changing in the social or metadata landscape and how each of these elements impacts the relevance of the rest.  xPatterns can put the ME in the “I want my ME TV” anthem of the next generation.

In reflecting with internal and external colleagues in the past few weeks and hearing from Gartner analysts, I’ve become aware of the emerging area of Pattern-Based Strategy. Pattern-Based Strategy is the wave of organizations transitioning to discovery of and adaptation to trends in their business and industry.

I wanted to describe xPatterns in the context of this trend.  One of the most mature technologies leveraged by organizations driving Pattern-Based Strategy is Predictive Analytics. Predictive Analytics is usually delivered using a suite of tools run by a team of statistical analysts on bespoke problems in the BI team.  Predictive Analytics’ next wave is the downward penetration of Predictive Analytics practices to businesses without statistical analysts on staff, or as an in-house BI function.  This penetration is being precipitated by the emergence of Predictive Modeling Solutions, out-of-the-box technologies that make it possible for marketing and business personnel to build predictive models to analyze and take action upon their structured enterprise data.

Analogously, many enterprises are becoming aware of the need to be able to act on silos of increasingly dynamic data that are unstructured.  Today, the most advanced enterprises create and maintain semantic ontologies to allow them to identify and act on semantic concepts expressed in their unstructured data, we call this space Content Analytics.  This is actually a data structuring process, which is hard to scale, and manual to perform.  For example, who would have thought of the association between “winning” and “Charlie Sheen” just a few weeks ago?  xPatterns provides the next generation of unstructured data processing solution – Learning Semantic Search. xPatterns dynamically creates and adapts models of semantic relevance, and makes semantic connections actionable. Such connections for example, can be between the persona, location, and context of an individual, and relevant news or offers, or public discussion of a brand or product.

Today’s model for personalization involves tracking, collecting, and exchanging data about YOU.  While some of this data is generalized based on location, demographics, and behavior other components can include personally identifiable information or cohorts of semi-personal data. Where and how YOUR information is snagged, stored, shared and even sold is largely outside of your control. But we see a very different future of privacy taking hold, where you don’t have to retract from the online world to maintain your privacy or alter your behavior to avoid intrusive tracking.

Imagine if YOUR information was transformed into a truly private and portable persona – and put YOU in charge.

This future of personalization will enable a new wave of innovation for online experiences and services without requiring federal regulation, that many fear will stifle growth and commerce. You could receive personalized content and experiences based on your preferences, without ANY specific components actually being revealed to an outside party. With privacy based approaches to user data, trusted applications acting as a “concierge” will host an understanding of you, derived from your persona, which will bring together: preferences, behavior, social graph, location info, your roles/intents, and much more.

For example, a user might want Hertz and the Marriott to get access to their identity, location, and preferences when planning a trip, but not when they are browsing the news or going to the park with their kids. Or they may want Facebook friends to be able to get their music preferences so they can be invited to concerts of their favorite artist. The “concierge” will act on your behalf without revealing the contents of your persona. It WILL enable 3^rd party applications to facilitate an understanding of what ad, offer, or item of content may be most relevant, but NEVER specifically why.

The technology allows a unique approach to privacy; to assess a particular piece of content, the attributes within the persona do not need to be shared with the content provider. The content description and any semantic metadata are simply indexed with xPatterns and under the permission and control of the owner, only a “score” of the content affinity to the persona is presented to enabled content providers. In Atigeo’s version of the persona, the contents are expressed in unstructured form as lists of lists of concepts in combination with semantically-expressed spatial, temporal, and behavioral context. When this data is applied to xPatterns indexes of content semantically appropriate actions, items, and offers can be presented to the user at a meaningful time by the content provider.

This model requires a rich understanding of the relationships between different themes and concepts that can in part be learned from the actions that people take. However, equally important is the learning derived from how things are described and talked about relative to one another (which really conveys what something means). With the use of hierarchy-free ontologies, Atigeo’s xPatterns enables systems, in real time, to determine semantic relationships between concepts – simply from reading and reviewing large bodies of unstructured text information about the domain.

Olly Downs
Chief Scientist, Atigeo

It’s a battle cry heard around the world in enterprises and start-ups.  Hundreds, if not thousands of these initiatives start every day.  From the social networking start-up to the world’s largest enterprises…They all start by getting a group of people to decide on what data to put on a profile.  Sounds easy right? Try it. Get a group, much less departments within an enterprise or competing corporations, to agree on what data is important to collect.  Next, hand out a schema to individual or enterprise software providers, and convince them to change their system to query your definition of the consumer.  It’s hard!

There is hope.  The semantic web promises to make all of this easier; where the meaning (semantics) of information and services on the web is defined, making it possible for the web to “understand” and satisfy the requests of people and machines to use web content.  The fuel of the semantic web is ontologies.  It’s possible to envision a semantic web where ontologies are used to develop a deep understanding of a customer with a standard set of tools to access and manipulate.

Sounds great.  But the semantic web isn’t here yet.  Do we need to wait until the ontology and schema are standardized by each industry?

We have a proposal to realize the vision of the semantic web now, while we wait for ontologies to emerge and gain mass adoption.  We call it Schema-Agnostic Action on a Profile.

Imagine if you could launch a unified view of the customer initiative and didn’t have to get a group of people or departments to agree on what data to collect or what format that data should adhere to.  Now imagine third party applications could query this profile without understanding the structure and content.  This transformational approach would allow a more unified view of the customer, initiatives to take less time/ require less effort, and yield greater results.

Schema-Agnostic Action on a Profile

We want to share what we think is one of the most unique concepts behind the technology at Atigeo – the notion that it is possible to act on the attributes of an entity without knowledge of the schema with which that entity is represented.

To share or cooperatively act on the profile of an entity (business, person, object, thing) today requires agreement between the parties interacting in terms of the profile or content attributes/metadata. Take for example MPEG7/21 efforts around media metadata.

Most commonly, systems interact through query-style interfaces where results are returned based on attributes or filters on attributes that are matched.  Among the most sophisticated systems today, some are able to step beyond attribute matching to exploit ontologies, graphically-expressed relationships between attributes or attribute sets.  These allow relationships that are understood (“my sister is my son’s aunt” and “the Nexus One is a Google Android-based cellular phone”).

Not only do the underlying semantics of some domains evolve rapidly, but often “local terminologies” develop for the same concepts that subsequently need to be reconciled. (BTW – the latter turns out to be a very interesting graph-theoretic problem. See the work of the SHER team at IBM Research).

There are many domains for which there isn’t a standard data format. Additionally, there are many more domains for which, until the Semantic Web becomes pervasive, ontologies don’t yet exist.

Our technology is a solution to automatic hierarchy-free ontology discovery. As a corollary, this allows us the ability to determine affinity of an unstructured profile to content in the absence of “structured/direct match.” Uniquely, the ontology discovery process allows learning, refinement and expansion through user interaction, and through real-time tracking of content generated in the domain of question.

Ok – Got It.  But So What?

Imagine an enterprise launches yet another unified view of the customer initiative.  Now instead of teams of people battling it out over what attribute should go on a profile, we just set up a profile and start attaching data to it. Now imagine handing out a set of APIs to enterprise IT staff and 3rd party application developers that allow them to act on the unified view of the consumer without providing a schema in a domain for which there isn’t an ontology.  Possible?  With a more humanistic query language that understands the meaning of the data, the process, and the persona, applications can be built quickly without being constrained by the definition of the data.

In our next blog posts we will explore the proposed framework for a transformational query language that breaks the schema and ontology shackles off of the developer and the data they are acting on. Additionally, we will illustrate how this technology allows a unique approach to privacy. Stay tuned!

There are two technology drivers that have emerged over the past decade that make the prospects of gleaning actionable intelligence from big data much more realistic.  The first technology driver is semantics.  Even though there is a rapidly expanding sea of data surrounding consumer activity, enterprises currently have a very shallow understanding of their customers, based primarily on speculation and assumptions. To make experiences intuitive and relevant for customers, we need technologies that allow understanding to be derived from explicit and implicit expressions of user preference, role, and context.

As humans, we most naturally express ourselves through language which is understood by humans, but it is a real challenge to codify that understanding for computers. This is where Semantic Technologies come into play. Semantic Technologies are automating the understanding of information expressed in language so that machines and humans can effectively reason with data.

For example, technologies like eXaudios can analyze call transcripts to determine if the caller is happy or frustrated with the service they are receiving.  OpenCalais can identify entities such as movies, songs, people, and places in the text on web pages.   By bridging between related concepts expressed in language, it is possible to make connections between understanding of users and the understanding of content. Moving well beyond keyword matching, people can be connected with content at the “right time,” based on related meaning.

In the above example we use relfinder to explore semantic relationships encoded in dBpedia (the semantic ontology built on top of Wikipedia) specific to understanding the association between U2 and Daniel Lanois.

Another example, illustrated by Gnosis by ClearForest, identifies a number of different semantic elements in web pages as you browse them (such as people, events, business entities, and so on).  With technologies like these, we can understand content as it is generated or viewed, the relationships between actions in the physical and virtual worlds, and how they relate to user interests, roles, and context.

The second technology driver is the advent of web-scale data analytics – as powered by;

• Advances in computational horsepower, large-scale, high performance data storage, fast local and external networking bandwidth, and extensive wired and wireless connectivity.  It’s now possible to buy numerical computing power for $1,000 per TeraFlop, and have that running under your desk  – I know because we’re doing it at Atigeo!
• Advances in data mining, knowledge discovery and machine learning algorithms, designed specifically to work in massively parallelized fashion are allowing us to derive insight from data like never before.  Previously abandoned lines of research have been re-opened as the newly acquired computing power has finally brought them to life.

What we now see are high volume streams of real-time data being processed in-flight, datasets of unprecedented size being efficiently mined for nuggets of information, and novel nonlinear algorithms deriving insights from data with more depth and breadth than ever before.  This is all enabling real time insights into vast amounts of data. Billions of Twitter, Facebook, and blog updates are being mined to reveal outbreak of disease, political revolutions, Hollywood gossip, and even what night spots are hot.
For example, Sense Networks is deriving real time location insights by leveraging location data that is crowd-sourced from cellular phone network hand-offs . This is helping businesses identify the best location for their new store, public agencies to identify commuting patterns and evolving socio-demographic boundaries in a metro area, and allowing kids to figure-out where the most happening scene is at right now!

During the 2010 Super Bowl, Twitter was able to produce the graph below which represents a minute-by-minute reflection of people’s thoughts and emotions during the game.  The correlation of important aspects of both the game and advertising with peaks in related tweets is astounding.

Even the utilities are getting in on the act. Epcor shared the water consumption levels (shown below) for Edmonton during the 2010 Vancouver Olympics Gold Medal hockey game compared with the previous days.

Web-scale analytics are also allowing new markets to emerge. For example, Bing Travel analyzes airline ticket pricing, availability, and demand in real-time to help travelers to determine when to buy and who to buy from.

Additionally, new online advertising marketplaces are emerging where it is possible to bid on an online advertising impression in real-time (with the whole process taking <100ms while an end-user’s web-page is loading).  The value of the impression is being determined based upon guesses and knowledge of whom the user might be, association of the real-time online behavior with personality, and further associations of personality with consumption of goods – making “right-time” offers a reality.

In summary, semantic technologies are opening up access to even greater volumes of both consumer and enterprise data, by providing a mechanism through which to interpret and take action on unstructured data sources (that were previously untapped).  This explosion of interpretable data is being met by a revolution in cost-effective high-powered computing, and the rapid evolution of algorithms capable of dynamically aggregating and reasoning with this new explosion of structured and unstructured data.

In Atigeo’s next post – look for David Boardman discussing how our use of these technologies is making the proliferation of user data actionable, yet uniquely under the private control of the user.