Anant Jhingran's Musings

What's the big deal about APIs anyway?

All of us are familiar with SOA -- exposing applications, or application components -- so that new apps and new processes can be written by reusing existing functionality.  And we are all familiar with APIs, a set of (sometimes language specific bindings) that allow new apps to be written utilizing these bindings (windows APIs for example).  

However, the new world of APIs is the world of SOA brought to the end app developers -- often outside the firewalls of the organization.  And that means that many of the SOA concepts have to become accessible -- frictionless app creation is de jure.  Every enterprise has a core business -- Gamespy is in gaming platform, Netflix is in video and movie delivery, Sears is in retailing.  Attracting more transactions into the core business is *the* new growth model, or should be.  Traditional methods have attracted people to the "web front", and with hopes that that translates to more core transactions.  

However, there is a movement -- mobile apps!  The more mobile apps that tie in the enterprise's core business, the more transactions will flow through and more the monetization will happen.  However, there are two problems that need to be solved for this to happen...

1. When's the last time you have liked a mobile app that only took you to the "web front?"  Exactly.  So you want a direct connection between a mobile app and the backend systems' APIs that will fulfill those requests.  Which means that this is likely to make the IT managers groan -- the systems were designed for carefully managed class of applications, and now every Tom, Dick and Harry is going to pound on it?  Help.  Mediate, cache, rate control, block, protect these calls.
2. However gorpy the backend APIs are (and having written a few of these gorpy ones myself), they have to be made "accessible", i.e. brain dead simple to use.  Monetization will come not from charging for using the APIs but from the business that the API use generates!  Which means that RESTification is a pre-req, but it does not stop there.  The whole process of using the APIs has to become frictionless.

So there you have it -- the providers need some guarantees that popularity of their APIs will not kill them and the developers want the whole process of using a providers APIs to be frictionless.  That is what makes for a great API platform.

And that is what we at Apigee are doing!   

API and Data Analytics

We've seen this trend before -- at IBM, we talked of business automation and business optimization.  First the enteprises set an "application agenda" to get their business processes and apps standardized and consolidated.  And then, in a serious way, they got into business optimization using data and analytics.  At IBM, we called it "information agenda."  It is not just about getting visibility into processes and applications, or reports (which is important, but nowadays is more or less the cost of doing business). Enterprises that master the Information Agenda are getting a handle on information and leveraging it to optimize the business using analytics.

I am seeing the same thing around API's.  {I am assuming that most of you grock the concept that API's connect backend and enterprise systems to (primarily) front-end and mobile developers, and need to be managed and tracked as a valuable asset.}  The current spend is around the transactions that flow through the API's (*, see below) . And of course key clients (**, see below) want visibility into APIs - who is calling what, when is it happening, what error reports etc.  I call this IT and API metrics.

However, as I talk to customers, it is also clear that they want analytics at the business level -- what products are being bought, what money is being spent on games, what is being tagged most frequently?  I call this business metrics.

So the API team, the lines of businesses and the developers of the apps using APIs want analytics on IT and business metrics.  So far, that is natural.  After all, if something is being used, it must be tracked and monitored.

However, the real game changer is when the metrics are used to do prediction and to take action.  Analysis of an IT level metric might automatically lead to a policy change for caching during peak hours/months. Analysis of a business level metric might allow a game platform provider to make the games more sticky by predicting abandonment and automatic triggers of some offers.  APIs are going to be central to any future enterprise.  And predictive analytics on APIs, in my mind, is going to be central to those enterprises (hopefully 100%) that are trying to optimize, not just automate, their business.

There are two other trends that need highlighting.  One, of course, is cloud.  Can such analytics be delivered in a simple but gorgeous way through the cloud?  And second is data, with so much data out there that is external to an enterprise, can that be brought to bear to deliver some wonderful added value to analytics?

So this is what I will be helping Apigee do.  If you are an Apigee customer, then I will be spending time with you.  If you are not, then I hope that our data and analytics technology will convince you to give Apigee a try.

But most importantly, I will always engage in an open and honest dialog with the blogosphere so that we can collectively "rise all boats".

* I am so used to writing API's (note the apostrophe), but at apigee they will convert me into writing APIs, I am not sure whether I will ever master that :) and maybe I will convert the rest of my colleagues to my way!)

** oops, at apigee, it is customers, so another thing I will have to retrain myself on

A new chapter in my (work) life

As a few of you know, I left IBM last week to join Apigee.  IBM is an absolutely wonderful place with the best people and the best technology.  I have been fortunate to work there for 21 years, and have had the privilege of working with and for the best and the brightest.  I had friends as bosses, and bosses who became friends.  I had colleagues who respected each other.  And the best part of being an IBMer is that the customers (even when criticizing some feature or capabilities of our products) had the admiration fo what IBM brings to the table.  In some small ways, I have given back to IBM by helping it grow a few businesses, but what I have gotten far exceeds what I given back.  Thank you IBM!

But there comes a stage in life when one examines what's next and sometimes concludes that there is something else to prove.  I reached that point this summer and decided that I had to be in a startup that offered me a very different kind of challenge.  IBM succeeds when innovation leverages the IBM engine and when it does the success is BIG.  With the help of my colleagues, there are several technologies that we were able to bring success to using that engine.  The latest efforts in IBM's Big Data capabilities is one of the latest such example.  However, having done that, I wanted to prove to myself that I could leverage a very different engine, hence a startup.

Now my choice of the startup hinged on a few criteria I had set for myself.  

One, it should already have paying customers, a growing revenue and a deep pipeline of prospects.  This was important since the skills required to bootstrap a startup in my mind are very different than the one to grow it, and I did not believe that I had the former skills or interest at this time.  

Second, I am a firm believer that data and analytics are the defining technologies for the next decade, and if pressed hard, I will grudgingly acknowledge that mobile is a close second.  So I wanted a startup, and a role in that startup,  that would have a good intersection of analytics and mobile.  

Finally, and especially in a startup, one must work with folks who you respect and like and provide complementary skills. 

All of this led me to Apigee.  My day 1 was Friday.  I will write about my transition (the shock and the awe and the gradual acclimitization I hope!) but most importantly, I will write about the technologies I will help create.  For those of you who have followed me for my information management and cloud thoughts, you all might be a little disappointed in my future postings because I will talk about some slightly different things, but data and analytics and cloud would be undercurrents that would run through it.  So un -blogroll me if needed!  I would not mind :)

I want to end with how my last week in IBM transpired.  As I was going through this decision, my immediate boss, several senior vice presidents, and many other bosses I have worked for, talked to me about the pros and cons of what I wanted to do.  In the end, they were disappointed (deeply perhaps), but as I talked to them, it became clear to me that their treating of me as a colleague and as a human being is what makes IBM a wonderful place.  And then I talked with my colleagues and that was so hard. We have had 21 wonderful years.  In the end, one of them captured it beautifully, "Anant, we are sorry to see you go and wish you the best of luck.  We know that you will always love IBM and IBMers, and that you also know that IBM is more than one person and will be even stronger in the future, so go realizing that you have helped in some small way towards IBM's success."

Top 10 reasons to #knowsql, not just #nosql!!

First and foremost, see that I am saying "#knowsql, not just #nosql".  I am not saying "#knowsql, not #nosql." Second, I am not putting #hadoop in the #nosql category -- it is not a #nosql engine, it is "do what you want with it" engine.  Pig's and Hive's add data models to it, but that is a separate topic. 

10. SQL is the language that talks about "select project joins" but uses the word SELECT for project!

9. SQL databases do not ignore ACID properties unlike most of #nosql databases.  Have you tried to reason about "eventual" consistency a la Cassandra?

8. You have choice of implementation, not lock-ins to one technology, one implementation.

7. Your system does what you want and you do not have to tell how you want it done (Declarative vs. Procedural?)

6. Billions of R&D dollars are being spent on improving SQL databases per year.

5. 95% of the hottest database "system technologies" (like hardware acceleration, GPU exploitation, multi-core exploitation) focus on SQL, not #nosql.  {Obvious Anant perspective, can be questioned as to what I mean by "hottest".}  So SQL will get leaps and bounds of performance improvement over the next few years.

4. 95% of cloud projects are on SQL, not #nosql. {Anant perspective, but verifiable.  Working with @sogrady to see what numbers we can come up with.}

3. SQL databases can now handle RDF, XML, key value stores without sacrificing ACID.

2. Most real data persistence needs are sub ~200 TB, easily a sweet spot for SQL databases.

1. Every #nosql in the end either already has, or will, put up an SQL-like layer.  Look no further than GQL (google).

The education part of my keynote...

People who were there at the hadoop summit recall the three parts to my keynote -- entertainment (watson), education (industry use cases we are seeing) and philosophy.  I have talked philosophy before, and also entertainment.  So let me focus on education (which some not to be named folks like JF called "boring" (JF, you know who you are :) but others like Joe Reger Jr. specifically asked for..

So here it goes.  My view is that we are seeing enterprise use cases around the three V's -- volume, variety and velocity, with variety being the most important, and volume not that much so (hence my disdain for the term "Big Data", but that ship has sailed so no sense in being grumpy about it, unlike "data scientist" which is still not an industry term, so maybe some grumpiness from me and others will reduce its chances of becoming an industry term :)

and along the three dimensions, the ~100 engagements that IBM is engaged in are exhibiting various characteristics (edit based on comments received: this is not exhaustive, other industries such as telco etc are also using this extensively):

I will post later, if you all are interested, the top use cases that elaborate on column 2 above..

Why I do not like the term "data scientist"

Oh my god, am I a heretic among the believers?  Let me explain why:

1. Geekification might enhance the importance in the engineering culture of Silcon Valley, but where the technologists need to find jobs (i.e. enterprises), it only adds to the perception that this is too hard, why bother hiring such geeks?
2. It will also distract the community from what I think is a fundamental commandment, "Thou shall make the technologies more consumable."  If we continue to foster the concept that this set of capabilities are built by scientists, for scientists, we will lose sight of the fact that they should be built by scientists and engineers and interface folks and domain experts and ...  for scientists and engineers and interface folks and domain experts and business people.
3. It is an un-necessary rebranding of well know disciplines.  I do not buy the fact that magically some new skills have to appear.  Yes sure, hadoop is different.  But skilled people grock new stuff using their existing repertoire of the basic skill of grocking large scale systems, analytics etc. 

Am I too grumpy today? :)

My Hadoop Summit Keynote

It was amazing, and tremendously exciting, to see 1600 people register and attend the 4th Hadoop Summit.  Eric14's new company, Horton Works, was clearly the pre-conference buzz, and made my job of following up that much more difficult :)  I made three points:

1. "Entertainment": Watson, how it works, and role of Big Data -- essentially, the points I have made in this post.

2. "Education: What enterprise use cases we are seeing -- I have talked about it in the past, and will do more in a later post.

3. "Philosophy": Building upon my earlier post.

So let me elaborate on the Philosophy.  My essential points are that the "birthers" (where hadoop has been born) and "adopters" (where hadoop will be used in enterprises) have a strong intersection today, modulo some extras on both sides...

However, at t = 3 years from now, we can either go separate ways because of different demands...

or come together:

The choice is ours, as a community, we need to decide which way we want to go.  IBM and I are strong advocates of the following view:

I will follow up on some other points I made, especially in the education part, but the feedback I have received at the conference is very positive with the message I have been trying to convey.

Great community, let's keep it together!

Enterprise vs Online Hadoop Use Cases

As part of my keynote preparation at Hadoop Summit (organized by Yahoo) next week, I started to think through what I will say...  I could advertise IBM, but I know how much people love "commercials" :)  So I am going to go philosophical. 

Hadoop was born in the web world, on-line properties.  I call this the "T" use-cases, where the technology was born.  Enterprise use cases, where the hadoop community will make money, is different. I call these the "M" use cases, for money.  So the question is, what is the intersection and how large it is?  And which way will it go, will the intersection increase or decrease?

Is it this at t = 0 (now)?

or is it this?

I will argue that based on what I have seen, the intersection is extremely high...  Hence the latter.

However, I will also argue that if it drifts apart because the constituencies are different, then it is extremely dangerous.  And I will argue that we as a community need to do a few things to ensure that the drift apart does not happen.

See you at the summit..

 

Lessons from Watson (Jeopardy Computer) in the Big Data space

I have had a chance to speak about Watson in several fora -- the latest being the Economist Forum on Big Data, where I was the closing act.  I am incredibly proud of what IBMers have done.  And I think what they have done shows a lot of implications as we think through the data deluge...

I showed the inner workings of Watson, and made the following points:

1. Interface matters -- understanding natural language and responding in speech sometimes make a big difference, and Watson definitely blazed a new path there.  Google's innovation, in my mind, has been as much in interface as in the algorithms behind the interface.  As we think through Big Data, let's not forget the interface into the systems -- for asking questions, for receiving analysis -- q&a might not be the only form, but PIG's, M/R programs etc. is definitely not it.
2. Machines, big machines, big data machines, are brawn, maybe brainy-brawn, not brains.  Watson uses 10,000's of watts, the computer between the ears uses 20.  Sheer brawn might take mega watts, IBM's researchers were able to bring it down to 10,000's of watts, so I call it brainy brawn.  But brainy brawn is not a replacement of brain.  I like to use Intelligence Amplification (from William Ross Ashby) as an analogy, thought others such as Patty Maes use other terms like Intelligence Augmentation.
3. Size is not the only thing that matters :)  Watson deals with 200 million pages, wow.  That small?  At 10K a page, we are talking 2TB of information.  I routinely work with clients that have database sizes 100x of that.  That is why I think that we have done our industry a dis-service by calling it "Big Data."  That tends to get macho on size.  I like to think of it is as Big Analytics, and that is why our products in this space are called BigInsights -- deep insight matters, not size.
4. The world is not perfect.  Data tells us different things, often conflicting.  We need to weigh alternative hypotheses through some evidence based reasoning.  That is what Watson excels in -- it generates hypotheses (candidate answers), and then weighs different evidences ("popularity", "earlier categories"...) to determine the most likely answer.  This is the transformative aspect of Watson -- do not fixate on the oneright answer, focus on a range, and reason through them.
5. Hadoop matters -- the backend (the "boiler room" as I called in the room) is used to crunch through the documents to prep for the interactive Jeopardy matches.  There is no other system flexible enough to allow for the flexible knowledge extraction that we need.

I had many other interesting observations, which I will make over time.  However, my hats off to the Economist, and to the hosts, including Vijay Vaitheeswaran, for having such a well executed event.

I will be keynoting at the Yahoo Hadoop summit next week, and will make some of the above points, but will also talk a lot about what we are seeing with hadoop usage in the enterprises.  Hope to see you there.

ps: edited to correct some spellings.  i am not talking of household machines (nee, "braun"), i am talking of muscles (nee, "brawn").  

ROI on Analytics

A few months back, I was on a panel organized by SDForum, the panelists (including me) were asked a question: "have we reached the limits on what can be gained by analytics?" (I am paraphrasing, but it was approximately this.)  I was somewhat surprised (I would have been less so if someone had asked about process reengineering, automation etc.) that people even can imagine that we have reached the end of analytics.  I answered then that all the "smarter planet" activities that we in IBM talk about are great examples of using analytics to make people's lives better -- better water, better transportation, better government -- and I cannot forsee any limits to us improving people's lives using analytics.

But that got me thinking -- how does one quantify the benefits of analytics.  Sure, anecdotes abound.  But we are data driven.  So where is the data on the benefits of analytics.

Then I came across this paper.  (I got the link from the "Big" Big Data report by McKinsey.)  Through some analysis, and some self reporting by companies, they have concluded that DDD (data driven decision making) adopting companies have 5 - 6% better top line and bottom-line.  So for a $10B company, that means $500M improvement.  Think about it.  How much would that company spend to get that?  Just take a look at the cost of storage (I am being very approximate here), on a cloud, we are looking at around $1M/yr for a petabyte of data.  Throw in computation, 10x it, we are looking at $10M/yr.  $500M on $10M return. 

That's it.  Obviously, Big Data pays off!  Do not "let's throw away because it is too expensive".  Instead "store because the cost of analytics + storage more than pays for itself."