Posts Tagged ‘data’

There has always been a vibrant ecosystem around financial data. Financial institutions, such as hedge funds and investment banks, pay thousands of dollars for quantitative tabular data (financial data in spreadsheets). But now, the web has provided a mechanism to distribute and publish large amounts of data, but much of this data is raw (meaning, it’s not built into a spreadsheet format) and hard to find in a Google search. An finding the data, and then putting the data into a format that is easy to digest can be a laborious task. Y Combinator’s Data Marketplace is hoping to change this by providing a platform where financial professionals can request data sets and then data aggregators/consultants can then find and format the appropriate data.

Founded by two former analysts at investment banks, Data Marketplace is essentially the middleman in helping financial organizations find quality data on the web. Users can submit requests to Data Marketplace, and the site will send those requests to its database of 200,000 data aggregators, programmers, and consultants who specialize in finding financial data and essentially transferring it into a readable format.

Providers then post data resources to Data Marketplace, provide descriptive metadata, and also set a price. The stored metadata is used to help consumers find relevant data through traditional search engines and when browsing Data Marketplace. Data can also be posted on the site without a request, that users can search for. For example, here’s a data set of a complete list of Wal-Mart Store Locations, which is priced at $30.

Prices range for data, and can be anywhere from $5 to several thousand dollars. Data Marketplace co-founder Matt Hodam tells me he spent $10,000 in on year on data at one of the financial organizations he worked for. Data Marketplace takes a 14% cut of each transaction on the site, from the provider. Data Marketplace handles all of the payment processing and allows users to directly purchase and download resources in an accessible format online.

Hodam says that current models for selling and distributing data online are inefficient and expensive for financial organizations. Users only pay for what they need as opposed to plans or buying bundles of information. And providers don’t have many platforms where they can sell their data in a marketplace.

Data Marketplace is similar in some ways to Factual, which is a Wikipedia-like site for open data, and InfoChimps, which takes a more collaborative approach to open data.

Editor’s note: Big Data has been around for a long time between credit card transactions, phone call records and financial markets. Companies like AT&T, Visa, Bank of America, Ebay, Google, Amazon and more have massive databases they mine for competitive advantage. But lately, Big Data is finding its way to the smallest startups. The Web and cloud computing brings Big Data everywhere. But what exactly is pushing Big Data forward?

To answer that we brought in an expert, Bradford Cross. Bradford is the Co-Founder and Head of Research at FlightCaster. FlightCaster is backed by Y Combinator, Tandem Entrepreneurs and Sherpalo Ventures. The company analyzes large data sets to predict flight delays. Bradford is chair of the Dealing with Big Data track at Cloud Connect this week.

We are in a Renaissance for computer science, engineering, and learning from data right now. The scale of data and computations is an important issue, but the data age is less about the raw size of your data, and more about the cool stuff you can do with it. Now that there is so much data, it is time to unlock its value. Really neat things are happening already—like the way the people of the world can educate themselves on all manner of issues and topics, or the way data and computing serves as leverage in other scientific and technical endeavors. There will be lots of amazing stuff on the web, but innovation will come in other domains as well.

The recent big data trend is about the democratization of large data more than its growth. In articles like the Economist’s recent piece on the data deluge, we hear about big data everywhere. We hear about what big data and the cloud mean for the enterprise, but they have had big data for a long time. eBay manages petabytes in its Teradata and Greenplum data warehouses. Sophisticated startups extracting value from big data is also nothing new—it has been happening at least since the days of Yahoo! and Google, and they have done it without the data warehousing folks.

Now focused early stage startups can get up and running faster than ever. Less technical analysts at companies like Facebook and Twitter can access massive amounts of data easily. Even individuals can undertake cool projects with big data, such as Pete Skomoroch of Data Wrangling did with trending topics for Wikipedia.

Why Now?

We do not have to build all our own hardware and software infrastructure anymore.

Pioneers such as Amazon have given us the cloud, where we have the capability to run very large server clusters at a low startup cost. Pioneers like Google have paved the way for open source projects like Hadoop and HBase, that are backed by big company contributors like Facebook.

The combination has paved the way for a new class of data driven startup like Aardvark (just acquired by Google) and Factual, it has reduced both cost and time to market for these startups, as we showed with Flightcaster. And, it has allowed startups that were not necessarily data driven to become more analytical as they evolved, such as Facebook, LinkedIn, Twitter, and many others.

So we have big data, the cloud, and open source facilitating new data-driven startups. I like to break this trend down from the technical perspective into three chunks; storing data, processing data, and learning from data. I define “learning from data” to mean data mining, AI, machine learning, statistics, and so on.

Supersize my data. Oh wait, I’ll just have a Medium.

The first time I heard the “Medium Data” idea was from Christophe Bisciglia and Todd Lipcon at Cloudera. I think the concept is great. Companies do not have to be at Google scale to have data issues. Scalability issues occur with less than a terabyte of data. If a company works with relational databases and SQL, they can drown in complex data transformations and calculations that do not fit naturally into sequences of set operations. In that sense, the “big data” mantra is misguided at times. For instance, a GigaOm article about big data in the cloud states:

What is becoming increasingly clear is that Big Data is the future of IT. To that end, tackling Big Data will determine the winners and losers in the next wave of cloud computing innovation.

The big issue is not that everyone will suddenly operate at petabyte scale; a lot of folks do not have that much data.

The more important topics are the specifics of the storage and processing infrastructure and what approaches best suit each problem. How much data do you have and what are you trying to do with it? Do you need to do offline batch processing of huge amounts of data to compute statistics? Do you need all your data available online to back queries from a web application or a service API?

Once your data and its processing are large enough to require distributing the data and the work among machines across network boundaries, things get a lot harder. You have to deal with distributed computing and make tradeoffs like a real computer scientist.

Big Data & The Cloud: Viral Buzzwords 4.0!

The cloud, and hosted services, present very interesting opportunities. One of the greatest is that people can leverage the a la carte economics of elastic computing to do things that were prohibitively expensive due to the requirements of building and maintaining their own hardware infrastructure. The interesting parts about the current cloud are its lack of entrance friction and elastic cost efficiency, the speed with which new entrants can set up, and the elastic capability to run 100 machine clusters for 1 hour if that is what is needed.

We started Flightcaster almost a year ago, and it is a good example of how startups can leverage cloud compute and storage resources, mix some open source like Hadoop with some data mining, and create interesting new technologies with relatively low capital upfront.

The cloud is not cheaper in general. Once people scale to a certain point, they move off the cloud onto dedicated hardware—not the other way around. That may change, and better hosted services may play a role in the transition, but that will take a while. In the meantime, the interesting part of the cloud is the use of elastic resources and the ability to get up and going quickly. The interesting part is the freedom it gives startups to try things they would never otherwise do.

Another notable thing about the cloud is the new architectures emerging as a result of economic and resource tradeoffs.

Storage of large amounts of data in the cloud is much cheaper with blobstores like Amazon S3 than it is to maintain an always-up cluster for a distributed datastore. If you do mostly offline batch processing and you do not need bulk storage to be online, then it is an attractive setup.

Storage and NoSQL

Taking another glimpse from the future of big data in the cloud.

A Big Data stack…will also need to emerge before cloud computing will be broadly embraced by the enterprise. In many ways, this cloud stack has already been implemented, albeit in primitive form, at large-scale Internet data centers, which quickly encountered the scaling limitations of traditional SQL databases as the volume of data exploded. Instead, high-performance, scalable/distributed, object-orientated data stores are being developed internally and implemented at scale…large web properties have been building their own so-called “NoSQL” databases, also known as distributed, non-relational database systems (DNRDBMS).

There are several misguided points here. First, there is not going to be a big data or cloud stack. Distributed systems are about making trade offs and a move toward problem-specific solutions rather than one-size-fits-all stacks. Second, enterprises already have their solution—expensive data warehousing and consulting support. Will open source projects like Hadoop supported by people like Cloudera take a chunk of the business? Sure. But as I mentioned earlier, the most interesting part about big data and the cloud is not cheaper alternatives for the enterprise, it is the opportunities it facilitates for data-driven startups.

There is a lot of talk about the NoSQL movement. The big idea here is that distributed systems are hard, require tradeoffs, and sometimes we are better off with data storage and processing that are specific to what we are doing with the data. Sometimes even with a small amount of data on a single node, there are better alternatives to SQL queries and relational databases—time series data has long been a good example.

Processing and Hadoop: The Elephant In The Room

There is a broad range of needs for processing large amounts of data. These range from simple needs like calculations for log analysis that just need to occur at scale, to middle of the road needs like BI, to complex needs like scalable modern machine learning and retrieval systems.

There are a different approaches one can use to service specific needs. Again, we see the pattern of moving away from one-size-fits-all stacks, and toward building for your needs. That said, there are very generic abstractions like Map-Reduce that work well for a lot of use cases. Distributed systems are hard to get right, so when something like Hadoop gets a lot of momentum, it retains that momentum until alternatives have the time to mature enough to solve the hard problems with fault tolerance, performance, and so forth. Not everyone is Leonardo da Vinci, so people should not attempt to create these systems on their own unless they really know what they are doing. In that sense, the cloud and big data are facilitators of open source.

An important aspect of processing at scale is abstraction. Writing complex or even simple computations in raw Map-Reduce is verbose for programmers and intimidating for others who might want to play with the data. Abstractions over Map-Reduce like Pig and Hive make simple things easy, and abstractions like Cascading make hard things possible. The Map-Reduce paradigm, and Hadoop in particular, have been a big success. That said, Map-Reduce is not the only important piece of compute infrastructure. Message queues serve as the backbone of a lot of compute architectures – implementations of AMQP, such as rabbitmq, are a prime example. You can accomplish a lot with producers, consumers, and a messaging system. Distributed storage and processing systems can also be very tricky to configure and deploy, requiring a pretty deep understanding of the system – hence the business case for folks like Cloudera.

Learning from Big Data

Hal Varian, Google’s Chief Economist, recently said,

The sexy job in the next ten years will be statisticians… The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it

Unfortunately for those of us working on these problems in real life, it is not so simple. The archetypal data-renaissance man is mathematician, statistician, computer scientist, machine learner, and engineer all rolled into one. There are opportunities where you can lack some of these skills and work with a team that supplements your weak points—a startup is not one of those.

Now that we can store so much data, it is attractive to do previously unimaginable things with it. We are sure to see cool applications in fields from the internet to biotechnology to nanotechnology and fundamental materials science research. Almost all advances in every field of science and technology are now heavily dependent upon data and computing. Machine learning is serving a fantastic role as a bridge between mathematical and statistical models and the worlds of AI, computer science, and software engineering. We are exploring applications in learning from text, social networks, data from scientific experiments, and any other data sources we can get our hands on.

The data renaissance does present some difficult issues. There are not many places one can recieve a good education on working on these problems at large scale. Scaling our modeling and optimization algorithms is hard. We need to figure out how to partition and parallelize, or sometimes trade speed and scale for approximately correct calculations. Another issue is that we are often using simplistic models, albeit with pretty good results in many cases. We would like to move toward a deeper approximation of real intelligence.

But the data renaissance is here. Be a part of it.

This is a guest post by Robert J. Moore, the CEO and co-founder of RJMetrics, an on-demand database analytics and business intelligence startup. His last guest post was an analysis of Twitter user data.

It’s no surprise that Chatroulette is the latest media darling. It has all the elements of a good story: technology, mystery, celebrity, and sex. If you haven’t heard of Chatroulette, this Daily Show segment is a good primer.

We were itching to study Chatroulette in a RJMetrics Dashboard, but no one seemed to have any good data for us to explore. So, we decided compile the data ourselves by leveraging Chatroulette Map, some scrappy programming, and a passionate tech community. We soon had detailed data on 2,883 Chatroulette sessions that tied users to geography, gender, appearance, and more.

Here are a few highlights from our findings:

• About half of all Chatroulette spins connects you with someone from the USA. The next most likely country is France at 15%.
• Of the spins showing a single person, 89% were male and 11% were female.
• You are more likely to encounter a webcam featuring no person at all than one featuring a solo female.
• 8% of spins showed multiple people behind the camera. 1 in 3 females appear as part of such a group. That number is 1 in 12 for males.
• 1 in 8 spins yield something R-rated (or worse)
• You are twice as likely to encounter a sign requesting female nudity than you are to encounter actual female nudity

How We Did It

Thanks to RJMetrics, the analysis was easy. Getting the data, however, was a bit of a challenge. The good news, however, is that a roulette wheel is the statistician’s best friend. The central limit theorem tells us that a large set of random observations allows us to draw high-confidence conclusions about the underlying data set.

We started our process at Chatroulette Map, an awesome new site that plots screenshots from random Chatroulette sessions on a map.

It’s a little-known fact that anyone you chat with on Chatroulette can determine your IP address using a program like Wireshark. Chatroulette Map uses this IP data to geolocate and map random chatters on their website (along with still photos from their chats).

Chatroulette Map is also nice enough to expose all of its data points to anyone who clicks “View Source.” Right in the raw source code of their homepage is the image URL, latitude, longitude, city, state, and country of every chatter on their map. As an added bonus, the file name of each image is a UNIX timestamp of when it was taken. Jackpot. (Note: we tried contacting the creators of Chatroulette Map to participate in this story but did not receive a response.)

Once we had photos, times, and locations, we needed data on what was happening in each chat photo. We coded up a quick webpage that displayed a random photo from the data set and asked some basic multiple-choice questions about that photo. These included questions on age, gender, and what the person in the photo was doing. We coded up the backed so that a photo wouldn’t be taken out of rotation until two votes from different IP addresses provided an identical set of answers.

We posted the link to Hacker News on Saturday night. In under two hours, we received 10,770 photo assessments from 1,012 distinct IP addresses. Every photo received a corroborated profile. We had our data.

Five minutes later, the data was loaded into a hosted dashboard on RJMetrics and returning the results you see below.

Caveats

Before we get to the data, we should point out the uncontrolled inputs that could be skewing these results:

• We know nothing about how Chatroulette matches up chatters, and we act on the assumption that pairings are truly random.
• We know nothing about the methodology used by Chatroulette Map. If they excluded data points for any reason or did not sample randomly, our analysis could be skewed.
• Geolocation by IP address is an imperfect science that is typically only accurate within a few dozen miles. It can also be thrown off by users taking advantage of proxy servers or using other techniques to disguise their IP addresses.
• Human image recognition is imperfect (even if mitigated by our vote convergence system). Any images that were judged incorrectly could skew the results.
• It’s also important to note that statistics about “the average chat session” (which we present here) are not the same as stats about “the average user.” For example, imagine if female chats averaged 100 seconds each, but male chats averaged 10 seconds each. Even if there were equal numbers of male and female users, males would enter the pool more often and would therefore appear in front of you more often, making the “average session” more likely to contain a male chat partner. Because of this, all of our statistics are about the average session and not the average user.

The Results

Gender

As you might expect, you’re most likely to encounter a solo male in any given chat session. 72% of our chat sessions were with solo males. Interestingly, 11% showed no person at all while only 9% showed a solo female. So, if you’re looking for women on Chatroulette, be forewarned: you’re more likely to encounter an empty chair.

Also interesting is the prevalence of groups on Chatroulette. In all, 8% of chats featured a group of people (4% all-male, 2% all-female, and 2% mixed). If you include groups, your chance of encountering a female grows to 13%. However, this means that if you do encounter a female, there is about a 1 in 3 chance that she will be part of a group. In contrast, the chance a male will be part of a group is only about 1 in 12.

Age

This analysis excludes cams where age could not be estimated. As you might expect, most people were young adults (about 70%). About 20% were under 20 and about 10% were 40 and older.

When we combine age with the gender statistics that we tracked above, we learn even more. For example, females tended to be younger than males, with 23% under 20 (vs. 18% for males). Only 3% of females were over 40 (vs. 8% for males).

Groups of females were even younger. Female-only groups were “Teen or Younger” 65% of the time, while groups of males were “Teens or Younger” only 36% of the time. There were no groups whatsoever of people 40 or older.

Location

47% of the Chatroulette participants measured were from the United States. The most popular countries are shown below:

When we combine geography with gender and age, we learn even more:

• Italy had the highest concentration of solo males at 98%. It also had the highest concentration “Men over 40″ at 13% (more than 3x the US rate of 4%).
• The US has the highest concentration of groups at 13%, followed by The Netherlands at 9%.
• Canada had the highest concentration of solo females at 13%, followed by the US at 10%.

Perverts

If you’ve ever used Chatroulette, you probably noticed that not everyone is there just to chat. Some users, which we have affectionately labeled “perverts,” fit into any of these three categories:

• Appear to not be wearing any clothes whatsoever
• Are displaying explicit nudity
• Appear to be committing a lewd act

The overall pervert rate in Chatroulette is 13%. This means about 1 in 8 chat sessions will have something decidedly Rated R (or NC-17) on the other end. Of the perverts that were identified, only 8% were female. Combined with the overall female rate, that means less than 1% of chats feature a female pervert.

Below, we see the “pervert rate” by country:

The United Kingdom dominates the rankings here with a pervert concentration of 22%! Turkey, France, and Germany tie for second place with rates of 15%. Bringing down the global average is the United States, which boasts the lowest pervert concentration of the bunch: 10%.

Also worth mentioning are the users who display signs (like the one below) requesting female nudity.

Signs like this make up between 1% and 2% of all chats. This means that you’re twice as likely to encounter a sign requesting female nudity than you are to encounter actual female nudity.

Validation

In trolling through the thousands of photos collected by Chatroulette Map, I came across this extremely interesting image. It contains a statistical breakdown of what this user saw during his many Chatroulette chat sessions. Sound familiar?

These stats appear to be based on a data set of 1,090 points (pretty impressive for a single user). The numbers are generally in the same ballpark as ours (although we observed a higher pervert rate). We’re not sure who was behind this, but we like their style– they managed to sum up the gist of this blog post in a single image.

Conclusion

Scarcity of the data made this project both challenging and exciting. In an ideal world, it would be great to analyze things like average session length based on different attributes, chat user return rates, cohort analysis, and more. Because of the mostly-anonymous nature of Chatroulette, that data will be hard to come by. For now, at least you have a better idea of what you will see when you hit that Next button.

Guest author Robert J. Moore is the CEO of RJ Metrics, a startup that helps online businesses measure, manage, and monetize better. He was previously a venture capital analyst and currently serves as an advisor to several New York startups. Robert blogs at The Metric System and can be followed on Twitter at @RJMetrics.

We’re here at the SXSW festival in Austin, Texas where Twitter co-founder Evan Williams doing a keynote Q&A with Umair Haque. Williams may use the time to talk a bit about Twitter’s upcoming ad platform. Update: It’s actually an “At Platform” called At Anywhere — more here.

Interestingly enough, Twitter saw its first burst of popularity three years ago at this very conference.

Below find my live notes (paraphrased):

UH: Ev you have something pretty interesting you want to say today?

EW: Yeah, we want to announce something. We wanted to announce our new “At Platform” (undoubtedly to be spelled an @ Platform) – a way to integrate Twitter into any website. “At Anywhere” – basically this allows you to place the Twitter hovercards on any site. We have 13 sites we’re launching with including Amazon, ebay, Yahoo, Digg, Bing, Meebo, Salesforce.

UH: So what can you do with this?

EW: You can easily tweet from any page that is using this. Also, maybe you want talk to authors of posts without going to Twitter itself, you can just hover over their name and tweet them. Twitter is a very easy way to keep in touch.

UH: So this helps you contextualize information. But why would sites use this?

EW: A connection to users you didn’t have before – and it keeps people coming back. And it will result in more followers for a site. Also, hopefully more people who are your fans using twitter to talk about you or your content. And you can bring in users’ tweets talking about your site.

UH: So it’s a platform to juice up site’s networks and virility. But it’s an “At Platform” not an “Ad Platform”.

EW: Yeah, it’s about lowering the barrier for information.

UH: What makes 21st century businesses different? Like Twitter? The first principle to me is experimentation. Why are you willing to explore different possibilities?

EW: Experimentation lets you create value. “Whatever you assume when you start out, you’re wrong.” Most of the great businesses of our time have experimented. Like Google.

UH: So it’s about creating value, then figuring it out?

EW: Yes, it’s about creating experience for users and businesses. There is a ton of business use on Twitter today — it’s one of the biggest uses. We want to make that better, easier, faster.

UH: What is Twitter evolving to?

EW: What is Twitter has always been a tough question to answer. We think of it as an information network — different from a social network. It’s about getting info and also sharing. You can take advantage of Twitter without sharing anything about your life. We need to increase the signal-to-noise ratio.

UH: So better information, better connections, better choices.

EW: Yes.

UH: Experimentation is about iteration. So how does that happen at Twitter?

EW: We have a bunch of awesome people in the company now. We’ve grown very quickly over the past year. Our employee growth curve is almost like our user growth curve now. We have people on focused teams, like mobile, or internationalization. We’re worried about central thinking and slow processes. So we tell our teams to “go for it.”

UH: So what’s your role?

EW: I don’t get into the nuts and bolts of code, cause things would be a big mess. I spend most of my time thinking about the high level issues. And I think a lot about the company – how do we scale the company, about our culture, etc. How do we define the characteristics we want. I think there is a parallel between the service and the company — openness is huge, transparency.

UH: So openness is very important. Help us trace the arc of openness at Twitter.

EW: Yeah, it means a lot of things. We debated if openness or transparency. “A window is transparent, but a door is open.” The users have taken Twitter and morphed it into what they want it to be. Now developers are doing the same thing. Openness is really a survival technique.

I sit down with new employees when they start and go over 9 assumptions you should have about working at Twitter. One key one is assume there are more smart people outside the company than insides.

UH: What about giving the golden goose away? Why be so open?

EW: That was a big question for us – the deals with Bing and Google. These were the first guys we shared our full stream with. There’s a lot of debate about that. Because we don’t have a business model yet, so why give it away? But we went back to the principle of giving users the most value.

There are 50 million tweets a day, how do we show you the best ones for you? Right now, we don’t do a good enough job of that. But with these partnerships, we have more chances to do that.

UH: Was there a lot of internal debate about this?

EW: Yeah, there was a ton. But we decided it was good. And now we’ve expanded the deals – like with Yahoo. And a few weeks ago we talked about giving this data to thousands of others.

Now third party developers are building a lot of value. Like adding pictures to Twitter.

CoTweet and HootSuite are really interesting too. Twitter.com isn’t a good interface for doing customer support, but those guys are. CoTweet just got acquired by a company that wants to focus on that more.

We’d love to see much more focus on creating these deep experiences that create value.

UH: So experimentation and openness. Other companies want control, like Apple. How open are you guys?

EW: We’re pretty open – there is some control we need to employ because if we were infinitely open we’d be doing a disservice to users. Openness can work against you still. It has to be managed a lot. Having an open API makes it easier to make apps that will spam users. We send cease and desists everyday to companies making spam tools. We have to exert some control.

UH: I think shepherding is a good way to put it. So you had some interesting use recently – such as the earthquake in Chile.

EW: I got an email recently about the earthquake, thanking us for helping with the situation. This is very gratifying for us because we’ve always held it important for Twitter to reach the weakest signals in the world. We started out with a big focus on SMS – and it’s still really important to us. Because it reaches so many people. We have deals with 65 carriers around the world to send these SMS tweets.

We’re at the beginning. We’re seeing really strong growth in India where SMS is huge. And in the Middle East.

UH: I think this changing the world stuff is the future for entrepreneurs. It gets to the heart of the point about inclusiveness. So – what is an “active user”?

EW: To me it comes back to – is someone getting value out of Twitter? If they don’t have an account it’s hard to know, like people who search Google for tweets. In the beginning we put a lot of focus on telling the world or your friends and family what you’re doing. But now there is something interesting on Twitter for everyone – like the Flaming Lips being on Twitter, you can get updates on the band.

And as more people start getting information on Twitter, they’re more likely to get involved.

UH: Someone has started using Twitter inside the White House, right?

EW: Yeah, it’s really interesting that it’s from in the White House. It’s an official channel, but they’re using it a different type of way. It’s about reducing the walls between people with a lot of influence, and those who they influence. And that’s the most profound promise of the Internet. This is the wave I started on 10 years ago with blogging. It’s about the democracy of information. Anyone can put information on the web — that’s huge.

UH: Tweet Minister in the UK aggregates the tweets from members of parliament. This is re-wiring society in some ways. But we also have a counter-force – like state control of information.

EW: In some regions, yes, this is bad and hurting the web. But the Internet is a tidal wave that you will not be able to keep out. Like in China, who knows how long those firewalls will hold up – but not forever.

UH: Yes, there are many ways to get through the firewalls already. There’s a lot of pressure on them.

Let’s talk about “betterness.” I booked a trip to his five star resort in an exotic land. When I got there, it was a shack. The manager couldn’t do anything — so I put it on Twitter. Within 15 minutes the booking company called me, and in 20 minutes I got a new hotel. In a half an hour my vacation was fixed.

EW: That’s great. Our hope is that this is the norm, not a fluke. We have a bit of a dichotomy, because there is more everyday you want to search for. We don’t just want to maximize that, we hope to make Twitter more useful to you. We want to decrease time you spend on Twitter, not increase it.

Recently we went through a process to define our operating principles. The number one principle is “be a force for good.” Another principle is “pay attention.”

UH: David Pogue did a campaign against hidden charges from the carriers. It’s the same thing with the hotel operator and me. I know you’re a big fan of Warren Buffet – he also believes in creating real value.

EW: Yes, from a business perspective, Twitter needs to fundamentally be about helping people make better decisions. Or the help something happen that normally wouldn’t. Like the donations to Haiti through text message — we weren’t taking the money, but it spread virally through Twitter. People want to help each other out, we need to reduce the friction.

UH: Is that what you want to do with the new At Platform?

EW: Yes, totally. We’ll see what happens, the obvious stuff is more tweeting, but I think it’s a lowering of the friction as well.

UH: You ask yourself, how would i make Walmart better? Why ask yourself that?

EW: Because as we look at how businesses are using Twitter – we want our tool to help businesses get better.

The world is so often a black box where there is no communication. There’s a lack of dialogue and a lack of transparency. The promise of all these technologies is that this goes away. You close the loop.

UH: Outline for us your big picture goals.

EW: Fostering the open exchange of information. To be a force for good. The ease of exchange of information is important. Help out other people with something as small as a retweet. That’s our ambition.

UH: Google is all about archiving the world’s information. Yours is different — creating new information.

It’s all about advantage though – what’s your advantage.

EW: Our advantage will only come if everyone wins. We only do win-win deals. Because any deal where someone is losing is unsustainable. That’s why we haven’t turned on the revenue yet — there’s a lot of low-hanging fruit, but none of it is sustainable.

Creating an advantage for other people and not giving them a reason to work around you – that’s key.

UH: Is the Internet making a better media industry?

EW: I think there’s a huge shift going on – but it’s an ecosystem where everything is involved. This user-generated content just makes things richer. Blogging and traditional media work together. Twitter compliments traditional media. I was talking last night to some guys from CNN – it’s helped them change what they do. It’s a win-win.

UH: How will the At Platform speak to that?

EW: Hopefully these guys will us it to get the new out there.

UH: What makes you tick?

EW: There are two types of entrepreneurs. What drives me is creating things that didn’t exist before. Your product or service should be at the end of the sentence: “wouldn’t it be awesome if…”

It’s creating new stuff versus extracting from old stuff. There are people who look at money as the goal versus the teams. I create businesses to make new things. It’s a fuel for creating more things in the world. I’ve been lucky to stumble upon things that have helped change the world.

UH: Why focus on these things though?

EW: Largely luck. But maybe it’s what interests me. Twitter was a side project of Odeo – my cofounders came up with it. Blogging was a side project too at one point.

UH: If something is awesome, people will use it.

EW: Yes.

Also, helping others succeed is a sub principle of ours.

UH: Tell us one or two more of them.

EW: Be a force for good, pay attention — make things happen is another one. There’s also building a culture of trust.

UH: What are your big lessons to other entrepreneurs?

EW: Create something you want to exist in the world. Another is focus. Many people are trying to do a lot of things when they should be doing one thing. You may be wrong with whatever you’re trying out, but you’ll try other things.

A lot of the great companies are now coming from outside Silicon Valley. You don’t have to be there.

That’s a wrap.

Many people (including myself) have come to the conclusion that Gmail, with its threaded messages, spam filtering, and vast storage space, is one of the web’s best webmail providers. In fact, we like it so much that we use it for both our personal accounts and work accounts using Google Apps. But that also poses a problem: many of us wind up having to maintain two separate Google accounts, which means we have to swap logins whenever our Gmail, Reader, or other data is stored under the other account. Fortunately, there may be an end in sight for this juggling act.

As today’s SXSW panel on Gmail came to a close, the panelists revealed one last juicy tidbit: they’re working to resolve the problems with multiple namespaces that users have to deal with. The team didn’t get specific — they simply repeated that they have to deal with the same problems, as they have “@google.com” accounts for work and standard Gmail accounts for personal use. And they know it’s a pain.

There’s no time frame, and we have no idea what form the feature will take. But at least we know Google is working on it.

Image by Helico

Whether the bloggers like it or not , it’s looking like the iPad is a hit . Initial estimates show that over 120,000 iPads were pre-ordered on friday, according to Investor Village. Some estimates showed roughly 50,000 devices ordered in the first two hours

Here is the original: 
iPad sales estimated to top 120,000 on first day

Short Version: We now have so much storage in our homes that we could probably, each of us, start our own Rapidshare service. But how do we get all that data to the other machines on our network or, better yet, out onto the Internet

Originally posted here:
Review: Iomega iConnect Wireless Data Station

Here’s a clever yet simple DIY project for you, just in time for the weekend. You could probably even through this thing together before the next camping trip, even if you are heading out tonight

Go here to read the rest: 
DIY: Stereo Cooler

Need some 3G? Buy some 3G! Macrumors has some shots of the interface that will appear when you need to add data to your iPad data plan. You can either add 250MB for $14.99 or change to an unlimited plan for $26.99 a month

See the rest here:
iPad 3G will be a la carte

Business to business software can be a tough sell. Online B2B can be even a harder sell. While there is certainly money to be made, unless you’re one of the big players, the likelihood you’re going to succeed is pretty small. Starting today, Google is taking their roll as one of the big players and extending a platform to boost some smaller players.

Tonight, Google has unveiled their Google Apps Marketplace. This is an app store for enterprise apps in the cloud. Using a set of APIs, these third-party apps can deeply integrate their products within Google Apps, which already some 25 million people are using. And that also includes over 2 million businesses ranging from startups, to small businesses, to Fortune 500 companies.

For customers, this means a one-stop shop for a variety of applications that their business or organization can use. And it’s extremely simple to get started with apps in the marketplace — it just takes 4 clicks, Google says (though that initial click will have to come from your domain admin to approve the use of the app). For developers, particularly small startup developers, it means instant access to more users than they can likely imagine. It also potentially means something more important: money.

Like the popular mobile app stores (Apple’s App Store and Google’s own Android Market), Google is allowing developers to sell their apps through this Marketplace. And they’re actually offering a better deal: Google will keep just 20% of the revenue, while the developers keep the other 80% (compared to a 30/70 split with the Android Market). The reason for this better split is that Google believes the B2B market is a bit different, and they want to entice developers to join on board. And instead of Apple’s App Store, which charges a $100 yearly fee to developers, Google is charging a one-time fee of $100 to enroll in the program — and that’s for as many apps as you want to create.

As for what Google will do with their 20% share, they’re not entirely sure. “We don’t know what will happen with the revenue, but we think it’s a very fair rev share for the value we’re providing,” Google Vice President of Engineering Vic Gundotra says.

As you might expect, in the Marketplace, Google will feature certain apps on a rotating basis. And each will have a star rating system and reviews written by people who have used the app. Apps will be grouped into different categories to make it easier for customers to find exactly what they’re looking for. Once they do, the four steps alluded to above are:

1. Click “Add it now”
2. Agree to the vendor’s Terms of Service
3. Grant access to the data that the app is requesting.  Some apps require data access, some don’t – only grant access to apps you trust.
4. Turn it on and start enjoying your increased productivity

So how does this all work? Google connection points for integration into Apps are actually done through open protocols such as OAuth. And while signing-in may seem like a pain across different apps, Google has streamlined that as well thanks to another open protocol: OpenID.

Once an app is hooked in to Google Apps, it will appear on your main Apps Dashboard alongside the other Google-made apps you use. It will even appear in the “more” drop down that Google uses in the toolbar across its properties. And because these apps are so tightly woven into Google Apps, they can take advantage of the built-in Google Apps such as Gmail and Gtalk to easily communicate within the third-party apps.

And there’s more. While it’s not quite ready to launch just yet, in the second half of 2010, Google plans to launch flexible billing options for third-parties using their services. Basically, this will allow companies to use Google Checkout to handle complicated billings, such as subscriptions. This could mean trouble for startups specifically in this space, such as Recurly. Also coming later will be detailed analytics for transactions, we’re told. For now, developers are free to hook up their data to their own analytic programs to run their numbers.

While Google’s options for this Marketplace sound nice and open, there’s actually something even better: you don’t have to build your apps on their platform. Whereas a big player like Salesforce wants to keep the apps it works with in the Force.com ecosystem, Google doesn’t care where you build it — it can be on App Engine, or on anything else. You simply hook your app up to the APIs and you’re ready to go. It’s a model so enticing that even a big Google competitor in this space, Zoho, is ready to work with them, and is launching as an initial partner. All told, there are more than 50 companies partnering up at launch, including a winner of the audience award at this year’s TechCrunch50, Socialwok.

As to whether Google could eventually roll this app store model out to the more consumer facing apps they offer, Gundotra gave me the old, “We have nothing to announce at this time.” That reads suspiciously to me like a “yes,” provided this is the hit it seems like it should be.