Outside the AB testing box

Posted on December 28, 2011 by BorisDinkevich

How we got rid of 80% of our support calls

The year is 2010 and yours truly is deciding on opening a Groupon like coupon site back in his country. A bit of coding + a bit of selling and we were up in the air.

Beyond being the VP-RnD, CFO and Janitor, I had the dubious role of being the Head (and sole employee) of our “Customer Support”.

A wise man said once, “To know your customers, you must know your customers“. So instead of spamming my visitors with various pop-up/slide-on/drop-left/pop-off tools for suggestions/questions/injestions, I just went on and put my personal mobile phone. Right there on the buy page, in HUGE capital letters.

Number of people who emailed our support email during the next week: 2.
Number of people who called me up (thankful during reasonable hours): ~100

The next bit might surprise you: It appears that even in the year 2010, the average 30 year old mom (that is our main demographic – I asked) is utterly terrified of using her credit card on the web.

“Is it possible to buy by phone ?” She would ask.
“Well yes mam, did you have problems buying on the site ?“.
“Noo.. its just that I am worried about putting my card on-line“.

In the beginning I was naive enough to try:
“No need, we have SSL on the page and our servers are PCI compliant” line…
“Yeeaa… but.. can I buy over the phone ?“  is what I got every time.

So we proceeded with me opening the website on my PC and filling in her details, as she dictated them to me.

Not a a week has passed and the executive team started getting multiple complaints from the support staff (Me). And the VP-RnD decided that something must be done.

Something being done

Option A: Do some bad-ass A/B testing. e.g. : Move button 17 pixels to the left, change the first letter of some words to capitals and maybe even try switching the radio buttons from being round to being square.

Option B: Try to address the issue at hand.

Since my users weren’t very tech savvy and couldn’t/wouldn’t understand that the page is secured with SSL. It became apparent that they needed to be told. Proposed solution: Put a big-red-dotted border around the form and write in large, red and reassuring letters: “SECURE ZONE”

The form before:

How our buy form looked before

The form after:

Our form after being "secured"

For the Hebrew challenged, the writing in red is “This area is secured by SSL”.

 

The test

Next step: Ask (sample user: mom) to buy a coupon and provide feedback.

(In the initial test version, I also had her enter her name and email in fields that were outside the “safe zone”).

Feedback:  “I felt a bit uncomfortable filling in the name and email, since they were not secure.. but I filled anyway because its your site” (GREAT SUCCESS !!)

3 minutes later we went live, one week later I took my phone off the site.

Results: Even with increased activity, phone calls went from ~100/week to ~20.

Conclusions

You can A/B test the crap out of your site, but you might try asking your users if something is wrong first.

“Most people will assume a safe is safe if it looks like a safe safe”

 

Posted in startups, technology | Leave a comment

First employee of startup ? You are probably getting screwed !

Posted on August 31, 2011 by BorisDinkevich

Same story again

A friend of mine just told me about his exciting new opportunity:
He met two guys who are opening a new startup. They are known in the community and already had a small exit before. For the past half year they have been working on a startup, got some code and are now just finishing with a 500K$ seed raise.

And, they are looking for their first employee with significant equity !

After some interviews and f2f meet ups, they decided to offer him a job. Of course, they can’t afford to pay market salary just yet but they are willing to give significant equity, a chance to influence the company’s direction and become a core member of their team !

And he was just about to sign up.

Me: “So tell me more, what kind of numbers are we talking about.”
Him: “Good ! Half my normal salary and 1% of the company”

Now this might sound very reasonable and exciting, but perhaps its time for…

A bit of… math !

Taking the optimistic route
– In one year the company does series A and can pay full salary
– They raised 500K at a high 2M (pre-money) valuation, giving away 20%
– His normal 100K/years is cut to 50K/year.

From the looks of it, he is pretty much saving the company / putting in 50K.

What does an investor get for putting in 50K:
- 2% of company shares
- Priority on exit
- Invest small part of his capital

What does employee get for putting in 50K
– %1 company shares (options)
– 4 year vesting plan
– Exercise price
– No priority
– No control or rights of information
– Invest large part of own capital

Strange numbers, and yet people agree to them every day.
Because their 1% will turn into a million.
And as working for half their salary(or less).. hey.. its just for a while.

How low did you compromise ?

EDIT – 14:00:00 UTC
Interesting discussion on HackerNews

Posted in startups | 46 Comments

We are the McDonnalds of Shoe stores

Posted on August 10, 2011 by BorisDinkevich

 

“I am opening a new shoe store”

That is what a good friend of mine, happily announced to me.
“Not just a shoe store, we are going to be the McDonnalds of shoe stores !”
Shoe store ? Frankly I wasn’t too surprised, having heard that quite a few times already.
And he proceeded to happily tell me of his grand plans.

The shoe store he was planning was going to open in the huge mall in the capital.
Having thousands of patrons a day, made it an ideal target for him.
And regarding all the other hundreds of shoe stores there ? He didn’t seem to worried, on the contrary:
“Its great ! All those stores just means that there is a market for shoes !”.

But, he knew that just opening another shoe store was not enough, so, his will be special. Every person visiting, will get (hold for it….) a free high-quality coffee !
And who doesn’t want coffee ? Everyone will come for a drink and buy their shoes on the way.

“Wouldn’t you like free coffee ?” he asked me. And I had to agree I would. Sure it will put them at a loss in the beginning, but he made quite a few calculations in excel and in a year or two, the shoe sales will be high enough to cover the coffee and begin to become profitable. (Especially since they planned to move to their own coffee machines and bulk deals with coffee importers by then)

We chatted about coffee types for a while and then I told him of another friend of mine, who just days ago told me of his shoe store (that was already finishing their renovation) Their idea was: “massage chairs” – You get a massage while measuring shoes – people stayed longer – more shoes bought.

He dismissed them with a wave of his hand. “Bah ! We will get those too in time, and many other stuff others have”. In the end its all about execution. He had a great team and a professional designer who just left IKEA.
“With a bit of hard work, we will become the biggest shoe shop in the mall and then spread to other cities via franchise and be like Starbucks or McDonnalds”

I like it, I said, but there are hundreds of shoe stores in that mall, why not try something different ?
Or better yet, why no start in the smaller cities and villages where there are almost no stores, and the existing ones are small and bad ?
“Thats not how you do it if you plan big !” He and his partner just joined the state’s leading Coaching firm’s entrepreneurs plan. With the help of the firm, they will get money and connections needed to reach the Mall’s weekly newsletter and maybe even be featured on the large billboard at the entrance.

Sure competition was hard, sure there was lots of it. But their idea had a unique twist and with the proper execution ? The sky was the limit.

All that was three months ago, he has already quit his job and started working full time with his partner on designing the interior and the shoes selection.
They got some money from friends and a group of investors, moved to the big city near the huge mall and started looking for a good location for their shoe store.

And me ? Well, since all the shoes stores are in the big mall in the capital and none are in the hundred little cities around it (one of which I live in), Ill be driving tomorrow, 60 miles.. to buy shoes.

Posted in startups | Leave a comment

True web security from Javascript injections ?

Posted on April 15, 2011 by BorisDinkevich

With the coming of FireFox 4, a new (and wildly unnoticed) feature was added to the browser:
CSP – Content Security Policy.

The idea behind the CSP, was to try to mitigate many of attacks and abuses we have been seeing on the web in recent years. From Click hijackings, unexpected cross-site-scripting, injections by corporate / country or internet provider routers and many more.

How does it work ?

When serving a web page, the server can now specify to the browser, exactly where data can come from and where it can go.The policy can show which JavaScript files can run (and from which domains), where XHR request can be made and even where static resources (like pictures) can be loaded from.

To be exact a new HTTP header is added by the web server:

X-Content-Security-Policy: policy

The browser (currently only FF4) parses the rules and prevent the page from doing anything unauthorized.

For example:

X-Content-Security-Policy: allow ‘self’; img-src *; media-src media1.com media2.com; script-src userscripts.example.com

(More examples here)

A small test

One useful feature of CSP, is its ability to report violations. With the help of RoR and Redis, here is a little site, where you can check if any unexpected scripts are being injected to your plain HTTP pages:

Test your connection for JS injections

Read more about CSP

Posted in technology | 1 Comment

Why is USB so SLOoow..

Posted on March 21, 2011 by BorisDinkevich

Have you ever noticed that if, during the copying of a file to a USB disk, you start to copy another. The performance of both drops drastically ?

A small test

Take the simple test (If you have linux)

Create a 100 mega byte file:

> dd if=/dev/zero of=100m bs=10485760 count=100

Now create two test files:

# sequential_test.sh

cp 100m /media/usb_disk/file1
sync
cp 100m /media/usb_disk/file2
sync

# parallel_test.sh

cp 100m /media/usb_disk/file3 &
cp 100m /media/usb_disk/file4 &
sync

Run the tests:

> time sh sequential_test.sh
> time sh parallel_test.sh

Performance may vary, I got the sequential_test being 3 times as fast.
Depending on fragmentation and you OS, the difference can go up much higher than that.

How USB transfers work (Stack level) ?

USB is a multi-layer protocol with multiple stages at each layer.
We will examine only the relevant culprits ignoring the less relevant ones.

Transfer a 4K block from PC to HD:

OS: Determines where on the target disk to put the block
- Mass storage: A SCSI header is created
(containing information such as size of block (4K), target block location (LBA), direction (write) and others)
– Mass storage: A Mass-Storage protocol wraps the SCSI header with its own
(holding pretty much the same info again, don’t ask why)
– Mass storage: The header from above (31bytes) is scheduled for transfer
—- USB Core: Figures out the USB address of the device and endpoint to use
—— USB Host Controller Driver: Schedules an OUT transfer of 31bytes
—— USB Host Controller HW: Sends OUT token to device
(this tells the device to expect data)
—— USB Host Controller HW: Sends 31byte packet to device
—— USB Device Controller HW: ACKs the packet
—- USB Host Controller Driver: Discovers of finished transfer
(usually via interrupt and done-list traversal)
– Mass storage: Schedules the data transfer of 4K
–*** Same transfer sequence as above ***
– Mass storage: Schedules a read of 13bytes (status header from device)
–*** Same transfer sequence as above ***
OS: Transfer complete

Resulting with 3 data transfers on the USB for a single data transfer on the OS level.

How USB transfers work (Controller level) ?

Being a Master-Slave, Shared-Bus the Host synchronizes the communication on the bus by issuing a periodic SOF (Start Of Frame) token every 1ms (1/8ms for High-speed).
This effectively  divides each second into 8000 transfer windows (from now on we will we will speak of Hi-Speed mode only)
As a result controllers cannot usually add a transfer to the current window (it will be executed on next window).

Also, the ‘done-list’ interrupts and processing is done at the end of a window.

Sample 1 byte transfer sequence:

SOF0

* Controller Driver schedules a 1byte transfer
* Controller HW adds transfer to pending list

SOF1

* Controller HW executes Transfer

SOF2

* Controller Driver goes over the done list and notifies upper layer and schedules the next Transfer

From the above example, we can see that trying to sequentially send 1byte packets (ala serial line style) puts at a practical 4000bytes/sec limit.

It does go faster

Obviously USB can work much faster, how ?

The maximum size of a single bulk-transfer packet is 512bytes. A modern host controller, can usually push as many as 13 of those if not bothered by other transfers. It can also allow scheduling of large buffer chains by the driver, allowing it to fill frames.

This puts us at a possible:

8000 frames * 13 packets * 512 bytes / sec =  ~50M/sec
(we managed to get to around ~38 at our labs)

Back to bulk transfers

As we have seen the Mass-Storage protocol has 3 stages: Header, Data, Status. With a minimum of 2 frames for each stage, that gets us to 6frames for our 512byte transfer.

Theoretical speed: 8000/6 * 512 = ~666K/second

It should be obvious now, that getting the Data stage as large as possible will greatly speed up our transfer. The larger the transfer stage the smaller effect all the other stages will have.

But even smaller sizes, for example 32K:

Header: 2 frames
Data: (32K / 512)/13 + 1 frames
Status: 2 frames
= 10 frames per 32K

Effective speed: (8000 / 10) * 32K = 25M/sec
(Only 2 times as slow, thats because we spend 5/10 frames for data transfer instead of 10/10)

In practice, most device controllers / drivers / hardware, take much more time to parse and setup a transfer folliwing a header. And the overhead for a data transfer can rise from the theoretical minimum of 5 to 10-20 or higher. Making small transfer sizes much more expensive.

And here is where parallelizm hits:
Trying to send two data streams at the same time can still work very fast, if the OS will be willing to submit them in large chunks. Alas, in most cases, in the name of parallel execution, the transfers might get chopped down to smaller blocks.

Granted for normal magnetic disks, the seek times can be unavoidable for parallel copy. But the real performance hit is usually with the flash devices, that while have no seek times and should be able to handle parallel transfers very fast, have slow setup for the SCSI/Mass-Storage handling. Resulting in very poor performance on small transfer blocks.

Posted in Uncategorized | Leave a comment

Chrome Extension store – is Google brewing another fail ?

Posted on December 2, 2010 by BorisDinkevich

pimped car

Have you read about the upcoming Chrome Extension Store ?

Yet another pending failure from Google ?

As browser extensions are slowly gaining ground and popularity, I urge you to take a moment to look at the extensions you already have installed to see if there were any worth spending money on.

Now consider…

I know people with beautiful houses, that spent $10,000 on new kitchens.

I know people with nice car, that spent $1,000 on sound systems.

I know people with hi-end game PCs, that spent $100 on extra memory

I know people with shiny iPhones, that spent $10 on cool apps

I know people with a cheese-burger, that spent $1 to add onions and fries.

Everything is relative

When it is time to improve and upgrade, people usually use the original gadget cost to determine if the additional cost is reasonable.

Most probably wouldn’t even considering buying a $100 app for their $400 iPhone.

And yet find it easy to spend the same $100 to improve their $10,000 car.

Now tell me

How much would you pay to improve your FREE browser ?

Posted in technology | 3 Comments

Deflowering the local Moms

Posted on May 16, 2010 by BorisDinkevich

Let me tell you a story:

One of my current projects is a website, that sells to young moms directly online. Now (as a test) I put up my private phone on the site, stating that who ever is interested, may use it to pay, instead of the site. And soon enough, the calls started to pour in.

I would always start the process with a question “Was there some problem during the purchasing process, that made you decide to call by phone ?”
The normal reply was, that they called because they don’t trust the site.


Their solution ? Call a mobile phone number and give a (rather nice) guy on the other end, all their credit card details.


Attempts at gentle suggestions, that this method is even less secure (since I don’t take their details to the bank, but use the exact same system myself), were met with “Oh.. “. And I am happy to say I convinced some to trying paying online, for the first time.


And so, a week later. I can proudly say, that I have broken the (buying online) virginity of some of the local moms.


You can thank me later.




Posted in Uncategorized | 1 Comment

Why we got brilliant engineers for only 2,000$ / year

Posted on March 27, 2010 by BorisDinkevich

Small Alley

The What

There is a small cute neighborhood in the middle of Tel-Aviv. As I was walking there a few nights back a strange though struck me. I was on a long alley, with every other building having either a small coffee-shop, a bar or a restaurant. And surprisingly, they were all packed full.

And that reminded me of another place, very similar but with a very important difference: “The internet”

The Internet is not unlike a huge mall. It is packed full of tiny niche stores (startups) trying to make it big (or small).  And as any self-respecting small business, they will all burn a bunch of capital (usually time in this case) and close down if enough customers don’t show up. But unlike that little alley, most of them are almost empty now. Hardly a customer in sight. There are simply too many businesses and too few clients here. Too many niche coffee shops  and tiny “vintage water bottle” businesses.  In the restaurant business, people would tell you it is insane to open another tiny “mom’s breads” place. But the young entrepreneurs keep on coming, with the bright light of belief in their eyes.

At first I was sad, so many talented developers, investing so much of their time and effort. And all fighting for the same small crowd. Instead of working for hire and making huge salaries, they invest that time into projects that, for the vast majority, will not cover the investment. Thousands of programmer hours being thrown in vain..

But then the opportunistic entrepreneur in me woke up and slapped me hard, square on the face. Those millions of dollars in engineering fees are not really disappearing. They are actively being consumed by people like.. me. I am the direct benifactor, the one who keeps getting top quality work for virtually free.

The How

I already have about 5 different companies that I pay 10-20$ a month, for different services that would of taken me months to implement.

And the amazing bit ? They are working so hard to get customers and improve their product. That if something is missing from their offering, a quick email, and a few weeks later I have the features I want. For FREE!

Just imagine going to the local restaurant, paying 10% of the bill and getting the cook to add that custom vanilla and pepperoni sandwich that you love so much. Sounds crazy ? Well not in the current Internet world.

Sure it might look like a win-win situation. I get a product I need for ultra-low cost. They get a paying customer and improve their offering. But I still feel a little bad. Because I know that most of them will never makeup for the time and heart they invested.

But for now I got teams of brilliant people working for me, and that, at only ~2,000$ / year.

Now that’s a GOOD deal.

Posted in startups | Tagged | 8 Comments

How to remove a Digital Tattoo

Posted on March 16, 2010 by BorisDinkevich

First, a story

Back when I was 16, it was considered very cool to get a tatoo. Some of my friends actually got the nerve (to disobey their parents) and went to get tiny bits of color embedded (mostly un)skillfully into their skin.

Now I personally considered it to be what we like to call “a bad move”. Its not that they are bad in principle. Its just that they will have an affect on your life. And be it good or bad, you will never be able to remove it.

So I never went to get one. No tattoo for me…

* Skip forward 13 years *

The people I meet now

Let me tell you a secret. If you ever emailed me, called me or were even causally mentioned to me. I investigated you. (Yes you)

But its not as creepy as it might sound. Before any interaction with a new person, I tend to spend a few minutes looking them up in LinkedIn (Who they are professionally), Facebook (The human side) and Google (everything else). This usually sets the tone of the future engagement and gives me a “virtual first impression”.

A bit of free time

Thats what I had, so I decided it might be wise to do the same on myself. Basically a mirror experiment, to see how I appear to others with similar approach.

LinkedIn

- A fully professional overview that I try to keep updated.  [10/10 points]

Facebook

- Luckily I control (+/-) my Facebook account. Its 99% private and only displays a not-so-flattering photo.   [9/10 points]

Google

-  …. OH MY GOD !

Remember the time when I was 18 and a friend wanted to show me what RDND is all about ? And we signed up on some lame site with our REAL names ? Well I don’t. But guess who does ? GOOGLE.

Remember the time when I was giving “expert” advise on some programming forums ? Not really. But you know who remembers ? GOOGLE.

Remember the 1000 stupid things I emailed on some obscure long forgotten mailing list ? No ? Don’t I ? Yea, you figured out where this is going.

The bad news

Between old email addresses, forgotten passwords, mirroring and caching. It is simply impossible to delete these traces of stupidity.

Sure we can argue that most are not harmful. But they do create an image of you, the  ”First Virtual Impression” you leave on other people.

Just like a tattoo, not bad per-say, but with implications on your life.  Be it bad or good. Its not going away, it is there. FOREVER

Help!

Is there a doctor who can help me remove my Digital Tattoos ?

Posted in Uncategorized | 6 Comments

Whats the best time to post to HackerNews

Posted on March 8, 2010 by BorisDinkevich

Yesterday while pondering just that, I spoke with a friend and we decided to pull some data and make some graphs.

First order of business was to get a DB of posts, and it seemed like crawling was the only option.

But then luck strikes, and someone posts 6 months of HN data* !

Since what really interests me is the amount of traffic a post generates. I decided to get the average rating. With the assumption being:

  • People read -> People vote -> High rank

I have ignored the relative placement on the main page, since a high placement during low-hours does not drive higher traffic, making the placement a poor indicator.

So, a bit of python and here it is:

The average Points and Comments for each post, based on its submission time (UTC).

The average Points and Comments for each post, based on its submission day-of-the-week.

With timing you can get your post almost 50% more.

These charts show the relative percentage of submission of front page articles based on time of submission (Hour(UTC) and Day-of-week):

* Please note that the DB used, only provides information on posts that made it to the main page. “Failed” submissions are not counted and might affect the results.

[Keep me posted]

Posted in Uncategorized | 6 Comments