Vaibhav Verma

Occasionally, I blog about things that interest me.

This book was a total page turner. It has everything you’d want from a good story

1. famous, familiar characters like Travis Kalanick, Anthony Levandowski, Bill Gurley, Larry Page, Susan Fowler, Masayoshi San
2. incredible events - both good and bad
3. behind-the-curtains peek into a familar product and a hypergrowth tech company
4. diverse, international stage
5. high stakes power struggle with compelling characters
6. complicated plot - the line between good and evil is not so clear

I looked forward to the next chapter just as much as I’d look forward to the next episode of Game of Thrones.

If any of this sounds like your cup of tea, pick up the book and start reading it!

Otherwise, keep on reading and I’ll share what I enjoyed most about this book.

My reasons for loving this book (spoilers!)

The book made me respect and understand Travis Kalanick

Before I read the book, I knew of Travis Kalanick from Uber’s press stories over the last decade. I thought he was an obnoxious, negligent businessman that got lucky by being in the right place at the right time. I thought that he had no redeeming qualities and that I’d probably do his job better than he could.

As I read the book though, I had to revise my opinion.

A precocious child, he picked up his father’s skills with mathematics, impressing others with his ability to speed through arithmetic in his head where other classmates needed pencil and paper.

Kalanick himself was no slouch on the SAT. He scored a 1580, just twenty points shy of perfect, and whipped through the math portion of the test with plenty of time to spare. Friends remember his savant-like math abilities. “We were driving across town in Los Angeles once, and Kalanick saw a street sign that said we were seventeen miles from where we were going,” recalled Sean Stanton, a friend and former colleague. “He looked down at the speedometer and saw our average speed, and in a few seconds rattled off how long it would take us to get there so we could make it in time for our meeting. I mean, who does that?”

He and his computer science friends took full advantage of their T1 connection. They battled each other in games like Quake, Doom, and StarCraft. File-sharing parties were common; groups spent hours trading and downloading music, movies, and images, swapping files as if they were baseball cards.

Travis was a math whiz with a degree in Computer Science?! He was more similar to me than I thought, and easier to relate to. If I knew him in school, I’d consider him a fellow nerd.

Uber was not Travis’s first company. He founded two peer to peer file sharing companies before he worked for Uber - Scour and Red Swoosh. Let’s look at the quandries that Travis dealt with at these companies:

At one point, when Red Swoosh was running out of money, an employee dipped into the company’s payroll tax withholdings—money a company reserves to pay the IRS the taxes it owes—to fund operations.

He was later informed by an advisor that the company might be committing tax fraud. This would stick with Kalanick for years; he felt betrayed, put in legal jeopardy by a colleague

That cycle repeated for the next few years at Red Swoosh. Kalanick would run out of money, then secure a last-minute deal with a larger tech company and keep his business alive for another few months. He’d then find a way to parlay that deal into yet another venture investment, bailing out the company for a year or so longer.

Red Swoosh ran out of money, and one of its employees was committing tax fraud. Travis kept negotiating deals and investment rounds to keep the company alive. He kept the company alive for six years before he finally negotiated a sale:

Finally, after six years of tireless hustling, Kalanick negotiated his best deal yet: he sold Red Swoosh to Akamai for nearly $20 million. After taxes, Travis personally netted roughly $2 million.

I respect people that have gone through the trenches of building a company. Real experiences like these harden you for your next battle.

After the Red Swoosh sale, Travis didn’t exactly live a life of the rich and famous. He started a blog and advised young startups from his apartment in San Francisco’s Castro District.

For Kalanick, blogging was a way of marketing himself, along with the occasional talk delivered at startup mixers and cocktail parties.

Kalanick wanted to be a fixer like The Wolf. After buying his hilltop house in the Castro, Kalanick started investing small amounts of money in various startups with the understanding that he’d be available as their own personal fixer, willing to swoop in and solve problems whenever a founder needed his help. Got a problem with an agitated investor? The Wolf can handle it. Don’t know the first thing about hiring new engineers? Just call The Wolf. Maybe you have late-night thoughts on your company’s next move and want to talk it out. Never fear, The Wolf is here.

Travis is very relatable here. This is pretty much what I would do if I made it big - write an interesting blog, help young entrepreneurs make it, and earn a nickname like “The Wolf”!

When Travis became Uber CEO, he gave early employees autonomy and responsibility.

Kalanick trusted his employees with significant power. Each city’s general manager became a quasi-chief executive, given the autonomy to make significant financial decisions. Everyone was responsible for “owning” their position. Empowering his workers, Kalanick believed, was better than trying to micromanage every city.

that sense of freewheeling autonomy made employees lionize Kalanick’s leadership. It was as if Kalanick had hired a private army of mini-entrepreneurs and given them one mandate: Conquer.

After their time at Uber, Kalanick wanted his troops to go off and start their own companies. He saw the very act of founding a company as a virtue unto itself.

I can see how this city based management style helped Uber grow so quickly in so many cities. If all the important decisions for Uber’s growth were being made in San Francisco, there’s no way that Uber would have taken over the world as quickly.

I also think it’s no accident that the early Uber team is described as Travis’s “private army”. Travis would ask his soldiers to break local laws, stand up to its enemies, and play dirty to win. I’m not sure he would have succeeded if his employees thought they were cogs in the machine at a boring office job.

Uber’s stories of dirty tactics and ruthless competition are incredible

Uber started competing head to head with Lyft and Sidecar, and they held nothing back.

Graves, Geidt, and especially Kalanick weren’t above playing dirty. They started booking secret meetings with regulators in San Francisco and encouraging them to go after Lyft and Sidecar.

every race he [Travis] enters, in anything where he’s asked to compete against others, he seeks nothing less than utter domination.

Kalanick enjoyed the fight. At first he began to needle John Zimmer, Lyft’s co-founder, on Twitter. In playful jabs, he would troll Zimmer by asking about Lyft’s insurance policies, business practices, and other seemingly esoteric shoptalk. Then he would start picking Zimmer and Lyft’s business apart.

Kalanick didn’t really want to buy Lyft, anyway. He wasn’t into Zimmer. Something about the Lyft president’s personality irked Kalanick. The Uber CEO didn’t want to work alongside Zimmer. He wanted to professionally humiliate him.

Kalanick took pleasure in hurting Green and Zimmer, and showed them no mercy.

It is surprising for me to learn that Travis treated his competitors like sworn enemies. In my day job, I’ve seen my coworkers disrespect and belittle competitors, but I have never seen company leaders go after a competitor publically and with this level of intensity.

Uber also hired spies to gather intelligence on Lyft executives and on driver initiatives.

One Lyft executive grew so paranoid about being followed by Uber that he walked out onto his porch, lifted both middle fingers in the air and waved them around, sending a message to the spies he was absolutely sure were watching.

Another entity, the Strategic Services Group, the SSG for short, employed the most clandestine tactics of the bunch. It was made up of ex-CIA, Secret Service, and FBI operatives, and hired subcontractors on special anonymous contracts with Uber so that their names couldn’t be traced back to the company. This outfit of black-hat spies engaged in a wide range of activities, some of which eventually spun out of Uber’s control.

Undercover operations could include impersonating Uber drivers to gain access to closed WhatsApp group chats, hoping to gather intelligence on whether drivers were organizing or planning to strike against Uber.

The budgets for these black ops departments were obfuscated; Kalanick had purview over them. Nevertheless, Kalanick okayed budgets that spun into the tens of millions for surveillance activity, global operations, and information collection […] Buying intelligence from third-party firms to gain an edge was normal.

I didn’t think that companies hired professional spies to follow competitors and impersonate drivers. I thought I could come up with creative ideas for a company to beat its competitors. Uber’s tactics make me feel like my “creative ideas” are child’s play.

Real people went through serious and tragic events as Uber grew in cities

Often, as white collar tech employees, we forget that our work affects real people. In Uber’s case, the app sometimes affected drivers, employees and customers in extreme, emotional and hard hitting ways.

As general manager of Uber’s Milan office, she worked overtime to convince Uber drivers to stay on the road, even as taxi operators would hail Ubers to their location, pull drivers out of their cars, and beat them. And one night when Lucini was returning home from work, she found a sign hung from a power line not far from her apartment. On the sign was Lucini’s home address, and a message calling her a prostitute who provided her “services” to Milan’s transportation chief.

Taxi cartels in areas like Las Vegas and elsewhere had deep ties to organized crime, which meant serious and sometimes violent retaliation. Cars were stolen. Sometimes taxi owners would assault drivers and set their cars aflame.

Cabbies were aghast. Doug Schifter, a livery driver from Manhattan, faced financial ruin after the rise of Uber wrecked his income driving for traditional car services. Schifter drove to City Hall in Lower Manhattan on a cold Monday morning in February 2018, put a shotgun to his head, and pulled the trigger. “When the industry started in 1981, I averaged 40–50 hours,” Schifter wrote in a final post to his Facebook page. “I cannot survive any longer with working 120 hours! I am not a Slave and I refuse to be one.”

Wow - employees are being intimidated by tough guys, and drivers are taking their own lives as a result of pricing changes. No company wants its work to lead to events like these.

Uber faced unfathomable challenges in the international market

In China, Uber wanted to win in the market at all costs. Chinese customers responded to Uber’s incentives in a way that was hard to predict:

he [Travis] noted that in just nine months, the number of trips taken in Chengdu and Hangzhou were more than four hundred times the number of trips taken in New York, one of Uber’s largest cities, when those markets were the same age. What Kalanick left out was the fact that in many cities, more than half of those trips were fake, a complete waste of money brought in by investors.

But in China, drivers and riders colluded to scam Uber out of billions in incentives, divvying the rewards … drivers seeking a fake ride would ask for “an injection,” a reference to the small, red digital pin that signaled a user’s location inside the Uber app. A “nurse,” or scammer, could respond in kind to give a “shot” to the original poster by creating a new fake account and going on a fake ride with the driver. The two parties would then split the bonus incentive payment from Uber. Repeated over and over across dozens of cities, small driver bonuses mushroomed into millions in squandered cash … scammer would request rides from his “passenger” phones, and use his “driver” phones to accept those rides. He would then drive around the streets of Chengdu with dozens of phones spread across the front and back seats of his car, racking up fares for each of the “trips” he was completing for his fake customers.

Some scammers created giant makeshift circuit boards filled with hundreds of slots to insert SIM cards, the small microchips that allow mobile phones to communicate with a cellular network. Each SIM card in the circuit board acted as a new number that could automatically respond to a verification text for a newly created account, which the scammers then used to rack up more fake rides and bonuses.”

The ingenuity of these scammers is very surprising to me. They built custom circuit boards with hundreds of SIM card slots so that they could take fake rides on Uber and earn incentive bonsues.

Their Chinese competitor, DiDi, went even further - they planted engineers inside Uber as corporate spies.

One of DiDi’s preferred tactics was to send new recruits over to Uber to join as engineers. As soon as they were hired they acted as moles, feeding proprietary Uber information back to DiDi and carrying out corporate sabotage on some of Uber’s internal systems.

When I heard that corporate espionage was involved, it became clear to me that the conflict between Uber and DiDi was less like competition and more like war.

In India, Local taxi companies hated Uber and initimidated employees with violence.

In Mumbai, local taxi operators muscled up at Uber’s offices and tried to intimidate employees. Violence was not uncommon; in Bangalore, whenever BD took a ride home from work, he refused to let Uber drivers take him directly to his house; he knew competitors might follow him.

An angry mob of drivers—some who drove for Uber, others employed by taxi organizations all too happy to stoke anger—showed up outside of Uber’s offices in early 2017 with the dead body of the thirty-four-year-old driver, M Kondaiah, dumping the corpse on the company’s front doorstep.

Drivers and passengers in India and Brazil experienced horrifying, violent crime.

The man took out a canister, doused his body in gasoline and then brandished a lighter, threatening to set himself ablaze unless Uber raised its rates again.

Yadav switched off his cell phone, making the two untraceable to police or Uber headquarters. He found a secluded area, parked the car, climbed in the back seat and raped the young woman.

For six weeks, Uber employees in India even brought their parents and families into hotels with them; taxi officials were beating up Uber employees in the street.

[In Brazil,] Osvaldo Luis Modolo Filho, a fifty-two-year-old driver, was murdered by a teenage couple who hailed a ride using a fake name and chose to pay in cash. After stabbing Modolo repeatedly with a pair of blue-handled kitchen knives, the couple took off in Modolo’s black SUV, leaving him in the middle of the street.

At least sixteen drivers were murdered in Brazil before Kalanick’s product team improved identity verification and security in the app.

Nobody expects their white-collar office job to lead to violent crime like suicide, rape and murder. Despite knowing that such events happen every day in cities around the world, my heart becomes heavy upon learning of such human suffering.

The power struggle between the Uber CEO and the board gives a glimpse into high stakes corporate politicking

As Uber started to get bad press, Bill Gurley thought that Uber needed to make major changes, and replace Travis with another CEO.

It [the Eric Holder report] was hundreds of pages long, a winding, repetitive list of infractions that had occurred across Uber’s hundreds of global offices, including sexual assault and physical violence. After Ryan Graves read the report, he felt he needed to vomit.

Gurley and Bonderman worried Holder’s recommendations might not have gone far enough, since they stopped short of pushing Kalanick out for good. Some members of the executive leadership team were convinced Huffington had leaned on Holder and Tammy Albarrán, Holder’s partner, to convince them not to recommend Kalanick’s termination in the final report.

Travis Kalanick didn’t trust VCs and gave himself supervoting shares to keep his power.

Whenever Camp, Graves, or anyone else in Kalanick’s orbit began to chafe at his actions, he usually responded with some version of the same placative sentiment: “Do you know how much money I’m going to make you?”

But when Sacca began attempting to buy up shares of Uber from other early investors—a practice known as “secondary share purchasing”—Kalanick turned on him. The CEO stopped allowing Sacca to attend board meetings as an observer; the two rarely spoke afterwards.

Travis could be so confrontational with other executives and board members at Uber. I haven’t seen conflicts with this kind of intensity in people’s workplace relationships.

To oust Travis, Bill Gurley hatched a secret, elaborate plan with a syndicate of Uber’s VCs.

And he [Bill Gurley] proposed a plan that would rely on all of them [Uber’s VCs] to work together. Gurley knew Kalanick would never step down of his own volition. They had to force his hand. Gurley knew Kalanick, like a rock climber looking for a toehold, would search for any weakness in the syndicate’s attack. With enough time and effort, Kalanick would find one, exploit it, and sink them all. He was a survivor; they needed to box him in. The minute Gurley walked into the hotel room to negotiate a surrender, Travis would tell the six-foot-nine Texan to go fuck himself. They needed a neutral emissary.

They worked through every possible permutation of what could happen when Cohler and Fenton approached Kalanick. (Would Travis throw a fit? Would he accede immediately? Would he lunge over the table and murder them?) They typed up a dozen different versions of the letter they would deliver to Kalanick, one for each possible scenario of the coming showdown. Lawyers at Paul, Weiss—the venerable white-shoe law firm—vetted each draft.

An Uber executive recalled that Kalanick said he was “ready to take Uber’s valuation to zero” before he would ever leave the helm.

Now, as Cohler and Fenton slid his death letter across the table, Kalanick wondered how long they had been plotting his demise. It was the ultimate betrayal. Kalanick was lashing out like a cornered animal. He wouldn’t take it lying down. He wouldn’t accede to their demands. He was going to fight.

Bill planned for every possible response from Travis and forced his hand. It was crazy for me to learn that a company’s investors were bitterly fighting its CEO.

Travis agreed to resign but he probably didn’t see what was coming next.

The story hit the web at just after 1:30 a.m. Eastern Time, as a push notification from the New York Times smartphone app was sent out to the home screens of hundreds of thousands of subscribers simultaneously. “Travis Kalanick resigned as chief executive of Uber after investors began revolting over legal and workplace scandals at the company,” it said. […] Back in Woodside, the members of the syndicate were all in shock. Someone had leaked the entire story to the Times. In the end, all they wanted was Kalanick’s resignation, not his embarrassment. Somehow, in the scrum of the past forty-eight hours, things had gone sideways.

Seeing the tick-tock of events on page A1, followed by an enormous graphic of Kalanick’s face shattered—like pieces of glass—across the front of the business section, was too much for him to bear. He was livid; the venture capitalists screwed him, like he always suspected they would. They made a fool out of him in front of the entire world.

Someone leaked the story to the press, and Travis’s reputation was destroyed in the press. I thought this was a real stab in the back for Travis.

And when Uber began searching for a replacement CEO, both Travis and the VC firm Benchmark pulled out all the stops to influence the decision.

She [Meg Whitman] said that an Uber under Meg Whitman meant the end of Travis Kalanick—music to Benchmark’s ears. One of Kalanick’s allies, who knew of Whitman’s hard-line stance against him, planted the leak with the press to smoke Whitman out.

the firm [Benchmark] filed a lawsuit against Travis Kalanick accusing him of defrauding Uber’s shareholders and breaching his fiduciary duty, a stunning act of open warfare between board members at a high-profile company.

Gurley’s idea in suing the former CEO was to invalidate Kalanick’s rights to those board seats entirely.

The votes [for the new CEO] kept coming in deadlocked; neither side would budge. Then, some believe, Cohler made a miscalculation. The Benchmark partner gave the table an ultimatum: If the board voted for Whitman, Benchmark would drop its lawsuit against Kalanick. It read to the room as an ultimatum. This was the price of peace. Instead of following a fair process to determine the best candidate, Benchmark was effectively holding the board hostage to approve the [replacement CEO] candidate of their choice.

Again, I couldn’t imagine that an investor would file a lawsuit against the CEO that brought them all its returns. And I didn’t think that an investor would pressure the board to vote for a specific replacement CEO, and have it backfire on them.

The book ends with Travis’s departure from Uber, and the entire story arc is gripping, especially if you’re a frequent Uber customer like I am.

Your Turn

Pick up the book and tell me what you thought of it, and of this book review.

Showtime made a TV series out of this book. I haven’t seen it yet. I’ll update this post when I do. If it’s anything like the book, it will be a treat.

Thanks to Kike Ibe for helping me write this post.

You are an engineer in a 100+ person organization. Often, when you show up to work, you’re asked to build a new system to replace an old system. Maybe there’s some code that’s buggy or slow that needs to be swapped out. Maybe you’re switching a piece of cloud infrastructure - you’re going to run CI on Buildkite instead of CircleCI, or log metrics using Datadog instead of Cloudwatch. Or maybe you’re swapping out business process A for producing widgets with business process B.

Depending on the approach that you take, the final result can end up looking pretty different.

Result A: You build the new system, and you make it available to use in production

You test that the new code produces correct results. You integrate Buildkite and Datadog so that they work in production. You implement business process B so customers can start using it.

This is the bare minimum work that you have to do. If you stop here, your work is yet to add any business value. Someone has to actually start using the new system you built before the company realizes any of the benefits.

Result B: Result A + proof that the new system is better than the old system

Typically, you want to show that the new system you built is actually better than the thing it’s replacing. Sometimes this is simple - you check that the bug or performance problem is fixed. Often, it requires you to get some customers to start using the new system, and gather feedback to assess if things have gotten better.

If you skip this step, you risk making things worse. A few times in the life of a product, you might get away with shipping the new system and hoping for the best. But as your product matures, a rigorous evaluation of the old vs new system is the only way to guarantee that your work is having the intended effect.

Result C: Result B + you swap out frequent occurrences of the old system with the new system

Now that you know that the new system works, you start using it in the core parts of the product. You fix the bug in the most common codepaths. You ship the new cloud infrastructure to a majority of engineers. Cusotmers use business process B more often than business process A.

It is important that you adopt the new system in core areas of the product, and not in simpler toy use cases. This is not always easy, because the core areas of the business are more complex. But you get more value this way, and you ensure that your new system is a more complete replacement for the old system.

Result D: Result C + you eliminate the old system entirely in favor of the new system

If you’re a conscientious engineer, this is the holy grail for new systems you’re building. You switch to using the new system everywhere in your product. People that solve the same problem in the future will use the new system and get its benefits automatically. You can stop maintaining the old system, which means savings on cloud bills, but more importantly on engineering time, and the awkwardness of explaining why there are multiple ways to solve the same problem.

How you should use this information

As an engineer, you can use this framework to figure out what your project will and won’t achieve. You should aim to achieve Result D whenever possible, because it leads to a simpler architecture for the next person, or your future self. That being said, you might also encounter situations in which eliminating every last use of the old system is not desirable. For example, if you’ve decided to start storing data in SQL tables instead of storing them in a document store, it might not make sense to edit parts of the business that are infrequently used or infrequently edited. Data migrations are usually not simple, and the juice may not be worth the squeeze for the non-critical parts of the business.

As an engineering leader, you ought to establish a culture where engineers are rewarded for achieving maximum adoption for the new systems they build. Recognize teams for driving adoption of their features rather than leaving it as an afterthought. Encourage engineers to make cross cutting changes confidently by building tools for static analysis and mass refactoring tools. Encourage your teams to build playbooks for running experiments and measuring the impact of new features that they launch. The future of your organization depends on making such changes effectively, and unless you’re a ladder climbing politician, you ought to make life easier for your future self.

We’ve started using Google BigQuery extensively at Cruise as a data warehouse. The syntax for querying arrays in BigQuery isn’t obvious and this post will explain how it works.

Consider a table that stores a company and its executives with the following schema:

companies
- name
- founding_year
- executives
    * name
    * title

with some sample data

WITH companies AS ( 
   SELECT 
      "Apple" as name, 1976 as founding_year,
      [
        (SELECT AS STRUCT 'Tim Cook' name, 'CEO' title),
        (SELECT AS STRUCT 'Jony Ive' name, 'Chief Design Officer' title),
        (SELECT AS STRUCT 'Jeff Williams' name, 'COO' title)

      ] as executives UNION ALL SELECT
      
      "Amazon" as name, 1994 as founding_year,
      [
        (SELECT AS STRUCT 'Jeff Bezos' name, 'CEO' title),
        (SELECT AS STRUCT 'Brian T. Olsavsky' name, 'CFO' title)
      ] as executives UNION ALL SELECT
      
      "Twitter" as name, 2006 as founding_year,
      [
        (SELECT AS STRUCT 'Jack Dorsey' name, 'CEO' title),
        (SELECT AS STRUCT 'Ned Segal' name, 'CFO' title)
      ] as executives UNION ALL SELECT
     
      "AirBNB" as name, 2008 as founding_year,
      [
        (SELECT AS STRUCT 'Brian Chesky' name, 'CEO' title),
        (SELECT AS STRUCT 'Joe Gebbia' name, 'CPO' title)
      ] as executives UNION ALL SELECT
      
      "Square" as name, 2009 as founding_year,
      [
        (SELECT AS STRUCT 'Jack Dorsey' name, 'CEO' title)
      ] as executives
      
) SELECT * FROM companies;

We can use familiar SQL queries to query top level fields like name or founding_year

SELECT * FROM companies WHERE founding_year > 2000;

To query nested fields like executives.name or executives.title, we use a combination of CROSS JOIN and UNNEST.

For example, this query returns all companies with Jack Dorsey as an executive:

SELECT * FROM companies CROSS JOIN UNNEST(executives) as executive WHERE executive.name = 'Jack Dorsey';

To understand how this query works, notice that CROSS JOIN and UNNEST return the row with the flat value of each item in the array.

For example, consider this query

SELECT c.*, executive FROM companies c CROSS JOIN UNNEST(c.executives) as executive

This returns a row for each combination of company and executive.

Once, we have this table, it’s easy to see how WHERE clauses work like we expect on the executive.name and executive.title fields.

Given the results above - what would this query return?

SELECT c.name, executive.name as ceo FROM companies c CROSS JOIN UNNEST(c.executives) as executive WHERE executive.title='CEO'

If you guessed right, it returns the name and CEO of each company in our table.

Tip: You can use a comma in place of CROSS JOIN in your queries for brevity.

SELECT c.* FROM companies c CROSS JOIN UNNEST(executives) as executive WHERE executive.name = 'Jack Dorsey';

is equivalent to

SELECT c.* FROM companies c, UNNEST(executives) as executive WHERE executive.name = 'Jack Dorsey';

I hope this helps you explore datasets in BigQuery more easily.

BigQuery has worked well for us as a data warehouse. From engineers to data analysts and managers, I’ve seen users of all technical abilities have use BigQuery at Cruise in their day-to-day work to build a self-driving robotaxi.

I spent 5 years in the Computer Science BS-MS program. I was a peer mentor for CS111 for 3 years.

I’ve met a wide range of students in the CS department. I’ve helped students with assignments, and explained concepts to them over and over again. This blog post explains the patterns for success and failure that I have noticed.

Your performance in intro courses is incredibly important. Develop a good mental model of programming early in your coursework.

A good performance in the intro courses builds the foundation for your CS knowledge. Without it, you are as useful as a mathematician that cannot add fractions, or solve linear equations.

By the time you get through CS111, you should have a clear mental model of how problem solving is accomplished through code. It’s easy to get distracted as a 111 student by the complexities of Java, Eclipse, jEdit, your weekly assignments, and course project. This is especially true if you come to 111 with no programming experience. It might feel like all you’re doing is staying afloat week after week, assignment after assignment, milestone after milestone. Someone tells you that using Eclipse is good/bad. Someone tells you that you should put curly braces on the same line as a function, as opposed to putting it on its own line. It can be very overwhelming.

If at the end of 111, you are able to take a problem given to you, think about how it’s solved, write code to solve it, and debug your code until it works, you have done well. This seems like a hard thing to measure, so I’m going to give you a couple of litmus tests to gauge your performance.

Consider the problem below:

Write a program that prints the numbers from 1 to 100. But for multiples of three print Lemon instead of the number and for the multiples of five print Juice. For numbers which are multiples of both three and five print LemonJuice.

Here is what the output of the program looks like from 1 to 20.

1
2
Lemon
4
Juice
Lemon
7
8
Lemon
Juice
11
Lemon
13
14
LemonJuice
16
17
Lemon
19
Juice

This problem should be a breeze for you after 111. If you cannot write code to solve this in 30 minutes, without someone’s help, you are in trouble.

Here is another litmus test. If you can solve the first 5 problems on Project Euler without any help, you have learned problem solving. You can start worrying about other things. If you can’t do these problems, you need to spend more time learning to problem solve, writing code, and debugging code. You will not get very far in a CS degree before you do this.

You can write code to solve a Project Euler problem, and submit the solution by creating an account. This is also a great way to learn a new programming language (This is how I learned the basics of Python).

Data Structures is incredibly important.

Data Structures develops on your understanding of problem solving from 111 and teaches you about clever ways in which programmers manipulate data, and algorithms.

1. You need to know Data Structures to write C code (CS211 - Computer Architecture, and CS214 - Systems Programming).
2. You need to know Data Structures to do well in Algorithms (CS344).
3. You need to know Data Structures to get through a tech interview.

You can ask any upperclassman what they think the most important CS course is, and Data Structures will almost certainly be one of their top 3.

The Story of 50% of the students in CS112

Now, let me tell you the story of Alice. Alice is a metaphorical student that represents 50% of the CS112 roster.

Alice took CS111, did fairly well in it (got better than a B). In 111, Alice often found it hard to listen to the professor in lecture. Every once in a while, she zoned out, and couldn’t figure out what Tjang or Sesh was saying. She went home, looked at the weekly assignment, and finished it (perhaps with some help from a friend, TA, or someone at the iLabs). Since every concept in lecture was reinforced in a weekly assignment, she understood most of the material that was covered in 111.

When Alice took 112, she followed a similar path. Tjang/Sesh was lecturing about Heaps, Linked Lists, Graphs, Tail Recursion, Efficiency Analysis of Insertion Sort etc. She went on Facebook on her laptop/got a text from a roommate, subequently zoned out in class, and quickly lost track of what the professor was saying. Alice went home, there was a project due in two weeks. Alice worked very hard on the project. She definitely had some issues along the way, but she was able to ask Sesh/TAs/peers/people at the iLabs for help and get it done. She got an 85+ on all the 5 projects in CS112. Alice got around a 50 on both Data Structures exams, ended Data Structures with a C/C+, and moved on.

Alice is the canonical example of a student that does poorly in the Rutgers CS degree, and here’s why.

In 111, Alice had projects every week that forced her to learn material. The only thing in 112 that forces you to learn material is the exam, and that happens twice a semester. Sometimes, the first 112 exam is so early in the semester that there’s very little material covered on it. This means that the final covers a ton of material, and because Alice has been zoning out of 112 lecture on a regular basis, she does not know why Heapify is O(n), or why you would want to use a min heap as a frontier when implmenting Dijkstra’s shortest path algorithm.

In order to avoid Alice’s pitfalls, you need to make sure that you understand every concept in Data Structures. Practically every sentence that Sesh or Tjang says in 112 is important. I understand that you are human, and cannot keep perfect concentration in an 80 minute lecture. But you need to be proactive about understanding all the concepts that are covered. This means that if you zoned out in the lecture about AVL Trees, you still need to make sure you learn it. You can go read the textbook after class. You can look up the material online. There’s some very good online courses that cover data structures. I strongly recommend the lectures from Stanford’s Programming Abstractions course. You can ask upperclassmen to explain the concept to you.

You need to do this before the day of your data structures exam, because the course covers a lot of important material, and you cannot learn all of it in a day.

Pick classes carefully, especially after the intro sequence.

1. Pay close attention to the professor that’s teaching the class.
   • Don’t take a class just because the professor is easy. By doing this, you are learning nothing, and wasting both time and tuition dollars.
   • Understand that a lot of the professors teaching you haven’t written a line of code in 15+ years. They cannot and will not teach you web or mobile app development. If they attempt to teach you this stuff, their material will likely be outdated and poorly researched.
   • The theoretical classes are challenging and will require mental discipline. I highly recommend everything being taught by Martin Farach-Colton, William Steiger, or Eric Allender. These were my favorite professors. Their classes were engaging, memorable, and had challenging material.
2. Prerequisites don’t matter.
   • If you’ve done well in 111, 112, 211 and 205, you can take any other class in the department - even a graduate class.
   • Do you see Uli Kremer (an excellent compilers professor) teaching next semester? Take his class even though you haven’t taken 314.
   • But V! I’m not smart. Everyone in this class is going to have an edge on me. I’m going to get a bad grade, lose my scholarship, and end up on the streets. Here’s what I have to say to this - 50% of the people taking a class don’t learn a thing. At the end of CS112, I want you to ask 10 of your classmates if they can implement a hash table. At the beginning of 211 and 214, I want you to look around you as they struggle to implement one in C. They’re not learning in 112, and they’re not learning in the prerequisite courses that you want to skip. They will have no edge over you.
   • You will feel more challenged in a course if you take it without a prerequisite, but as long as you’re proactive about brushing up on material you don’t know, you’ll do just fine.
   • Logistics - you can’t register for a course on webreg if you haven’t taken the prereqs. Send the prof an email to get a prereq override - tell them you’re interested in the material, and you’ll put extra time and learn stuff you don’t know.
3. Stop focusing on grades. Focus on concepts you’re learning
   • The classes I loved the most were the ones in which I was engaged by the professor’s lectures and assignments.
   • I took Operating Systems with a smart but boring professor. I was not engaged in lecture, but his assignments, exams were easy so I got an A. I regret wasting my time, because my grade did not reflect my knowledge of OS.
   • I have friends that took OS with Ricardo Bianchinni. Their assignments and exams were much harder, and they spent much more time on coursework. Most of them did not get As - but they learned far more about OS than I did.
   • If your goal is to learn OS, don’t waste your time in a course that doesn’t teach you OS.
   • If your goal is to get a piece of paper with your name on it, and “Bachelors in Computer Science” under it, this advice isn’t for you. I loved learning about the hard problems in computer science from incredibly smart people that were experts in these fields.

If you are not branching out from the material you learn in class, you will have a hard time getting hired as a developer.

Your professors haven’t written code in 15+ years. They’re not going to teach you how to develop software - don’t waste your time taking CS431.

Start writing code in your free time to solve any problem that you have. I wrote RUBUS when I was in 112. I used to maintain static web pages for a board game my roommate played with his high school friends. Everyone that’s good at programming has put in lots of time into it, and you will need to do the same.

Talk to upperclassmen about ideas you have, and the things that they’ve built. Hang out in the CAVE, attend USACS events, and go to hackathons. All of this experience will add up, and help you land your first paid programming gig.

Get paid to write code as early as possible.

When I started taking CS classes, I didn’t understand why people got paid to sit at a computer and write for-loops. I thought the mysterious, legendary developers getting internships and part time jobs must know so much more about the web/mobile apps/algorithms. This is a classic case of impostor syndrome, and it creates a mental roadblock until you get paid to write code.

Use your summers for internships or part time work (not for folding clothes). For your first development job, I recommend Student System Administration at OSS, System Administration at LCSR (under Doug Motto), an internship at Too Much Media, or HackNY if you are lucky. You’ll want to do a more traditional tech internship at companies like Microsoft, Google or Etsy, but you’ll have an easier time getting these once you have some experience. If you hang around CS folks, you’ll hear about such opportunities frequently. Apply early and often - you’ll have an easier time in October than in April.

Participate in the community

The Rutgers CS community, while not perfect, has grown a great deal over the last few years. You’ve got access to events like HackRU, HackNY, PennApps, and HackTCNJ. You’ve got access to the CAVE, the Hackerspace, and the Makerspace. USACS, RuMAD and the Rutgers Hackathon Club are active, and full of smart people. People taking the same classes as you have gone on to amazing jobs, start companies, and sell companies. Start reading hacker news, and /r/programming.

Don’t forget to give back to the community. Teach CS111 recitation. Join the USACS board, and help plan events. Hang out at the CAVE and help underclassmen understand difficult concepts. Help the noobs out at hackathons.

Make friends out of your peers. Impossible looking homework assignments will become easier. You’ll spend a silly amount of time working on a CTF challenge, or writing a game. You’ll get one letter Github usernames together. After college, they’ll help you find jobs and offer you their couches.

Rutgers is a great place to study Computer Science, and I hope your time there will be as memorable as mine was.

If you’re like me, you’ve written loads of web apps, but you rarely set up SSL on them. SSL is a must for any production-grade web application, especially if you’re authenticating users or taking personal information from them. Otherwise all the contents of your HTTP requests are being sent in plaintext - user login info / passwords, cookies etc.

Usually, SSL certificates can cost lots of money (Verisign charges over $100 / month), and be annoying to setup. After paying for domains and hosting, this is the last thing you want to shell out money for. Thus, StartSSL Free is a very appealing product because it gives you a free SSL certificate valid for 1 year, that’s accepted in all major browsers. I’m using it to serve flipdclass.com over SSL.

Getting your Cerficates from StartSSL

• Start by Signing up for an Account at StartSSL.
• They’ll send you a verification code and install a client certificate into your browser.
• Backup this certificate to cloud storage and Sign in to StartSSL.
• Now, you’ll want to validate your domain. Go to the StartSSL Free Product Page > Validations Wizard > Domain Name Validation. Type in your domain name (vverma.net for example).
• StartSSL will give you a list of email addresses for verifying your domain. You can usually access one of these emails through your domain registrar if you aren’t already set up to receive emails at your domain.
• Verify your domain (sometimes it takes a few minutes), and click on Cerficates Wizard to start creating your very first cert.
• Pick Web Server SSL/TLS Certificate > Enter a Password. Note the password that you enter - you will need it later to encrypt your private key.
• This will give you a private key in a text box. Copy this into a text editor and save the file as domain_key.enc.
• Click Continue > Continue > Add a Subdomain to your key. If you’re making this for the top level domain (http://vverma.net), I would recommend adding www as the sub domain to the certificate. You’ll need to create a certificate for each subdomain that you want to access over HTTPS, unless you get a wildcard SSL cert (not available through StartSSL Free).
• On the next page, you’ll see a certificate in a text box. Copy this into a text editor and save the file as domain.crt. I would also recommend saving the intermediate and root CA certs, because you’ll need them for your webserver setup.

Setting up your Web Server

I’ll walk you through setting up nginx or apache for SSL.

• At this point you’ll want to copy all the certificate files onto your server (domain_key.enc, domain.crt, ca.pem, sub.class1.server.ca.pem ), probably via scp.
• I’d recommend moving all certificate files to a directory like /etc/nginx/ssl/
• Now, you’ll want to unencrypt your private key domain_key.enc, so that it can be read by your web server. Without this, your webserver will prompt you for a password everytime it is restarted.

$ openssl rsa -in domain_key.enc -out domain.key 
$ chmod 400 domain.key # only root should be able to read this.

• Next, you’ll want to configure your webserver to respond to HTTPS requests with the certificate.

Apache Virtual Host

<VirtualHost _default_:443>
    ServerName domain.com

    SSLEngine On

    SSLCertificateFile /path/to/certs/domain.crt
    SSLCertificateKeyFile /path/to/certs/domain.key
    SSLCertificateChainFile /path/to/certs/sub.class1.server.ca.pem

    ...
</VirtualHost>

Nginx Virtual Host

Nginx does not have a directive for SSL Certificate Chains, so you will to concatenate your certificate to the intermediate and root CA certs.

$ cat domain.crt sub.class1.server.ca.pem ca.pem > domain.chained.crt

Then you can configure your virtual host as follows.

server {
    listen 443 default_server ssl;
    ssl_certificate /path/to/certs/domain.chained.crt;
    ssl_certificate_key /path/to/certs/domain.key;
    ...
}

Now, you can reload your webserver, and if you did everything correctly, you should get a successful HTTPS connection to your web app. Make sure that you test your site on a few different browsers, because not all browsers will behave the same way with SSL certificates.

You should also consider configuring your webserver to redirect all traffic to HTTPS, in order to prevent users from leaking their sensitive data by mistake.

SSH is a powerful protocol that lets you access machines remotely and run commands on them. Rutgers has a cluster of linux machines for CS students, and I often run programs on them. Sometimes, I leave a program running for a while, and forget which machine it was on. In this situation, PDSH comes in handy. It lets me run ps aux | grep -i <username> quickly across all the machines.

PDSH - Run SSH in Parallel

PDSH lets you run a command in parallel across a bunch of machines. I start by creating a text file with a list of machines I want to shell into:

cd.cs.rutgers.edu
cp.cs.rutgers.edu
grep.cs.rutgers.edu
kill.cs.rutgers.edu
less.cs.rutgers.edu
ls.cs.rutgers.edu
man.cs.rutgers.edu
pwd.cs.rutgers.edu
rm.cs.rutgers.edu
top.cs.rutgers.edu
vi.cs.rutgers.edu
cpp.cs.rutgers.edu
java.cs.rutgers.edu
perl.cs.rutgers.edu
basic.cs.rutgers.edu
assembly.cs.rutgers.edu
pascal.cs.rutgers.edu
php.cs.rutgers.edu
lisp.cs.rutgers.edu
prolog.cs.rutgers.edu
adapter.cs.rutgers.edu
command.cs.rutgers.edu
decorator.cs.rutgers.edu
facade.cs.rutgers.edu
flyweight.cs.rutgers.edu
mediator.cs.rutgers.edu
patterns.cs.rutgers.edu
singleton.cs.rutgers.edu
state.cs.rutgers.edu
template.cs.rutgers.edu
visitor.cs.rutgers.edu
builder.cs.rutgers.edu
composite.cs.rutgers.edu
design.cs.rutgers.edu
factory.cs.rutgers.edu
interpreter.cs.rutgers.edu
null.cs.rutgers.edu
prototype.cs.rutgers.edu
specification.cs.rutgers.edu
strategy.cs.rutgers.edu
utility.cs.rutgers.edu

Let’s say I save this file as machines.txt. I can then run a command in parallel across all these machines:

$ pdsh -R ssh -w ^machines "<command>"

Here are some things you can do with PDSH that you might find useful

Find all python processes running on these machines.

$ pdsh -R ssh -w ^machines "ps aux | grep -i python" ```

Kill any processes being run by my user. (Super useful if you forget to log out of a lab machine.)

$ pdsh -R ssh -w ^machines “killall -u whoami” ```

Check a specific log file for errors. $ pdsh -R ssh -w ^machines "grep -i error /path/to/log"

It’s a handy UNIX tool to have in your arsenal when working with lots of machines. Clearly, I am only showing the usage of pdsh in the most basic way. Check out PDSH on Google Code for a more detailed description of everything PDSH can do.

Web Scraping is a super useful technique that lets you get data out of web pages that don’t have an API. I often scrape web pages to get structured data out of unstructured web pages, and Python is my language of choice for quick scripts.

BeautifulSoup - Why I don’t use it anymore

In the past, I used Beautiful Soup almost exclusively to do this kind of scraping. BeautifulSoup is a great library for web scraping - it has great docs, and it gets the job done most of the time. I’ve used it on lots of projects. However, I find that it doesn’t fit my workflow.

Let’s say I wanted to scrape some data off a web page. I usually inspect the element in the Chrome Dev Console, and guess at a selector that might give me the data I want. Perhaps I guess div.foo li a. I quickly check to see if this works by running this selector in the console $('div.foo li a'), and modify it if it doesn’t.

Even after using BeautifulSoup for a while, I find that I have to go back and read the docs to write code that scrapes this selector. I always forget how to select classes in BeautifulSoup’s find_all method. I don’t remember how to write a CSS attribute selector such as a[href=*foo*]. It doesn’t let me write code at the speed of thought.

lxml.cssselect

LXML is a robust library for parsing XML and HTML in Python that even BeautifulSoup is built on top of. I don’t know much about lxml, except that I can use CSS Selectors with it very easily, thanks to lxml.cssselect. Look at the example code below to see how easy this is.

import lxml.html
from lxml.cssselect import CSSSelector

# get some html
import requests

r = requests.get('http://url.to.website/')

# build the DOM Tree
tree = lxml.html.fromstring(r.text)

# print the parsed DOM Tree
print lxml.html.tostring(tree)

# construct a CSS Selector
sel = CSSSelector('div.foo li a')

# Apply the selector to the DOM tree.
results = sel(tree)
print results

# print the HTML for the first result.
match = results[0]
print lxml.html.tostring(match)

# get the href attribute of the first result
print match.get('href')

# print the text of the first result.
print match.text

# get the text out of all the results
data = [result.text for result in results]

As you can see, it’s really easy to use CSS Selectors with Python and lxml. Instead of spending time reading BeautifulSoup docs, spend time writing your application.

Installation of lxml and lxml.cssselect

LXML and CSSSelect are both Python packages that you can install easily via pip. In order to install lxml via pip you will need libxml2 and libxslt. On a standard Ubuntu installation, you can simply do

sudo apt-get install libxml2-dev libxslt1-dev
pip install lxml cssselect

Check out the lxml installation page and lxml.cssselect for more information.

Having used Linux almost exclusively for the last four years, I miss efficient window management on Macs. Coming from the awesome window manager, I find that OS X does not have good support for a two monitor multiple workspace workflow out of the box. After tinkering with third party software, I believe I’ve found a good solution for most of my complaints, and have a workflow that I feel productive with. In my experience, this works best with multiple monitors, a standard keyboard (think Dell not Apple), and a three button mouse (I’m not a fan of touchpads or Apple mice).

Setting up Multiple Desktops

I really like the use of multiple desktops in my workflow. I usually set up four desktops. I keep Spotify open on the very last one. The middle ones are my “work” desktops that I use for terminals, browsers, IDEs, and documentation. The first one is usually a “distraction workspace” that will have my email, and Adium open. This helps me keep my windows organized, and keep focus when I need to.

In order to set this up, I add additional desktops (up to 4). The easiest way to do this is to open up Mission Control (usually Control-Up), hover over the Desktops, and click the Plus button on the top right.

Once this is done, I would recommend setting up easy keybindings to switch between desktops. To do this, you go to System Preferences > Keyboard > Keyboard Shortcuts > Mission Control. Then you can set up keybindings for Move left a space, Move right a space, … Switch to Desktop 1-4. I use Ctrl-Alt-Left/Right to move between desktops, and use Command-1/2/3/4 to jump to a desktop.

Using Slate for Window Management

On OS X, it’s sometimes pretty cumbersome to perform window management tasks like moving windows between monitors, and maximizing windows efficiently. This is where Slate comes in. Slate is a configurable third-party window management application, that makes these window management tasks super easy. I will explain how I use Slate day-to-day.

Setting it up

You can install Slate, pretty simply by downloading the Slate dmg. After installing and starting Slate, you will want to make sure it’s properly configured.

Here is my ~/.slate.js configuration file that describes the keybindings I use with it. Right click on the Slate icon in the topbar > Relaunch and Load Config, to apply configuration changes.

//Save this in ~/.slate.js
//This configuration file came from http://vverma.net. 
//Credit to Gerard O'Neill, http://goneill.net for introducing me to Slate.

var left = {
    'x': 'screenOriginX',
    'y': 'screenOriginY',
    'width': 'screenSizeX/2',
    'height': 'screenSizeY',
};

var right = {
    'x': 'screenOriginX + screenSizeX/2',
    'y': 'screenOriginY',
    'width': 'screenSizeX/2',
    'height': 'screenSizeY',
};


//half screen.
slate.bind('right:ctrl,cmd', function(win) {
    var screen = slate.screen().rect();
    var win_rect = win.rect();

    // if we are at the edge of a screen on the right.
    if(Math.abs(screen.x + screen.width - win_rect.x - win_rect.width) < 5) {
        var curr_screen = slate.screen().id();
        slate.log(curr_screen);
        if(curr_screen < slate.screenCount() - 1) {
            var shift_screen = _.clone(left);
            shift_screen['screen'] = curr_screen + 1;

            win.doOperation(slate.operation('move', shift_screen));
        }
    } else {
        win.doOperation(slate.operation('move', right));
    }
});


//half screen.
slate.bind('left:ctrl,cmd', function(win) {
    var screen = slate.screen().rect();
    var win_rect = win.rect();

    // if we are at the edge of a screen on the left.
    if(screen.x == win_rect.x) {
        var curr_screen = slate.screen().id();
        if(curr_screen > 0) {
            var shift_screen = _.clone(right);
            shift_screen['screen'] = curr_screen - 1;

            win.doOperation(slate.operation('move', shift_screen));
        }
    } else {
        win.doOperation(slate.operation('move', left));
    }
});


//maximize
slate.bind('up:ctrl,cmd', function(win) {
    win.doOperation(slate.operation('move', {
        'x': 'screenOriginX',
        'y': 'screenOriginY',
        'width': 'screenSizeX',
        'height': 'screenSizeY',
    }));
});

//center
slate.bind('down:ctrl,cmd', function(win) {
    win.doOperation(slate.operation('move', {
        'x': 'screenOriginX + screenSizeX/4',
        'y': 'screenOriginY + screenSizeY/4',
        'width': 'screenSizeX/2',
        'height': 'screenSizeY/2',
    }));
});

Here is the mapping of keybindings

Cmd-Ctrl-Left - Split window in half vertically and move to the left. (Moves to the next screen if you are at the edge.)
Cmd-Ctrl-Right - Split window in half vertically and move to the right. (Moves to the next screen if you are at the edge.)

Cmd-Ctrl-Up - Maximize window.
Cmd-Ctrl-Down - Center window in its current screen.

Feel free to modify this slate configuration to suit your needs. You might find the Slate documentation helpful.

Miscellaneous

Adium

If you use Adium as a chat client on your machine, I recommend setting up a Global Keyboard Shortcut. This allows you to switch the focus to Adium anytime on your machine by pressing the key sequence. It’s super handy to instantly switch to Adium when you get an Adium notification.

I set my global keyboard shortcut to Cmd-Shift-/. To use this, you’ll have to get rid of the keybinding for the Help Center first. Do this by removing the keybinding in System Preferences > Keyboard > Keyboard Shortcuts > Help Center.

To set the global keyboard shortcut in Adium, go to Preferences > General > Global Shortcut .

Now, you can press Cmd-Shift-/ to switch to Adium, and press Cmd-/ to show/hide your buddy list.

Spotlight/Alfred

You should already be using Alfred as your primary application launcher. It lets you launch applications with your keyboard really easily, and do much more. It’s also way faster than Spotlight.

If you use multiple desktops, sometimes you’ll want to create multiple windows of the same application. Alfred will get in your way here, because if you try to launch an application that already has a window open, it will take you to the window instead of opening a new one. This happens to me all the time when I want to create a Chrome window, when you already have one open on another window.

The easiest way to do this is to go to an existing Chrome window, press Cmd-N to create another window, drag the newly created window, and while dragging the window, press Cmd-1/2/3/4 to take the window to another desktop.

Spaces

Like I mentioned before, I use Spaces in my development workflow. One thing that I really dislike about Spaces is that when you move between Spaces using Ctrl-Alt-Left/Right, it takes a second to animate the movement. I don’t like this because it feels clunky.

You can run this in a terminal to make this animation a lot faster.

defaults write com.apple.dock expose-animation-duration -int 0; killall Dock

Credit to Gerard O’Neill for showing me a lot of this workflow.

Rutgers Open Systems Solutions mirrors a bunch of Linux distributions, and you can use these to download packages quickly when you’re on campus. When downloading on the Rutgers campus, your bandwidth will also not be throttled which significantly improves your download speeds.

Adding a Mirror to Ubuntu

To add a mirror, open up your /etc/apt/sources.list file.

sudo [editor] /etc/apt/sources.list

It should look something like this:

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://us.archive.ubuntu.com/ubuntu/ [distrib] main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib] main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ [distrib] universe
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib] universe
deb http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://us.archive.ubuntu.com/ubuntu/ [distrib] multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib] multiverse
deb http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib]-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ [distrib]-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ [distrib]-backports main restricted universe multiverse
...

[distrib] here is the name of your distribution. This is one of natty, oneiric, precise, quantal, raring, or saucy. You can find a full list on the Ubuntu Wikipedia Page

You are going to add the following lines at the top of the file.

deb http://mirrors.rutgers.edu/ubuntu [distrib] main restricted universe multiverse
deb http://mirrors.rutgers.edu/ubuntu [distrib]-updates main restricted universe multiverse
deb http://mirrors.rutgers.edu/ubuntu [distrib]-backports main restricted universe multiverse
deb http://mirrors.rutgers.edu/ubuntu [distrib]-security main restricted universe multiverse

Be sure to replace [distrib] with the name of your distribution. Now, save the file and quit, and run

sudo apt-get update

You should now be downloading packages from the Rutgers mirror. Try to install a package, and make sure that you’re making requests to http://mirrors.rutgers.edu.

When building web applications, you sometimes want to retrieve JSON data from APIs and domains that are external to your service. Because of the Same Origin policy in browsers, you cannot retrieve data from other domains via AJAX.

JSON-P to the rescue

Usually to get around this, APIs will have endpoints that support JSON-P. JSON-P is a nifty technique that loads JSON data via <script> tags instead of loading the data via XMLHttpRequests (AJAX). To understand this, let’s look at an example.

Let’s say you have a service on http://myservice.com/data.json that returns the following JSON.

{
  "from": "myservice",
  "status": 200,
  "data": ['foo', 'bar', 'baz'],
}

An application that lives on http://anotherapplication.com cannot access data.json in client-side JS via AJAX because anotherapplication.com and myservice.com are not the same domain.

As the author of myservice.com, you can solve this problem by turning your JSON endpoint into a JSON-P endpoint. To do this you write myservice.com in such a way that hitting http://myservice.com/data.json?callback=procedureName returns the following:

procedureName({
  "from": "myservice",
  "status": 200,
  "data": ['foo', 'bar', 'baz'],
})

Now, the author of anotherapplication.com can load data.json by adding the following script tag dynamically to the client side DOM.

<script type="text/javascript" src="http://myservice.com/data.json?callback=procedureName">

Now, the function procedureName will get called with the data from data.json. Using this trick, does mean that you have to trust http://myservice.com, because any content returned by it can get executed by your client side JS.

Where YQL Comes in

Most web services will support JSON-P if they expect you to retrieve their data on the client side, but some do not.

For services that do not support JSON-P that live on the internet, you can use YQL to proxy the request through Yahoo’s servers, and retrieve data in the same way.

Here is a snippet of jQuery code that would normally hit http://myservice.com/data.json

$.ajax({
  'url': 'http://myservice.com/data.json',
  'dataType': 'json',
  'success': function(response) {
    console.log(response);
  },
});

Here is how you modify it to proxy via Yahoo’s servers.

var yql_url = 'https://query.yahooapis.com/v1/public/yql';
var url = 'http://myservice.com/data.json';

$.ajax({
  'url': yql_url,
  'data': {
    'q': 'SELECT * FROM json WHERE url="'+url+'"',
    'format': 'json',
    'jsonCompat': 'new',
  },
  'dataType': 'jsonp',
  'success': function(response) {
    console.log(response);
  },
});

The snippet above will send a request to Yahoo and get back data from myservice.com as a response. This does mean that http://myservice.com needs to live on the open web (not on an internal server), so that Yahoo servers can hit it.

jQuery will automatically add a callback parameter to the request, and give that name to the success function, so that it gets called appropriately.

Often times when building web applications, I used to spend time deploying my web applications via ssh and scp. Then I used Heroku for a few projects, and I really liked that deploying to heroku was as easy as it could be.

    git push heroku master

I wanted to have a similar deployment scheme on my own projects that aren’t deployed on Heroku.

How it works

Since git is a distributed version control system, you can push the code that lives on your machine to another machine very easily via ssh. So your first instinct is to set up a repo in the location that your code needs to be deployed, and push to it via git. This is a good instinct, but git does not allow you to push code to a working copy. To resolve this, you will create a bare repository on your server, and push to it. You will also set up a git hook to automatically deploy your application when code gets pushed to the bare repository.

Setting it up

Before you start, your codebase needs to be in a git repository. This could be a Github repository that you use for version control. I will assume that your codebase lives in one directory called project on your development machine, which I will refer to as develop.

This codebase will be deployed to your server. I will refer to your server as deploy.

Now, you are going to create a bare git repository on deploy, and you will be able to push to it from develop.

username@deploy:~$ mkdir repos # this is the dir where all your repos will be stored.
username@deploy:~$ cd repos
username@deploy:~/repos$ mkdir project.git 
username@deploy:~/repos$ cd project.git # You can replace this with the name of your project.
username@deploy:~/repos/project.git$ git init --bare
# Initialized empty Git repository in /home/username/repos/project.git

You will now set up your codebase on develop to push to the repos/project.git directory on deploy.

username@develop:~$ cd /path/to/my/project
username@develop:~/code/project$ git status 
# This must be a git repo.
username@develop:~/code/project$ git remote add deploy username@deploy:~/repos/project.git # This is the path to your bare repo.
username@develop:~/code/project$ git push deploy master 

This will push your codebase, to the bare repository you just created on deploy. You can verify this by cloning the bare repository if you’d like.

username@develop:~$ cd /tmp
username@develop:/tmp$ git clone username@deploy:~/repos/project.git
# Cloning into 'project'...
# remote: Counting objects: 666, done.
# remote: Compressing objects: 100% (417/417), done.
# remote: Total 666 (delta 255), reused 632 (delta 221)
# Receiving objects: 100% (666/666), 621.96 KiB | 462 KiB/s, done.
# Resolving deltas: 100% (255/255), done.
username@develop:/tmp$ cd project
username@develop:/tmp$ ls
# make sure your files are here.

Now that we are pushing to the repos/project.git directory on deploy. Let’s set up our repository to actually deploy its code. I’ll assume that your application gets deployed to /var/www/myproject.com .

username@deploy:~$ cd repos/project.git
username@deploy:~$ ls
# HEAD  branches  config  description  hooks  info  objects  refs
username@deploy:~$ cd hooks
username@deploy:~$ [editor] post-receive

The post-receive hook gets called by git right after code gets pushed to a repository (right after git push deploy master). We will make this hook deploy your application to /var/www/myproject.com . Using an editor of your choice, place the following in the post-receive file.

#!/bin/bash

### This file gets run when code is pushed to the project.git directory.

GIT_WORK_TREE=/var/www/myproject.com git checkout -f

Make the hook executable.

username@deploy:~/repos/project.git/hooks$ chmod +x post-receive

Make sure that your user has permissions to write to /var/www/myproject.com. This is it! You can now deploy your code anytime you want by running:

username@develop:~/code/project$ git push deploy master

Verify that your code is deployed when you push, and you should never need to use scp to deploy ever again.