Monday, March 21, 2016

There's No Test Like A Production Test

Dress Rehearsal


Enter stage left. Pacing a slow and deliberate gait to center stage, it's time to deliver your lines. Dress rehearsal is here. After weeks of memorizing lines and working with your follow cast members practicing scene by scene, you're nearly ready for opening night.

As the director walks everyone through each scene of dress rehearsal, small issues come to light that were never noticed before. Whether it's the wrong lighting arrangement, a misplaced prop, or a scene change that doesn't transition smoothly enough, this is the opportunity to make last-minute adjustments to make sure the show is prepared for success.


Opening Night


The cast and crew take their places as the curtain opens. Opening night is packed with eager patrons and critics ready to experience the well-promoted show. The opening act comes to a close to a thunderous applause. Midway through the second act a stage prop malfunctions, causing an awkward pause and a stumbling delivery, breaking the audience's immersion in an otherwise excellent show. The rest of the show goes off without major incident, ending to a satisfying ovation.

Despite all of the hard work and planning by everyone involved, one unexpected or anticipated error can mar an otherwise excellent performance. In a sufficiently complex production, no matter how much you prepare ahead of time, it's impossible to anticipate every possible thing that might go wrong. You must address any issues as they arise and learn to respond better next time

The Software Stage


The analogy is the same in the world of software. It does not matter how many layers of testing suites you wrap around your software. It does not matter that you've tested it in three distinct environments and subjected it to a wide array of contrived and random test data. It does not matter that you've gone over a test deployment twice and three times. You simply cannot know that your software is truly ready for production until it's put to the test with real users and data.

I've been around this game long enough to see well-tested systems run into all kinds of surprising bugs. Many bugs would simply never have been caught, even by adding dozens of programming, testing, and hardware resources prior to production release. These bugs are typically inconsistent, difficult to reproduce, or related to users exercising the software in unexpected ways.

It's a case of diminishing returns -- performing a variety of basic tests will generally pay immediate dividends as obvious, simple bugs are found. Performing additional rounds of testing will often flush out a few more. As more bugs are found, the rare gems, well-hidden beneath the surface, will become harder and more time consuming to find. Often, it is simpler to release the software and fix bugs as they are reported than to waste valuable resources looking for outliers.


Raise the Curtain


As with the theater, it is true in software: the show must go on. Once the production is prepared and the dress rehearsal has worked through the obvious problems, the only thing left to do is to perform for the waiting public and wait for the reviews.

If there's one thing I would suggest you take away from this, it's that there's rarely such a thing as a flawless release. Always expect that your software will fail in a way that you've never seen or anticipated. Therefore, be prepared to respond quickly, lest the critics give you a scathing review and the fickle public refuses to come back for an encore.

Cheers,

Joshua Ganes

Tuesday, December 01, 2015

Security Month - Episode 3: Social Engineering

End Of The Month


This past month has sure flown past for me. I intended to create one more security month entry, but life got in the way and it will have to wait for another day. Today's post is based on a presentation I gave at work a few years ago on a topic that is probably the most important of all security: the human component. It doesn't matter how good your locks and alarm systems are if you neglect to turn on the alert and lock the front door as you leave your house. Similarly, it doesn't matter if your system is designed properly if your users do not know how to protect the security of the system and their own credentials.

The Conundrum


When designing a secure system, you must walk a fine line between burdening your users with a barrage of security measures and keeping attackers at bay. Legitimate users should be able to access the system quickly, easily, and with minimal interruptions while at the same time keeping it "impossible|" to access the software without permission.

Typical applications rely on single-factor authentication (username / password). When authentication relies only on a secret key (the password), when a password becomes compromised, the system becomes compromised. The more users a system has, the more opportunities exist that at least one user will become compromised.

Social Engineering


It's often simpler to manipulate people than machines. While machines may be rigid in terms of interactions and enforcement of rules, humans tend to be more flexible -- especially when subjected to emotional influences. Social engineering is the process of "hacking" people to gain access to secure systems. This may involve tricking or convincing them to give up sensitive information such as passwords, or manipulating users into performing actions on the attacker's behalf.

Mandatory xkcd Reference

https://xkcd.com/538/


Pretexting


In order to manipulate users into making rash and emotional decisions, attackers may pose as authority figures and create plausible urgent scenarios where sensitive information needs to be divulged. This may include authentication credentials, or other details to be used in subsequent attacks. It's important to inform users to never give out their credentials to an unknown party regardless of urgency.

Phishing


In a phishing attack, a hacker will send a link to an illegitimate web site impersonating a legitimate page. A classic example is an email claiming to be from PayPal, and telling you that your account may have been compromised. It asks you to visit a link and enter your username and password. Ironically for those who are duped, the very attempt to save their credentials is what ultimately divulged them.

It is very simple to copy a legitimate site using copy/paste tools and modify it for devious purposes. Many people are not savvy enough to know to look for a legitimate secure (https) address. Teaching users not to "click the dancing bunnies" and how to verify a legitimate web page is a battle we are constantly waging.


Baiting


A rarer and more targeted attack vector is baiting. In this attack, a hacker will leave a physical device such as a USB thumb drive somewhere to be found. They may leave it in a company parking lot with an enticing label in the hopes that curiosity gets the better of their victim. Upon inserting the device into a secure system, the payload is delivered to either steal information or run a program or script to do something on the attacker's behalf. Users must learn never to connect hardware of suspicious origin to a secure system.


Quid Pro Quo



A hacker may attempt to butter up users by giving them something, either physical or performing a service, in return for sensitive information. While this may come in the form of a bribe, it may be as simple as establishing a friendly relationship over several days and requesting a simple favor.

Perhaps the most devious attempt I've heard described is to call random employees at a large company posing as "technical support". Many non-technical people will accept support on their latest computer problem and can be tricked into entering commands on the attacker's behalf.

All Good Things...


If there's one lesson to be learned from this post, it's to always be on your guard. "Trust no one" is bad advice to live by, but essential advice when it comes to sensitive credentials. Computers do not have human judgement and will gladly accept validated credentials from the shadiest of characters. As is demonstrated so well by social engineering, humans and machines do not behave the same way and your interactions with them ought not follow the same rules.

That brings us to the end of Amish Programmer security month. I hope you learned something new, or at least refreshed yourself.

Here's wishing you a merry Christmas and a happy New Year. I hope you return next year for some non-security topics here at Amish Programmer.

Cheers,

Joshua Ganes

Monday, November 02, 2015

Security Month - Episode 2: Storing Passwords

Trust No One


Two weeks after being promoted from Junior to Intermediate Programmer at Paranoid Software Ltd. you've been tasked with implementing a secure sign-in system for the company's new up-and-coming website. Due to "government conspiracies" and myriad other reasons, your tinfoil-hat-wearing boss has mandated that you use no third-party libraries for passwords and authentication. Only his most trusted and loyal employees are qualified to create something so important and sensitive. He'll be reviewing your work at a later time in his secure bunker...

Not being allowed to research industry best practices on this subject via the corporate internet connection (again, "conspiracies"), you sit down to draw up designs for a new database table as follows:

UserNamePassword
WayneGretzkyGr8One99
SidneyCrosbySidTheKid87

Each user will sign using a unique user name and corresponding password. When the user submits the sign in form, the submitted credentials will be compared against the database values found in the table and access will be granted or denied accordingly. This design allows for passwords to be verified while remaining flexible enough to allow for additional new columns for features such as temporary passwords, expiration, and more.

Plain Text Woes


You set to work building a new sign in page based on this design. Everything is coming along fine until forgetful Tom drops by your desk after your coffee break. "Looking for something, Tom?", you ask casually. "Maybe...", says Tom, "... I can't remember what I'm looking for, but that's not the only reason I'm here..." Tom scratches his head for a moment before his face changes in a moment of sudden recollection. He begins to explain how he can only remember one password for everything, and he's afraid of what certain other employees might do if they copied his password out of the new database table. They could potentially use it to access his account on another system.

Tom's a nice enough guy with some serious clout and connections around the office. You don't want to upset him, so you revise your design as follows:

UserNameEncryptedPassword
WayneGretzkyTe8Bar99
SidneyCrosbyFvqGurXvq87

In this revised design, the passwords are stored encrypted using the "ultra secure" ROT-13 algorithm. When a user's sign in credentials are submitted, the database value is decrypted and compared against the submitted password to authenticate the user and grant or deny access to the system.

Clever Estelle


As your project nears completion it needs to pass a review from Estelle, the company's oldest programmer. Estelle got her start in the industry working with VAX machines in the '70s. Nobody knows the details of her top-secret projects, but her office is strewn with archaic servers and mainframes cobbled together with assorted wires and duct tape. She left a barely-legible handwritten note on your desk asking you to stop by for a review of your project.

A light blue haze of smoke oozes out from under the door as you approach and knock briskly. You hear a loud cough before Estelle's unusually deep and raspy voice beckons you to enter. As you enter, you notice the yellow-brown stained walls of her private office. Distracted, you trip over an old server, yanking loose a bundle of wires. "Don't worry", Estelle reassures you, "that machine deserves it!"

Estelle likes your design, but due to her advanced technical savvy, she's figured out how to decrypt the passwords stored in your database and suggests that you use one-way hashes instead. She hands you a ream of fan-fold paper retrieved from an old dot-matrix printer. "Here's some materials I've prepared for you," see grunts as she urges you towards the exit.

After reading her notes about a wide variety of one-way hash algorithms, you once again draft up a design for your database table:

UserNameHashedPassword
WayneGretzky$2a$12$/Ox24ETwuRKwqCASxD40x.O
7Y1HOuaLzVp7nwf6FwUQjnRGnJok0m
SidneyCrosby$2a$12$o7chUYiQh0htw463/m5df.o
lmVQzkIjINY9CyyQmtC7xHV53OxtEC

This time there is a subtle change to the password verification logic. Instead of decrypting the stored password, the submitted password is passed through a one-way hash (in this case bcrypt with 12 iterations) and the hashed values are compared to see if they match.

A good hash algorithm is easy to calculate, but nearly impossible to reverse. They also distribute hashes effectively to avoid collisions (two similar strings hashing to the same value), which makes them statistically secure for use in verifying that you have a matching password.

Tom's Back!


Having passed Estelle's second review, your project is mere days away from being installed. You're enjoying a well-earned break in the lunch room when Tom pokes his head in looking confused. "I'm looking for someone," says Tom, "but I can't quite remember who... Oh yeah, it's you!" You notice that Tom is holding the same dot-matrix print-out about hashes in his hand. He explains that he's found a way to break your password system. He's created a script to generate every string up to an arbitrary length, calculate its corresponding bcrypt hash, and store those in a new indexed database. With this upfront work complete, he can use the newly indexed table to perform a reverse password look-up from the hash of any password short enough to be covered by the script. He calls this generated table a "rainbow table". This could potentially be used to reveal thousands of user's passwords.

After racking your brain late into the evening, you modify your design one last time:

UserNameRandomSaltHashedSaltedPassword
WayneGretzkyPQw#{hC*bDvyHnhVeOGaVOnG
dcsiAYETeNBbannZbdtu?kLN
dDXjbfDe?slovDjJfw$m2uGf
ZRneg-7bDJeeVwSwDGEsonI4
Cini1p*zuJ&HwJRgC 3grzzU
c!EKxrSt
$2a$12$tasiR59CA7uefis0.
4r4CuTjo2gD7MkP3Mzzn77yr
/QWuRbV8kyf.
SidneyCrosby;ZjmYbJeNMstnqu\q|gbLpfs
zb@6PVyOoglbBuLUb-Hulgau
nwCR.qvCUikOEkEQroqi8lRY
J4gVA{VwlWwh0nyQnZG5JYCm
k.cStNNyy|hzgiIqPjSdTlkh
zxaZIi_C
$2a$12$o3aiDUMmSDqX9/fpA
SJ/VOB483GsYvtfESeZbrON.
Gz080gVtlBqW

In this final incarnation, each password gets assigned its own unique random "salt" value. This value is first concatenated with the password. This "salted" password value is then run through the hashing algorithm to populate the database. When the a user submits a password at sign in, this process is repeated with the submitted password. It is first concatenated ("salted") with the stored random salt, then hashed. The final result is compared the hash stored in the database to grant or deny access.

Your tinfoil-hat-wearing boss emerges from his bunker to shake your hand for a job well done and asks you if you wouldn't mind helping him recover his password...


Transition


The best way to secure your users' passwords is to protect them in such a way that even a programmer with full knowledge and access to the system cannot get at them. A corollary of this is that if you ever submit a lost password request to a website that returns your password in plain text, you know they're doing something wrong.

Several years ago I came across a collection of sites with multiple sign in systems that were at various stages of progression from the story above. Some were storing passwords in plain text, some were storing them using different forms of simple encryption, and others were hashing the passwords (using varying strengths of hashing algorithms) without salting them first. It was my job to bring these all up to snuff without breaking live systems with active users.

The plain-text passwords were the simplest. I already had the passwords -- I salted and hashed them and moved along.

The encrypted passwords were almost as simple. I decrypted the passwords to their plain text form, then treated them as in the plain-text case above.

The hashed passwords were the most challenging. I created an extra database column to list the hash algorithm and imported the hash values into the database with no salt value. When one of these users signed in and was authenticated, I could then use their submitted password value to populate a strong salted hash value on their very first visit under my new system.

There were also some accounts that were not so active. We conveniently had a 90-day password expiration policy. After 90 days, I ran a script to reset all of the remaining passwords to new temporary values and purge our system of all insecurely stored user credentials. Any users who had not signed in within 90 days would have to use our password recovery system to regain access to their accounts.

Take It Away


If there's one thing you should take away from this episode it's that you should do your best to store passwords such that they cannot even be recovered by someone with inside knowledge.

If you know of any code that stores passwords without securely salting and hashing them, please do your best to correct the situation. If that code belongs to you, please go fix it. If not, please write a friendly but firm reminder to get those passwords under control and securely stored.

Cheers,

Joshua Ganes

P.S. In case you missed the joke, ROT-13 just rotates the letters through the alphabet by 13 characters. Please don't use this for any form of security.

Tuesday, October 27, 2015

Security Month - Episode 1: Injection

Starting With A Bang


Injection attacks, in all their various flavors, are now the number one most exploited security flaw in software. Their impact can range from something as innocuous as displaying extra text on a web page to the full compromise of your host machine.

Setting A Bad Example


Do you see anything wrong with the following snippet of PHP code?
(Note: this is a simplified example for demonstration purposes only)

// Initializes necessary variables, functions, etc.
require_once( "important.libs.php" );


$accessLevel = GetAccessLevel();

<snip>

// Defaults to empty string if not present
$name = Posted( "name" );

// Find contacts with matching first name in database
$sql = "SELECT * FROM Contacts WHERE ".

       "FirstName = '$name' AND ".
       "AccessLevel <= $accessLevel";
$result = $db->query( $sql );
// We should really do some error checking here

// Print details for each match found
while( $row = $result->fetch_assoc() ) {

    $firstName = $row[ "FirstName" ];
    $lastName = $row[ "LastName" ];
    $phone = $row[ "Phone" ];

    print "Name: $firstName $lastName, Phone: $phone<br />";

}

<snip> 


As you might have guessed from the title of this article, this code is vulnerable to injection. In fact, it is subject to two common forms of injection: SQL injection, and cross-site scripting (XSS).

What Is Injection?


Injection happens every time we receive input from an untrusted source and submit it to an interpreter without first sanitizing the input. In the case of SQL injection, that interpreter is the database interpreting the text of our SQL query. In the case of cross-site scripting, the interpreter is the browser attempting to display our HTML page. Whatever the scenario, allowing potentially dangerous user input to be interpreted as a trusted command or request can have disastrous .consequences.

What Could Go Wrong?


In the scenario above, imagine that a hacker wants to get a list of all contacts in the database. They are currently restricted only to those contacts whose AccessLevel value is lower than a configured value. How could they use SQL Injection to gain access to everything?

What if, in the form that requests this page, they entered "' OR 1 --" as the name to search? What would our query look like now? (Assuming AccessLevel = 2)

SELECT * FROM Contacts WHERE FirstName = '' OR 1 --' AND AccessLevel <= 2

With a new terminating quote and comment, suddenly our query is changed to be much more permissive. The attacker has tricked our script into revealing more information than intended. What if instead of a SELECT, we were working with a DELETE query? The entire table full of valuable data might be lost.

The second vulnerability is to a cross-site scripting attack. What if, instead of entering their name, an attacker entered some raw HTML such as "<script>alert('XSS!')</script>"? Suddenly, they've inserted their own elements and even javascript code into your web page. If they can get this to happen on a page displayed to another user, the attacker can hijack your page to show you them anything they want and it will look like it's coming directly from your page.

It Could Be Worse


Imagine a different scenario where input is passed to an exec() call and run on the system. Now instead of limiting damage to the database system, a clever attacker can run commands directly on the server. Depending on the privilege of the process and security of the underlying system, the attacker could potentailly gain root access to the machine. They can use this to access sensitive data, change important settings, or even run any program of their choosing. The entire system can be compromised and placed at the mercy of a malicious hacker.

Don't Forget Secondary Sources


Here's one that can really trip you up. Imagine an application that performs a task based on values stored in the database. It's easy to forget that despite the fact that the database belongs to you, the source of your database values may not always be clean. "We've traced the call, and it's coming from inside the house." Always assume any source of data is dangerous unless you can verify its source and security.

How Can We Fix It?


The good news is that there is a simple way to protect yourself from injection attacks: sanitize user input using readily-available libraries.

If your language of choice is PHP, there's plenty of built in functions to safely encode all varieties of user input:
Please remember that this is not a comprehensive list. There are so many different languages and environments out there. Whenever you're wielding user input, be careful to consider whether your code may be subject to injection attacks and research appropriate measures to counter such attacks.

Where Do We Go From Here?


If you feel this post is yesterday's news, I hope you take it as a friendly reminder and refresher on the topic. If any of this is new to you, I suggest you take the next opportunity to study up on the subject. Read some other blog posts. Follow some examples online. Revisit a piece of recent code and try to break it (in a test environment please!). Once you've found some vulnerable code, learn the proper way to secure it and and run your tests again to prove that it's been fixed.

If there's one thing to take away from this it's this: always treat user input as suspect. Never let raw user input pass from your application to another system without sanitizing it first.

Now go and write some secure code.

Cheers,

Joshua Ganes

Thursday, October 22, 2015

Announcing Amish Programmer Security Month

From Humble Beginnings


I started programming from a young age. I distinctly recall being in grade 5 and being curious about how some of the games in our school's computer lab worked. Having absolutely zero clue, I checked out a book from the school library on programming for the Apple IIe. My first programming experience was typing out Apple BASIC programs straight from the book and watching them fail as my two-fingered hunt-and-peck typing resulted in multiple typos.

My exposure to computers was reasonably limited until my sister and I pooled our money to buy a 100MHz 486 (with TURBO on for maximum fastness). I'm pretty sure I got the better end of that deal as I began spending a lot of time both playing video games and experimenting with QBASIC and its elaborate but confusing help files.

In high school, I took a few courses in Turbo Pascal and C++ programming. This mostly involved running through a bunch of easy exercises in each chapter and then spending the rest of our class time playing in the lab or working on our frighteningly terrible video game group project. That Pascal text book may be the only high school textbook that I've ever read from cover to cover.

After puttinsg four years of university and another year's paid internship under my belt, I was sure that I knew almost everything I would ever need to know as a professional software developer. I had no idea just how much more I still had to learn...

What Do You Mean This Is Insecure?


With so many courses behind me, I was sure that I had covered most of the important topics. If security was so important and difficult, surely someone would have covered it in one of my course by now. I was surprised when one of my colleagues pointed out security flaws in my code. I did not even recognize it as a flaw when it was pointed out to me.

Over time, with the help of more experienced mentors, bloggers, and some job-sponsored training, I began to learn a lot more about secure coding techniques. For many of my experienced readers, these have likely become second nature. Still, everyone can stand a refresher on the basics now and again. For those who, like I once was, may still be in the dark about secure coding, you have a great deal to learn.

Amish Programmer Security Month


With that in mind, I declare the next month to be security month here at Amish Programmer. I will strive to get back on board the posting train and publish a brief post covering such security topics as injection, buffer overflows, and more.

These topics have all been covered in depth by others. I will try to present a fresh take on these topics using code snippets and straightforward examples. You can tune in in the coming weeks for a short discussion of several fundamental topics in secure coding. I hope to see you back here soon.

Cheers,

Joshua Ganes

Wednesday, October 14, 2015

Challenge Your Assumptions

Today's brief post was inspired by my recent road trip with my wife from here in Vancouver, BC through the states of Washington and Oregon.

When it comes to road trips, I'm all about efficiency. Pit stops run like clockwork. The guys from NASCAR have nothing on me. That's why I prefer self service with pay-at-the-pump when stopping for gas. My fingers fly with precision as I fetch my card from my wallet, insert it in the machine and punch my way through the menu to start up the pump and select my grade of gasoline to begin fueling.

On my first stop of the trip, everything was running smoothly right up until I was stopped in my tracks by the following prompt:

Enter Zip Code:

 

Cancel         OK

Unfortunately for me, as a Canadian, I don't have a numeric 5-digit zip code. Instead, I have a six character postal code with a mix of numbers and letters. Try though I might, I could not find a way to enter my postal code or skip this prompt.

Edit: I've since been told that you can simply enter the numeric portion of your postal code followed by zeros

After a few moments of poking around at the machine, I eventually resorted to pressing cancel and walked into the station to speak with the cashier. Within a few minutes I was back on track (a lap behind the race leaders).

My experience with the zip code prompt was hardly life altering. Still, it illustrates some of the problems that can come up when your software makes incorrect assumptions (e.g. all gas station customers have a zip code). Challenge your assumptions and do your best to consider all possible users. Question whether all the information you prompt for will be available or match your expected format. If not, your software should gently guide the user on how to proceed.

Cheers,

Joshua Ganes 

P.S. Did you see the seventh inning of today's Blue Jays vs. Rangers game? It was well worth a second look! Go Jays!

Wednesday, May 27, 2015

Any News? - Working With Third Parties

I Am A Rock


We programmers tend to be an independent bunch. Like a lone wolf, we often prefer our own company to that of others and like to work alone. It's been said that, "no man is an island." I have to agree (unless you happen to be Paul Simon). Even programmers need to interact with their coworkers, managers, other departments, and even third-party contacts to meet the day-to-day demands of their jobs.

While networking and politics may not come naturally to all developers, we're hardly worthy of our titles if we cannot find a way to communicate clearly and accurately. Programmers must express requirements, prerequisites, technical issues, and design ideas in ways that others can understand so that all players can do their jobs and assist us in completing our own work.

Have you ever found yourself in this situation? You're beginning to wrap up a project, but you notice a seemingly simple, yet annoying bug in the library you are working with. Being diligent, you follow up with the library provider to report the issue. You ask if this is a known issue. You ask if you're using their library correctly. You ask if there are any workarounds or alternative approaches they can suggest. You go ahead and make a note to yourself to revisit the problem once the provider has had time to review your questions. Fast forward a few weeks. You've cleaned and polished every other corner of your software, but you really need to address this one remaining issue before you can build a final software release. In the meantime, the only feedback you've received is that they're, "looking into it."

On Dependencies


A wise piece of advice for any project is, "eliminate the dependencies." Just as in the example above, making yourself dependent on someone else's assistance can be a very compromising position. Not being able to deliver a solution because you're waiting for someone else to act can be uncomfortable, awkward, and frustrating. If you can eliminate these dependencies, you may still run into roadblocks of your own, of course, but you will retain the power to work through them and implement a solution dictated by your own schedule and budget.

It's not really fair to say, "eliminate the dependencies", and leave it at that. For many developers, creating a dependency is not really a choice. Maybe your boss, team, or client is demanding a specific technical requirement. Maybe you can't afford any alternatives. Maybe, as is often the case for me, your work involves creating an interface between your own software and a third-party tool. In this case, if you eliminate the third-party tool, you eliminate the need for your own software entirely.

Now, before I go accusing anyone of wrongdoing, let's try the shoe on the other foot. The development team in charge of that third-party library has more than just one bug fix on their plate. They need to balance support and bug fixes against their company's need for revenue-generating new features and routine maintenance. They may have a significant collection of issues that go through a triage system, and your bug report hasn't yet make the cut. I'm sure there's several times when others were waiting on me to complete my fix before they could proceed. Any fingers I point here need to be pointing back at myself also.

Planning For Success


The key to managing the risks of creating project dependencies is to identify those dependencies and understand them. Do your best to perform a reasonable analysis of the risks (if possible). Try to understand the expected flow of communications between both parties. Manage expectations by discussing project timelines and turnaround times for both technical inquiries and potential bug fixes. Understand the implications for your project when you cannot get the other side to take quick action. Are there alternatives? What are they? Will they be just as risky? If the risks are too high, sometimes the only winning move is not to play. Don't allow yourself and your organization to be sucked into wasting resources on an endeavor that will never succeed.

One of the best ways to reduce delays due to third parties is to be prudent and up-front to find and report any issues. Fixing bugs takes time. The sooner a bug is reported, the sooner it will be fixed. Therefore, schedule any development and testing work dependent on third-party support at the earliest opportunity. As soon as an issue is discovered, take time to understand the problem in reasonable detail and communicate it as soon as possible. This will give everyone the most time to respond and allows you to proceed with other features until the bug can be resolved.

Once you have requested support from an outside party, follow up and keep in touch. Remember, the contact at the other end is a person too. Treat them with respect and professionalism. Keep your communications positive and inquisitive. Try not to accuse or pester. There are differing opinions on how frequently you should send messages, so I won't offer specific advice. Instead, I suggest you give your best effort to judge the tone of their replies to see if the other side has become annoyed. Never hesitate to follow up if a written deadline or estimate is missed. Your goal, again, is not to accuse, but to make sure you are fully informed and kept up to date.

If worst comes to worst, don't be afraid to escalate. Give your best effort to work through the default channels provided. If, however, you simply cannot get the answers you need, the time may come where you need to ascend the ladder of power. Do your best to do so as respectfully as possible. Ask your contact if there is an option to expedite or escalate the communication process. I have seen cases where two lower-rung employees went back and forth for weeks. A single brief phone call between two big wigs was all that was required to sort things out. Nobody wants to be disrespected and have someone go over their head. Reserve a power move like this as a last resort, but don't be afraid to pull the trigger if all other options have been exhausted.

The very last resort sometimes required implementing a heroic workaround. For this to work, you need to understand the third-party system well enough to automatically recognize the failure and wield the power to fix it through one or more additional steps. Rather than having the third-party implement a fix, you can choose to live with the existing bug, masking its existing by coding your own workaround fix. On a personal note, while this option is often the best business decision, it often feel like an enabler. We just let the bug continue to fester and bend over backwards to get past it rather than correcting it at its source.

Good Days And Bad Days


In my professional experience, I have worked with a fair number of third parties. Some have been eager and helpful. They are quick to respond and provide clear details needed to solve my problems. Others have been slow to respond and very terse and unclear in their replies. After working through a few issues with a contact, you will quickly develop an opinion of their competence. This can be used in further risk assessment. My advice is to go with your gut. So far, I have never once changed my opinion from that initial impression following subsequent communications.

What's your best or worst story of working with a third party? What went right and what went wrong? What would you do differently? Let me know in the comments.

Cheers,

Joshua Ganes