From Lousy to Beautiful

From Lousy to Beautiful

This guest post is by James Schorr, who has been developing software since 1999. He is the owner of an IT consulting company, Enspiren IT Consulting, LLC.  He lives with his lovely wife, Tara, and their children in Kansas City, Missouri. James spends a lot of time writing code in many languages, doing IT security audits, and has a passion for Ruby on Rails in particular. He also loves spending time with his family, playing chess, going to the shooting range, hiking, fishing, and investing. His professional profile is on LinkedIn.

James M. Schorr Due to the inherent flexibility of Ruby, there are often several ways of doing the same thing. But how do you recognize the best way, the “beautiful” way to go about something? It can be tough to describe and there may indeed be several “best ways” to do something. The ease of expressing real life through Ruby can give us a false sense of security. I have come to realize that terrible code can be written — even with such a beautiful language.

Let us work through a simple example together. In our example, we’re calling a remote API service with HTTParty, and parsing through the JSON response, which consists of a simple array of stats, which we’ll refer to as “Today’s Stats”. Please note that comments in the code intended just for the reader of this article will be prefaced with ‘## Reader’. Hopefully, you’ll see the progression of the code’s quality as we go.

Here’s our first attempt, which is pretty lousy. Can you spot a few of the reasons why?

Attempt 1 – Blind Trust

module StatsApi
  require 'json'
  require 'httparty'
  require 'jsonpath'

  def find_users_stats(users_handle)
    params = { user: users_handle }
    a = HTTParty.get('http://928just-a-bogus-example-site.com/stats', params)
    t = JsonPath.on(a.body, '$.').first
    u = t.select {|k,v| k['user'] == users_handle }.first
    @users_stats = u['stats']
  end
end

Did you spot the issues? There are quite a few, some that affect us now and some that might affect us in the future. For now, let’s focus on the most pressing, immediate issues:

  1. We are trusting that our call to the API will succeed.
  2. We are trusting that the account that we pull back will indeed have the user that we’re looking for.
  3. In both cases, we are then attempting to parse through the data that may not even make it back to us or have what we expect it to have.
  4. What does ‘a’, ‘t’, and ‘u’ represent? Would the next developer working on this code know what I’m referring to? While it saves space, it isn’t very friendly to the next developer.

Attempt 2 – Handling the Unexpected

module StatsApi
  require 'json'
  require 'httparty'
  require 'active_support/core_ext/string' ## Reader: provides .present? and .blank?
  require 'jsonpath'

   def find_users_stats(users_handle)
    params = { user: users_handle }

    a = HTTParty.get('http://928just-a-bogus-example-site.com/stats', params)
    ## Reader: .present? (or .blank? when appropriate) can be nice to use as they checks for more than just nil.
    ## Additionally, we make sure that 'a' is 'present?' before calling 'response' on it.
    if a.code == 200
      todays_stats = JsonPath.on(a.body, '$.').first
      if todays_stats.present?
        users_info = todays_stats.select {|k,v| k['user'] == users_handle }.first
        if users_info.present?
          @users_stats = users_info['stats']
        end
      end
    end
  end
end

Now we seem to have solved a couple of issues, but this code could still be improved a bit. Notice that:

  1. Nothing is returned if the call to the API fails or the user’s stats aren’t found. Now, of course in the code that calls this method, we could handle that there but is that the “beautiful” thing to do? Perhaps in the future we’ll have 10 areas in our code that call this method.
  2. Nested if statements are present — ugh. When I was newer to Ruby, I would often write a lot of these; reading through them months later made my head hurt. It certainly wasn’t beautiful to look at and made it tough to follow as more logic was added over time. Thankfully, there is a better way.
  3. Note that we could add more conditionals to ‘a’ to ensure that HTTParty returned a response correctly, but we’re going to choose not to as it’s a well-tested gem.

Attempt 3 – Bringing Clarity

module StatsApi
  require 'json'
  require 'httparty'
  require 'active_support/core_ext/string'
  require 'jsonpath'

  def find_users_stats(users_handle)
    # This will call the Stats API and pull back the user's stats.
    params = { user: users_handle }

    todays_stats = HTTParty.get('http://928just-a-bogus-example-site.com/stats', params)
    raise "Today's Stats not found." if todays_stats.code != 200

     # Finding the User's information...
    users_info = todays_stats.select {|k,v| k['user'] == users_handle }.first
    raise "User's stats not found." if users_info.blank?

    # Determining the User's Stats...
    @users_stats = users_info['stats']
  end
end

Okay, we’re getting closer. Now we have ‘todays_stats’ and ‘users_info’ in the code so that it’s more  clear as to what we’re referring to and an appropriate error message is returned if the account or user are now found. Notice how we added a little white-space and comments to make it easier to read. However, we’re still calling the API and also parsing through stats in this one method. This might be fine for now if we only call the API for one purpose. But what about the future? What if a need arises to adjust the stats, add new stats, etc… via the API?

Let’s try to future-proof our code a bit by making it more flexible and configurable. We’ll use the excellent Settingslogic gem (you can add that yourself) to make the API settings configurable. This will help when the Stats API’s URL, version, etc… changes. We’ll also add a method to adjust the user’s stats and one that performs the actual API call. Notice how we can accomplish this by using ‘send’. Thus the various HTTP verbs can all be used as desired while keeping the code pretty DRY.

Attempt 4 – Sustainability

module StatsApi
  require 'json'
  require 'httparty'
  require 'active_support/core_ext/string'
  require 'jsonpath'

  ## Reader: note that @error is raised so that we can easily handle that in our parent methods...

  def find_users_stats(users_handle)
    # This will call the Stats API and pull back the user's stats.
    # All of the API settings are found in config/settings/api.yml.

    # Finding the Account...
    params = { user: users_handle }
    call_stats_api('get', 200, true, params)
    ## Reader: note that the following line is optional as you could have conditional code in
    ## the parental method based on @call_succeeded.
    raise @error = 'Account not found.' unless @call_succeeded

    # Finding the User's information...
    ## Reader note that we raise @error so that we can easily handle that in our parent methods...
    todays_stats = JsonPath.on(@result.body, '$.').first
    raise @error = "Today's Stats were not found." if todays_stats.blank?
    users_info = todays_stats.select {|k,v| k['user'] == users_handle }.first
    raise @error = "User's Stats not found." if users_info.blank?

    # Determining the User's Stats...
    @users_stats = users_info['stats']
  end

  def adjust_users_stats(new_stats)
    # Adjusts the users stats with the new_stats Hash.
    call_stats_api('put', 201, new_stats)
    ## Reader: note that the following line is optional as you could have conditional code in
    ## the parental method based on @call_succeeded.
    raise @error = 'Stats adjustment failed.' unless @call_succeeded
  end

  private
  def call_stats_api(http_verb, success_code, response_body_reqd=false, data=nil)
    begin
      # This calls the Stats API, passing along the necessary params. The data refers to what
      # should go in the JSON payload.  The success_code is the desired code that the parent method
      # would consider to be 'successful'.  This allows us to pass in the actual HTTP status codes
      # that the 3rd party API returns.
      ## Reader: Note that the various Settings mentioned below reference the
      ## config/settings/api.yml file in order to make it very easy to change things in the future.

      ## Optional but more future-proof; these don't have to be configurable but probably should be:
      params = { headers: AppSettings::Api.stats_headers, version: AppSettings::Api.stats_version,
                 format: AppSettings::Api.stats_format }

      # Adding the JSON data to the params...
      ## Reader: notice that we don't blindly trust that the data parameter is indeed
      ## a hash; what if something else was sent to this method by mistake?
      raise 'Invalid data received' unless data.is_a?(Hash)
      params[:body] = data.to_json

      ## Reader: notice that we use Object's 'send' so that we only need one method
      ## for API calls with the desired HTTP verb. The following line will be what's
      ## returned to the parent method.

      ## Optional (you can just write in the URL if you wish) but more future-proof:
      @result = HTTParty.send(http_verb, AppSettings::Api.stats_url, params)

      # Determining the success of the response.
      code_matches = @result.code == success_code

      ## Reader: if the response_body_reqd is set, then a call is only considered successful if the
      ## response code matches and the body has data in it. Otherwise, only a matching code is considered
      ## successful.
      @call_succeeded = response_body_reqd ? @result.body.present? && code_matches :
                                             code_matches
    rescue => e
      ## Reader: notice that in production-ready code, we'd never return the real error message to the
      ## end user as it would expose too much and not be very human friendly.  It would probably be best
      ## to just return a "human-friendly" error message and log the atual one.
      @error = e
      Rails.logger.error("Sorry, an error occurred: #{e}")
    end
  end
end

Now our code is quite a bit more readable, more flexible, and easier to maintain. There are a few spots where we could probably get away with removing our comments since the variables are so aptly named. We could also (and should if there are needs for other types of APIs to be called) create a new class that calls APIs. We’ll leave that for another day, though.

In general, though, our code has improved quite a bit.

Here are a few Ruby resources that I would like to recommend (discounts given with author’s permission):

Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Technorati Tags:


(Powered by LaunchBit)

The Ongoing Vigil of Software Security

The Ongoing Vigil of Software Security

This guest post is by James Schorr, who has developed software since 1999. He is the owner of an IT consulting company, Enspiren IT Consulting, LLC.  He lives with his lovely wife, Tara, and their children in Kansas City, Missouri. James spends a lot of time writing code in many languages, doing IT security audits, and has a passion for Ruby on Rails in particular. He also loves playing chess and spending time with his family. His professional profile is on LinkedIn.

James M. Schorr TThe news is often filled with stories about exploits affecting large corporations and widely used software (LinkedIn, Yahoo, Windows, Linux, OS X, *BSD, Oracle, MySQL, Java, Flash, etc…). However, a tremendous amount of successful hacks and exploits take place on a daily basis on smaller-profile systems that we never hear about.

Some of the reasons that we keep seeing these types of exploits are that the “bad guys” are much smarter and more determined than we give them credit for, we’re much lazier and more ignorant than we take responsibility for, and security is difficult to manage properly. As we become more and more reliant upon software, it is imperative that security be taken more seriously.

What’s the big deal?

Consider this somewhat over-the-top thought exercise:

Think of your systems, databases, and code as a ship floating in the middle of the Atlantic. The ship was fairly hastily constructed as the management team pushed the various craftsmen to get done in time for the journey.

It’s the middle of the hurricane season. The waves are getting higher, sharks are circling your boat, and aboard are quite a few passengers. Most of the passengers are of a fairly decent ilk, but some are not. This latter group, partially due to the insufferable boredom that accompanies their long journey, have taken delight in drilling holes in the side of the boat (with the tools that were discarded during construction). Other troublemakers spend their time throwing chum overboard to the encircling sharks and even, when no one is watching, throwing each other overboard. A few of the cleverer sort spend their time impersonating the crew and using their new privileges to look for ways to take over the ship. Sadly, even some of the crew members have been persuaded into joining their mutinous ranks.

As time goes by, the remaining crew loses its ability to prevent damage to the craft and protect those on board, as a result of sheer exhaustion, the tenacity of the passengers, and the natural wear and tear of the elements.

What’s the point of this mental exercise? We need to realize that unrelenting attacks abound, both from within and without the system. If not properly addressed, they only escalate over time.

Security is a word that has a long, storied past. According to most dictionaries, one of the definitions of security includes, “free from danger”. Of course, stating that a system, code base, network, etc. is secure is quite naïve at best, dangerous at worst. Recognizing the threats is the first step toward positively addressing them.

Ask any IT team member that is charged with “securing” anything and you’ll quickly find out that it is an extremely difficult, often thankless task. Even in a tightly controlled environment, it can be pretty tough, especially during times of extreme change, turnover, growth, etc.

Why should we care?

We need to care because our applications, databases, and systems:

  • are regularly being threatened from the inside and the outside, often without us even being aware of it.
  • are depended upon by users who have invested some degree of their money, trust, time, or work into using them.
  • haven’t “arrived”. There is always a way to circumvent the “system”.
  • typically depend upon the “happy path” scenarios (e.g. when all goes well).

What can we do?

Thankfully, there are quite a few things that can be proactively done to help mitigate the risks and stave off the threats. For brevity’s sake, I’m going to give a high level overview of what can be done to help prevent exploits:

Team Security Measures

  1. Who should be in charge of our project’s security? Involve the right people, taking the time to get to know their character and mindset. Not everyone is cut out to think with the type of mindset needed to properly manage security. Unless someone is really into security, is trustworthy, is assertive, and unafraid of conflict, they simply aren’t the right person for this task.
  2. Who has need-to-know? Need-to-know is an essential principle in projects. Data leakage often inadvertently occurs by team members that probably didn’t need the information to begin with. Those that realize the “big-picture” usage of the data and need access to it for their tasks typically realize the need to keep the data private.
  3. Separation of duties with each area managed by a small core team. While not always possible, it is helpful to have one main realm of responsibility per team. Also, the core team of each area/realm needs to remain just that – the core team. In other words, the more people added, the tougher it is to keep things secure.
  4. How, when and to whom do we communicate? The procedures for securely communicating need-to-know information are critical to establish. Various methods need to be implemented to allow team members to exchange information in as secure a fashion as possible. An example might be the usage of an encrypted volume in a shared drive (retaining the control of the encryption details).
  5. Knowledge Transfer: when someone leaves the team, great care should be taken to transfer the knowledge to the new member in a secure fashion. Additionally, all relevant credentials should be changed immediately, no matter how trusted that individual or group was. A simple exit checklist – that is followed – can greatly help with this.

Technological Security Measures

  1. Testing is critical: we are testing, right? In dev-speak, tested_code != secure_code but tested_code.class == SecurityMindset. In other words, it is possible to write insecure, tested code, but proper testing does seem to inherit qualities from a security mindset and to encourage more thoughtful programming. In my opinion, testing generally falls into two main types:
    1. Code-based Testing: I’ll let others bore you with a long list of what’s available out there but do want to point out that real-world progress can be made towards better securing code with the usage of tools/methods such as: Rspec and friends, TDD, BDD, etc.
    2. Human Testing: sometimes nothing beats enlisting the help of others to pound away on our beloved projects. You’d be surprised at how many issues are found by this approach, often leading to cries of, “But users aren’t supposed to do that!”
      1. Non-technical users: enlist someone who can has a hard time finding the / key. This type of person will usually do all sorts of unexpected things. The unexpected behavior can quickly reveal the hidden weaknesses in the UI, workflow, and security.
      2. Enlist the upcoming geeks: you know those kids who are always jail-breaking phones? After issuing a few half-hearted reprimands, ask them to “conquer” your app. Offering a prize can’t hurt.
      3. Enlist an expert to audit your code, procedures, and projects.
  2. Logging:
    1. What to log: in general, the more information about transactional details (transactional referring to any actions that involve change), the better. Note that anything related to attempted security breaches needs to be logged. Admin alerts should also be automatically sent out; these alerts need to be designed with great care to not transmit anything that would harm the system if intercepted in transit.
    2. What to never, everlog:
      1. Credentials: passwords, API keys (abstract before logging: e.g. if Bob does X with an API key, put a different identifier in the log file, not the key).
      2. Credit card numbers, PINs, debit card numbers, anything banking related unless we are doing so in compliance with PCI standards.
      3. Medical information (see HIPPA – Health Insurance Portability and Accountability Act or your country’s corresponding laws).
      4. Anything that can be used to compromise the systems or it’s users.
    3. How to log: I personally prefer a two-pronged approach: 1) writing to log files which are automatically transferred offsite, 2) an audit trail via a NoSQL database (using a fire-and-forget type of approach; in other words send the insert but keep on moving, a failure to log to the audit trail should alert admins but not slow down or impact the user’s use of the application at all).
    4. When to log: as close to the event as possible, to minimize the chance of data loss.
      1. Log Alterability: Think, “if I was a hacker and compromised this system, I’d want to clean up after my activities”. How do I make my logs non-alterable, even by support staff?
  3. Access Levels: these typically fall into the following:
    1. Users
      1. What can they access and why
      2. Who can change their level (e.g. can the user manage their own level via subscriptions)?
    2. Support Staff
      1. Level 1 CSR
      2. Level 2 CSR
      3. Level 3 Admins
      4. Dictators (can do anything with no recourse)… careful with these types.
  4. Crucial Elements:
    1. Account Lockouts
      1. Users are locked out for some period of time when they fail to login after X attempts or try via different IPs, etc.
      2. Users are locked out and admins alerted when they try to get around the system (these types of lockouts do not expire with time but rather require a Support Staff person to unlock them based on their discretion).
      3. Ability for Support Staff to lock and unlock users very quickly after following a procedure to record why they’re doing so. A permanent record needs to be kept as to who unlocked whom and why.
    2. Account Password Policies: password strength, requirements to change the password every X days, password history (can’t reuse old passwords), etc.
    3. Other: click-limits, IP address binding, geographic-binding, usage of Oauth 2, etc.
  5. Frameworks and Software Libraries: it’s fairly common to have security vulnerabilities “appear” due to the integration of code from other sources. Of course, no one has time to re-invent the wheel, so to speak; nor should they. It is a good practice to always read through the source code and reported issues of 3rd party software prior to implementation.
    1. Take the time to search for some of its common exploits and best-practice methods of usage. Have we taken the time to test what X library (framework, gem, plugin, etc.) would mean for our application’s speed, stability, and security?
    2. Refrain from handling some things ourselves. A good example is credit-card processing. Why handle it yourself when a 3rd party, tested service will likely do so in a more secure manner? Look for a project that has been around for a while and has a good track-record of quickly closing vulnerabilities.
  6. Servers and Hosting: it may save some money to host on a shared host or cousin Bill’s server, but will the data be secure? It’s best to strive for meeting all three of the CIA principles (Confidentiality, Integrity, and Availability) when choosing a host, striving for at least a medium-level for each principal.
    1. Keep the servers up-to-date.
    2. Use intrusion detection applications (e.g. psad, fwsnort) to alert admins of attempts to break in the system.
    3. Use a properly configured firewall that is easy to adjust quickly.
    4. Send the logs offsite (e.g. not on the same “box”) to a secured server on a frequent basis.
    5. Backups: ideally, these should occur nightly of the entire codebase, logs, and database dumps; these backups should be kept offsite in the same manner as logs.
    6. Imaging: frequent images of servers can be helpful for forensics in the event of an exploit and for data recovery.
    7. Server-side miscellaneous applications (Apache, Nginx, SSH, OpenSSL, etc.): disable unused modules, limit connections, use non-default ports, etc. (see Resources for more ideas).
    8. Schedule checks for rootkits and malware on a daily basis; be sure to alert admins if any is found.
  7. Database(s): Familiarity with the database(s) is key to keeping them secure. For instance, if a development team is very familiar with MySQL and decides to add in a secondary technology alongside (maybe some MongoDB databases), it would be wisest to evaluate the architecture and security implications prior to implementation.
  8. Credentials:
    1. Where and how should we store the credentials that our app needs (e.g. api keys, database credentials, etc.)? A good thing to ask ourselves is, “if someone did get into our server (as non-root, as if they did as root, it’s game-over anyhow), what could they get and who would it hurt?”
    2. Are we deploying our credentials to GitHub or other VCS? If so, we’re blindly trusting that 3rd party to be and stay secure.
    3. Changes should be planned for and completed whenever there is a change in personnel and on a periodic basis. This can become a real hassle unless thought is given along the lines of, “How do we quickly change these credentials?”

I hope that this article has given you at least a few ideas of how to better improve your software project’s security. If so, I’ll consider it a success. Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Resources

Below are some resources that may be helpful (those that I have found extremely helpful over the years are denoted with a * next to them):

Technorati Tags: ,


(Powered by LaunchBit)

How Can We Develop For Tomorrow’s Needs?

How Can We Develop For Tomorrow’s Needs?

This guest post is by James M. Schorr, who has been in IT for over 14 years and been developing software for over 11 years. He is the owner of an IT consulting company, Tech Rescue, LLC, which he started along with his lovely wife, Tara, in 2002. They live in Concord, NC with their three children Jacob, Theresa and Elizabeth. James spends a lot of time writing code in many languages and a passion for Ruby on Rails in particular. He loves to play chess online at FICS (his handle is kerudzo) and to take his family on nature hikes. His professional profile is on LinkedIn and you can read more of his writings on his blog.

James M. Schorr The average developer is often forced to get code out the door as quickly as possible, primarily due to unrealistic deadlines and budgets. As a result, the quality and future expandability of software is greatly harmed. Software is now used in medical machinery, our vehicles, power plants, stock markets, aircraft, weapons, etc… As software becomes more and more critical in our lives, the need to think long-term is becoming increasingly critical.

Obviously, the quickest way is almost always not the best way. I hope to give some practical steps to those involved in software development that will help in the development of stable, long-lasting software. A proper strategy session involving the below steps can help save a lot of wasted time and money.

Quality, future-resilient software is tough to define, but reveals itself when it does what it’s supposed to without unpleasant surprises, handles unpredictable user input and system issues in gracious, non-devastating ways, and, in general, makes the user’s life easier. The tough part is that user’s needs and systems change. How do we engineer for tomorrow’s needs?

The keys to successfully developing long-term software are:

Establishing the Purpose: What is the point of the software? Do the needs that it are anticipated to be met look as though they will be the same core needs in the foreseeable future? In other words, will the main needs be met by this software and can we easily build out from there? If not, we need to keep the anticipated future needs in mind as we “scope” out the architecture of the project and provide “space” for them.

Choosing the “Stack”: (what technologies, languages, etc… will be used). The stack should be chosen carefully, based upon:

  • proven stability. For example, it may be “cool” but unwise to write the software in the latest-and-greatest language. I’ve seen instances where a language/framework is chosen strictly due to its current popularity. This is typically a recipe for disaster, as those who go (and enjoy) that route typically move on to the next greatest thing, leaving code behind for non-fad-following developers to handle.
  • current in-house knowledge. For instance, maybe our developers love and know Ruby, should we really force them to write an app in VB? Or perhaps it is a Microsoft-shop, are time and funds available to facilitate the learning-on-the-fly of non-MS technologies? I don’t believe that it is ever appropriate to write mission critical software using a language/framework that is unfamiliar to the developers. There are times, however, where the software is so mission-critical and matches a language’s abilities to the point where it makes sense to pull in new talent. It can be argued that software can be written in almost any language, that the language itself doesn’t matter much. But sometimes it really does, both in terms of expressiveness and developer satisfaction (note: I still contend that a happy developer is a good developer, or at least becoming one).
  • infrastructure requirements. Do we have the hardware and network necessary to decently support the software and its anticipated usage? Disk space, memory requirements, OS, network speed, etc… All of these matter, a lot. It’s best to always plan for 2-3x the anticipated usage. For instance, for a web app, if we anticipate 1k users, let’s build for 2-3k users, with built-in monitoring of the resources being used and a plan of how to scale up quickly when we hit a “soft” threshold.

Planning:

  • Architectural Drawing: I’m a big fan of having at least the “skeleton” of the project drawn out, particularly on a white-board (I’m a bit old-school, I know). It doesn’t have to be a fancy diagram or complicated UML diagram, just a simple drawing; the more understandable, the better. This high-level overview provides guidance when we’re deep into code, as we can look up and see if we’re on track (as it’s all too easy to go down a code “rabbit trail” if we’re not careful). It is counterproductive, however, to draw out every little detail, as this will stifle creativity and overwhelm us while we’re writing code (we just won’t look at the diagram then).
  • Establish Deadlines: we do need to know the deadlines. It’s best, in my opinion, to have several small deadlines with a semi-flexible final deadline. This helps us keep on track and measure our progress little by little. As we hit the small deadlines, our confidence builds, which then improves our productivity and, in general, our code quality.
  • Using Available Expertise Wisely: does it make sense to assign Bob, the awesome Python programmer, to doing CSS and Bill, the great designer, to slinging code? Obviously not (I have seen some managers try this, though), we may lose both team members or end up with Google copy-and-paste code and animated GIFs in our Project. Cross-training is a nice and potentially valuable concept, but it should be done outside of a software project with its accompanying deadlines. Future minor features might provide a better opportunity ground for cross-training. If Bob’s swamped, maybe we need to find him some decent help. :)
  • Determine the Deployment Strategy(for both during and after the Project):
    • code should be checked into our version control system prior to any deployments.
    • maybe we should only deploy code after business hours after alerting such-and-such a group. If our project has any possibility of negatively impacting others, notification is not only kind, but often necessary, especially for large changes.
    • a rollback strategy must always be in place. This strategy must be easily understandable with simple steps, so that little, if any interpretation by support staff, needs to be done in the “heat of the moment” support calls. Even if our developers are top-notch, until code gets into Production, we cannot be 100% sure that it will not need to be rolled back. This is why major companies often have to release an update quickly after a major release. Some things just can’t be easily discovered until they’re released into “the wild”.

Building with Expansion in Mind:

  • One of the wonderful aspects of developing software is also its most dangerous aspect: flexibility. A feature or component can often be written in different ways. There typically are only one or two best ways, though. It can be very difficult to determine, unless one steps back from the project and thinks it through. Well-known software principles help a great deal with this, but come up short if they are not “placed up against” the anticipated needs of the future (in other words, if we don’t understand where we’re going, our code will still be awful even if we follow DRY, OOP, GOF, etc…). As much as possible, this needs to be done not by the developer but by someone outside the code, so to speak, perhaps a technical team lead, etc…
  • When adding core features, we need to at least take a few moments to think through possible future implications of what we’re doing. For example, our Component A is currently parsing JSON from website B using C credentials. Component D depends upon Component A’s data. Wouldn’t it make sense to have these in an encrypted setting field somewhere to make it easy to change in the future? If Component A’s data was slightly different, would Component B “blow up”? Maybe we can abstract all of this a bit?
  • Avoiding Spaghetti-Code: proper design and a commitment to sticking to the design in the future helps to prevent our code from such entanglement. In other words, we need to commit to never, ever quickly throwing code in to the project, as this leads to “spaghetti-code”. Of course, there may be the exceedingly rare occasions, where we may need to do such a stop-gap measure due to an emergency, but we must then learn from our mistake and commit to re-engineering that portion of the code properly.

Data Safety:

  • As we depend more and more upon data, it’s becoming increasingly important that we do our best to have automated backups, which are then checked frequently by a person. This cannot be emphasized heavily enough. All too often, properly designed backups can stop working without anyone noticing until it is too late.
  • If encryption is used:
    • the encryption keys need to be stored off-site in at least 2 secure places. Imagine if we lost our server(s), our office burned down, our VPS provider goes offline, etc…- even if we had backups, could we get to the raw data if needed? No one wants to start over from scratch.
    • Does the encryption depend upon a certain cipher? If so, what is the game plan for when that cipher is cracked someday? How easy will it be for us to move to a new cipher?
  • Does our data depend upon a specific version? For instance, maybe database X version Y can open the data but no other versions can. Do we have a backup of that version to access the data if needed? Better yet, this reveals a key flaw in our design. Our data should not be heavily dependent upon any software version.
  • Would our data be understandable if a new developer 10 years from now is assigned to work with it? For instance, if a column for a user’s API Key is called usrscr_ak12, we may understand it, but it’s not future-proof (a better term may be “future-resilient”, since nothing is truly future-proof). Such obfuscation attempts provide little security, as if someone can get that far (to the data), we’ve lost the security “battle” anyhow. Data should be clearly understood by those who can access it.
  • Can our data be exported easily when the software that we’re lovingly developing now someday gets decommissioned? All software will eventually get replaced by something better. How easily can our data be decoupled from our application?

Pin-pointing Possible “Dominoes” in our project and code-base (e.g. if A happens, does it affect B, which then affects C, etc…, these can be like dominoes). Amazon’s recent AWS issues in 2011 revealed the criticality of this step. The more time that we spend anticipating what can go wrong, the more we can establish quick steps to both prevent such issues and to mitigate possible damage. At the bare minimum, the possible “dominoes” and recommended quick steps need to be written down somewhere. This can greatly help to expedite future troubleshooting.

  • Our Software: We must try to anticipate, as much as possible, what the interdependencies are in our project and its surrounding infrastructure. These dependencies should be in written form and re-reviewed as further functionality is added to the software in the future (e.g. ITIL Change Management).
  • Dependent Software: What software or systems will depend upon our software? When our system goes down, will other software be slamming our system asking for a response?
  • Dependent Systems: if we saturate our network, is our software designed to “back-off” and retry after an appropriate, randomized delay?

Obviously, none of the above can be done overnight. If even some of the above is done, however, the chance of our software having a longer-lasting, positive impact will be greater. I recommend that the start of each project have at least a 3-5 days dedicated to going through these steps. Gathering input from the teams of people who are responsible for various components (e.g. clients/end users, network, sysadmins, developers of other dependent software, etc…) will be invaluable. The payoff will be great.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Also, do check out James’ other article: “Do You Enjoy Your Code Quality?” on RubyLearning. Thanks!

Technorati Tags: ,

Do You Enjoy Your Code Quality?

Do You Enjoy Your Code Quality?

This guest post is by James M. Schorr, who has been in IT for over 14 years and been developing software for over 11 years. He is the owner of an IT consulting company, Tech Rescue, LLC, which he started along with his lovely wife, Tara, in 2002. They live in Concord, NC with their three children Jacob, Theresa and Elizabeth. James spends a lot of time writing code in many languages and a passion for Ruby on Rails in particular. He loves to play chess online at FICS (his handle is kerudzo) and to take his family on nature hikes. His professional profile is on LinkedIn and you can read more of his writings on his blog.

James M. Schorr There’s something innately satisfying about programming; or, at least, there should be. It’s a creative act in and of itself that has incredible power and potential to affect change. Craftsmanship, in and of itself, typically attracts those who like to think outside the box and those who like to create. In this way, it really is more of an art than anything else and can produce a great deal of satisfaction and delight.

Compare the carpenter that hand-crafts a beautiful cabinet to a factory worker who pushes buttons to cause a machine to form identical cabinets over and over again. While both can experience a measure of satisfaction in their work, only the carpenter enjoys long-term satisfaction. The goal of this article is to enable you to improve code quality and, thus, transform the mundane into the beautiful. No matter where you’re at on the spectrum, beginner to advanced, there is always room for improvement. As the code quality improves, your ability to delight in it and enjoy what you’re doing does as well.

I have a number of applications in Production and have learned a lot over the years about the code quality, systems, and programming in general. I’ve also been regularly called on to carefully check a tremendous amount of others’ source code in my consulting business and earlier jobs. Here are some practical, real-world ways to improve your code quality and development habits in general. I have learned the hard way on a few of these. Some may only apply to the non-employee developer, but almost all are pretty universal:

Pre-Development

  • Gather the requirements and “stories” from the client. I like the idea of stories better because it’s more in line with how the non-developer thinks. In other words, “I want A to happen as a B when I do C”. Make sure that those you’re meeting with are told to come prepared with as much information as possible, especially from the end users of the planned application.
  • Question every need and distill them down by asking “Why?” In general, the requirements of users are based on what they know the possibilities are and what they’ve used before. You, however, may quickly see that a need is better met a different way, or even cause issues if fulfilled.
  • Clarify what is necessary and “nice-to-have”, as this has a direct impact on the timeline and budget.
  • Refuse to reproduce lousy software. In other words, turn down work, if necessary, that will need you to reproduce some other system that wasn’t designed well unless you’re given the freedom to do it right. It’s just not worth it, both for you or the users. How do you recognize lousy software? Look at the UI, speed, security, and stability; would you want to use it?
  • Reject unrealistic timelines nicely, “Yes, I could do this in 3 months, but you won’t get the quality that you deserve. To do this right, we need 5 months.” Stick to your deadlines if possible. Ask yourself, “Do I want to work 17-20 hours a day to hit an unrealistic deadline?” Been there, done that, lesson learned.
  • Accept the least amount of features for functionality for the first version. It’s far more important to have a stable foundation than a ton of features. Slate other features for future releases after the foundation is laid properly. You can expect future needs (and should), but only include the minimums for the first version or iteration.
  • Realize that just because we “can” doesn’t mean that we “should”. Anything’s possible, but not everything’s advisable. This is a tough one, especially in the pre-development excitement, brainstorming phase. For instance, yes we could pull in a Google map for every row, but should we? How long will the requests take as the data set size increases?
  • If you don’t know the answer to “Why” for a feature and “What” it will impact, don’t even start answering “How”.
  • Ensure that you’re using a great version control system (I’m partial to GIT) and a hosted code repository if necessary. Backups are essential as well.
  • Block off large amounts of undivided time to work on the project. For instance, it’s not a good day to write code if you have to leave every few minutes.

Development

  • Spend as much time as possible understanding the real-world and data needs of the client before writing any code. This is absolutely critical to make sure that your application is designed from the ground as a stable, relevant application. Don’t move forward until you think that you have a pretty good grasp of the needs; get help if you need to.
  • Out-engineer user-error as much as possible. In other words, never trust that the user will do what you expect, especially when entering data. For instance, about 4 years ago, I was reviewing a proposed payroll-entry system software for a Fortune-500 company. The software trusted the user to enter the number of their dependents properly for tax deduction purposes. When I questioned the developer, “What if someone enters a negative number?”, his face grew pale. We found that an employee would end up with a raise, instead of a deduction! Ouch.
  • Don’t reinvent the wheel. For instance, it makes no sense to develop your own login system for Ruby on Rails when there are great ones already out there that have run through a gauntlet in the real-world.
  • Don’t implement anything unless you understand, as much as time allows, what you’re “plugging in”. Read the issues reported, read through the source code, and familiarize yourself with how it works. Don’t choose something merely because 3 people recommended it in some forum. I’ve seen this often: a developer quickly slaps in a 3rd party open source app or gem and a few months later, there’s an issue that they do not know how to resolve. Usually it’s because they thought they’d found a quick fix, without thinking about any future issues. Perhaps the worst example that I’ve seen is hot-linking to Javascript code on a website somewhere, in blind trust that the link will stay live.
  • Be open to including other languages and technologies where appropriate. For instance, just because your app is written in Ruby, if a Perl or Python is better suited for a certain backend feature, it may be good to use it. Even if you don’t integrate it, reading through it can help you see a better way to solve an issue at hand.
  • Don’t do in code what your database can already do, as this will cut down the speed quite a bit. For instance, why use Date in Ruby in a SQL query on a MySQL-backed application when MySQL’s CURRENT_DATE() will work fine (and faster)? In other words, know the capabilities of the technologies that you’re using.
  • Avoid including “experimental” or “cutting-edge” features in your project if possible. While fun, the risk of issues just isn’t worth it unless your project is just a learning exercise.
  • Communicate what you need to keep the client in the loop. That being said, too much communication about every little snag and solution will only frustrate them.

Post-Development

  • Review your code for speed, stability, security, and usability.
  • Have someone else review it or discuss it with them.
  • Have non-technical people do real-world testing on your product. You’d be surprised at how many things you think are intuitive and easy that really aren’t to an average user.
  • Never, ever, ever stop learning, even if it’s unrelated to your current development language of choice.
  • Don’t be afraid to ask for help. Even the best developers can gain a lot of insight and knowledge from reading others’ code and interacting with others.
  • Revisit old code and see what you would’ve done differently. Often, you’ll find yourself encouraged as you see how better your code is now than then.

Enjoying Your Development:

  • How do you work best? In quiet, with music, with lots of lighting, dim lighting, etc…? A happy developer is much more likely to produce quality. Spend some time setting up your work environment the way that you’re most comfortable with. It’s amazing how mood-based programming can be. Improve your mood and you’ll improve your code.
  • What tool(s) do you like to use? As much as possible, insist on those tools, “Yes, I can write this app in TextEdit or Notepad, but I can do it better with this $79 tool X.”
  • Give yourself time to think and rest. There are some days where I just can’t write code well; other days where it’s just flowing. This is due to how your brain functions. You need sleep and a change of pace and scenery now and then. Schedule breaks if you must and realize that some days are going to be tough. On tough days, learn something new.
  • Walk away for a while. It’s easy to get “tunnel vision” and think that you’re close to solving a problem and to think that more effort will solve it. While true at times, I find that it’s best to walk away from the issue at hand for a while and do something completely different. For example, I tried once for 7 hours to power-through a complicated area of code and just couldn’t get it right. I finally stopped working on code that day, came back the next day to it, and solved it in 16 minutes. You would be surprised at the ideas or solutions that will spring into your mind as you are thinking about or doing other things.
  • Set realistic deadlines. For instance, before telling your client or boss that this will be done next Friday, add a week. You never know what issues you might hit and will find yourself, more often than not, very grateful for the extra time to get it done right. If you end early, you’re a hero!
  • Remember, you’re a person, not a robot. In other words, you deserve time to eat, sleep, get away, etc… When necessary, remind others nicely.
  • Know your limits and enforce them. In other words, “No, I can’t and won’t recreate Gmail.”

As the work quality improves, it will stand out and you will be called upon more and more to help others, do other projects, etc… As your level of expertise grows, help others by tactfully pointing out improvements, respecting them, and encouraging them. In doing so, you will find yourself enjoying the quality of your work.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Technorati Tags: , ,