North Star Engineering Principles For Successful Software Development Organisations

In the world of fintechs, digital payments and technology transformation its critical to have a defined north star of engineering principles. This defined north star of engineering principles keeps organisations and their workforce to stay on course of its purpose, its customer, its products and the promise made to its stakeholders with minimal or no digressions. So let's take a look at evergreen principles which should always be kept as north guiding development organizations. 

Evergreen Principles

User experience at the heart of engineering

Solving a real user problem should be an essential purpose of any software. Once ready, user experiences of using the software solution is critical to success. Customer delight is ultimate goal that needs to be targeted and tracked religiously. Metrics need to be defined and adopted to measure user experiences. Some of them can be behavioural (what users does) and some of them can be attitudinal (what users say). Constant monitoring of behavioural (error rate, time for task, response time) and attitudinal (Customer Satisfaction Score(CSAT), Net Promoter Score (NPS), System Usability Scale (SUS)) user experiences should be a mandatory for all organisations.

Plan for these, right from the start.

Security needs to be at the top of the table

In financial world security takes a defacto number 1 position. Though this is true for most other domains, apart from financial domain. Since protection of the users (business and consumer) and their data in an increasingly sophisticated cyber world is critical, to build the trust required to onboard and retain customers. Lack of security can quickly hurt businesses built over time through loss of revenue, intellectual property, and reputation. It should always be a top priority, no matter what software you build, operate or use.

Standard software security practices, such as no trust between application interfaces, protection of data at rest and on move, secure coding practices for developers, regularly reviewed authentication and authorisation policies should be a part of DNA of every software development company.

Build and Design Supporting Run Agility and Scalability from Start

Right from the start of SDLC the speed of delivery and scalability needs to be thought through. Every enabling factor needs to be brought in and every factor that slows down or blocks scalability needs to be weeded out. Ensuring simplicity and repeatability through design and implementations are key. Availability, redundancy needs to be factored in at start to ensure 'it always works' solution/user experience is delivered. 

At the same time overdoing can drag down, take build maintenance and operations costs higher, killing a software business unit or making it inefficient to compete in market. 

Questions like following need to be answered: 

Do we have single codebase to deploy ephemeral and disposable solution instances at will?  

Do we have single point of failures? 

Do we have automated zero downtime deployment capabilities? 

Do we setup and teardown automated tests and environments, similar to production? 

Are we active active behind load balancers? 

Do we need and have infrastructure as code capability?

Is design supportive of scaling up and down quickly?

 

Automated Testing is Critical 

Consider application/service code to be incomplete if it is not covered with automated test code for unit, functional and performance testing. Tests should be included in deployment process and service management process. Tests should include the validation of deployment and service management process. 

Always Build 12 Factor

Following is an introduction provided to 12 factor application methodology:

"Software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that:

• Use declarative formats for setup of automation, to minimize time and cost for new developers joining the project;
• Have a clean contract with the underlying operating system, offering maximum portability between execution environments;
• Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration;
• Minimize divergence between development and production, enabling continuous deployment for maximum agility;
• And can scale up without significant changes to tooling, architecture, or development practices.

The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc)." 

Let's take a closer look at twelve factors. The twelve factors to build software as a service successfully are:

1. Codebase: Have one version controlled codebase, capable of deploying ephemeral instances.
2. Dependencies: Explicitly declare and isolate dependencies. Dependency declaration manifest needs to have complete and exact declaration of all dependencies. Use dependency isolation tools during execution to ensure no dependencies leak in.
3. Config: A clear segregation of code and config is mandatory. Since code should remain constant across instances but config varies. An applications 'config' is everything that is likely to vary between different environments such as staging, production, development etc. including the following:
• Credentials to external services 
• Pre-deployment values, such as hostname/address for deployment 
• Resource handles to database, Memcached, and other backing services
4. Backing Services: Treat backing services as attached resources. i.e. any service the application consumes over the network as part of its operation/functionality needs to be treated as an attached resource. The code in a 12 factor application does not make distinction between local and third party services. Some common examples of backing services include caching systems such as Memcached, messaging queueing systems such as RabbitMQ, datastore such as Mongodb  or cassandra, metrics gathering services such as New Relic/Dynatrace, SMTP services for outbound email such as Postfix.
5. Build, release, run: Three steps transform a codebase to a deployed application, i.e the build, release and run. Twelve factor applications have a clear segregation of build, release and run stages.
• Conversion of code repo into an executable bundle by compiling binaries, fetching nd packaging dependent libraries and assets, on a specified version of code is commonly known as 'Build'.
• Addition of current or required config to the 'build' produced earlier is referred as release ready to be executed in the designated environment.
• Launch of the application 'release' in an environment is referred as 'run'.
6. Processes: Execute the app as one or more stateless processes. 12 factor apps/processes are stateless and share nothing. Persistence is managed generally via a backing service like a datastore. 
7. Port Binding: Export services via port binding. For example, web apps export HTTP as a service by binding to a port and listen to requests coming on that port. A 12 factor app is completely self-contained and does not rely on runtime injection of a webserver into the execution environment for creating a web-facing service.
8. Concurrency: Scale out via the process model. Processes are first class citizen in a 12 factor app, having similarities with unix process model of running service daemons. Each process handles their own complexity, threads, events; depending upon technology being used. Share nothing, horizontally partition-able nature of twelve factor applications makes concurrency easy and reliable.  
9. Disposability: Maximise robustness with fast startup and quick graceful shutdown. 12 factor application processes can be started and stopped as required i.e. they need to be disposable anytime and easy to create anytime. It facilitates robustness of production deploys, rapid changes to code/config through new deployments, and fast elastic scaling. 
10. Development and Production Environment Parity: Keep development, staging, and production environment as similar as possible. Minimise time, tool and people differences across environments including code and backing services. This prevents tests passing in lower environment, failing in production.
11. Logs: Logs should be treated as event streams and need to planned from start to enable metrics monitoring and tracking using synthetic monitoring.
12. Admin processes: Run admin or management tasks as one of processes.

Enable and Support a Culture of Seamless Open Collaboration of Code and Teams

Open-source model is a decentralised software development model that encourages open collaboration. A main principle of open-source software development is peer production, with products such as source code, blueprints, and documentation freely available. 

Maintain a service catalog of services and libraries to leverage the power of peer development. Have standard code development, management, design and documentation so that cross team/group collaboration is frictionless and easier.